diff mbox series

[v3,01/12] mm/huge_memory: zap deposited page tables after an RCU grace period

Message ID 20260911-rcu-pagetable-freeing-v3-1-7b8c86103821@kernel.org
State New
Headers show
Series mm: make userland page table freeing RCU-safe | expand

Commit Message

Lorenzo Stoakes (ARM) Sept. 11, 2026, 7:36 p.m. UTC
When an anonymous mapping is collapsed for THP, a PTE page table is
'deposited' with the installed PMD entry.

This is done in order that a split can be performed without needing to
allocate additional memory.

The freeing occurs in zap_deposited_table() and is done directly without
any delay via pte_free().

This is currently not a problem as existing page table walks are protected
by the mmap or anon rmap lock.

However this becomes problematic in a future where RCU-only page table
walkers exist, as there is nothing to prevent a page table walker that
started the walk prior to collapse having its PTE table freed underneath
it.

Commit 13cf577e6b66 ("mm/pgtable: add pte_free_defer() for pgtable as
page") already provides us the mechanism by which to solve this -
pte_free_defer().

Therefore, as a prerequisite to a future commit which will permit fully RCU
page table walks, update zap_deposited_table() to use pte_free_defer()
rather than pte_free().

Note that the IPI sync in collapse_huge_page() is still required to ensure
refcount correctness against a GUP-fast operation.

This is because GUP-fast might increment refcount, but
__collapse_huge_page_isolate() determines whether it is safe to proceed by
checking folio_ref_count() against folio_expected_ref_count(), so the two
must be mutually excluded.

Acked-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
---
 mm/huge_memory.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comments

David Hildenbrand (Arm) Sept. 16, 2026, 2:42 p.m. UTC | #1
On 9/11/26 21:36, Lorenzo Stoakes (ARM) wrote:
> When an anonymous mapping is collapsed for THP, a PTE page table is
> 'deposited' with the installed PMD entry.

Right. Or when we allocate an anon THP.

> 
> This is done in order that a split can be performed without needing to
> allocate additional memory.
> 
> The freeing occurs in zap_deposited_table() and is done directly without
> any delay via pte_free().
> 
> This is currently not a problem as existing page table walks are protected
> by the mmap or anon rmap lock.

Or VMA lock?

> 
> However this becomes problematic in a future where RCU-only page table
> walkers exist, as there is nothing to prevent a page table walker that
> started the walk prior to collapse having its PTE table freed underneath
> it.


Wait, but wouldn't it be really problematic to punch a page table that is still
being walked into the deposited list where it can just be allocated from another
PMD->PTE split?

Note that pgtable_trans_huge_withdraw() just dequeues *some* PTE page table in
the list attached to the PMD table.

Something is odd here.
Lorenzo Stoakes (ARM) Sept. 16, 2026, 4:59 p.m. UTC | #2
On Wed, Sep 16, 2026 at 04:42:04PM +0200, David Hildenbrand (Arm) wrote:
> On 9/11/26 21:36, Lorenzo Stoakes (ARM) wrote:
> > When an anonymous mapping is collapsed for THP, a PTE page table is
> > 'deposited' with the installed PMD entry.
>
> Right. Or when we allocate an anon THP.

That case doesn't interact with page walking though, as they're
just-allocated right?

Am focusing on collapse case for that reason rather than listing all
possible places deposit can happen.

>
> >
> > This is done in order that a split can be performed without needing to
> > allocate additional memory.
> >
> > The freeing occurs in zap_deposited_table() and is done directly without
> > any delay via pte_free().
> >
> > This is currently not a problem as existing page table walks are protected
> > by the mmap or anon rmap lock.
>
> Or VMA lock?

Yup indeed, can update on respin/ask Andrew to add VMA lock here.

>
> >
> > However this becomes problematic in a future where RCU-only page table
> > walkers exist, as there is nothing to prevent a page table walker that
> > started the walk prior to collapse having its PTE table freed underneath
> > it.
>
>
> Wait, but wouldn't it be really problematic to punch a page table that is still

Punch? You mean deposit?

> being walked into the deposited list where it can just be allocated from another
> PMD->PTE split?

In general, RCU-only page table walkers are _only_ guaranteed that the page
tables are not freed from underneath them, as per the cover letter:

	"As a result, page table walks can now be performed safely under RCU without
	 any risk of page tables being freed underneath a walker.

	However, this is the only guarantee that this work provides - page table
	walkers must still ensure that page table entries are as expected
	throughout."

So this series doesn't actually have to answer that :)

But a PTE PTL -> pmd_same() check will flag anything like that,
a.k.a. pte_offset_map_lock().

The RCU lock replaces stablisation on stuff other than VMA/mmap or rmap
lock.

If the walker needs to be sure the PTE is actually valid and belongs to the
expected walk then further is required.

Alternatively (like GUP-fast) if you wanted to avoid locking at all, IRQs
off would be required paired with the tlb_remove_table_sync_one() in
collapse_huge_page().

Another strategy could be used I guess with looped checks but it gets a bit
sketchy with timing etc.

But actually maybe we can avoid all that...

>
> Note that pgtable_trans_huge_withdraw() just dequeues *some* PTE page table in
> the list attached to the PMD table.
>
> Something is odd here.

...Since this code path already allocates (the huge folio), so allocation
here isn't an issue (as long as done outside of lcosk), why not have it
also allocate a new PTE to deposit and RCU-free the existing PTE instead?

That eliminates the one place in the kernel (afaik) where a
page-table-walker-visible table can just get yoinked over somewhere else.

Then a lockless PMD check works.

E.g. something like the below?

diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index f49a6710933b..d9cb6b406ebb 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1293,7 +1293,7 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s
 	LIST_HEAD(compound_pagelist);
 	pmd_t *pmd, _pmd;
 	pte_t *pte = NULL;
-	pgtable_t pgtable;
+	pgtable_t pgtable = NULL;
 	struct folio *folio;
 	spinlock_t *pmd_ptl, *pte_ptl;
 	enum scan_result result = SCAN_FAIL;
@@ -1310,6 +1310,12 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s
 		goto out_nolock;
 	}

+	if (is_pmd_order(order)) {
+		pgtable = pte_alloc_one(mm);
+		if (!pgtable)
+			goto out_nolock;
+	}
+
 	mmap_read_lock(mm);
 	result = hugepage_vma_revalidate(mm, pmd_addr, /*expect_anon=*/ true,
 					 &vma, cc, order);
@@ -1433,8 +1439,8 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s
 	spin_lock(pmd_ptl);
 	VM_WARN_ON_ONCE(!pmd_none(*pmd));
 	if (is_pmd_order(order)) {
-		pgtable = pmd_pgtable(_pmd);
 		pgtable_trans_huge_deposit(mm, pmd, pgtable);
+		pgtable = NULL;
 		map_anon_folio_pmd_nopf(folio, pmd, vma, pmd_addr);
 	} else {
 		/*
@@ -1453,6 +1459,9 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s
 	}
 	spin_unlock(pmd_ptl);

+	if (is_pmd_order(order))
+		pte_free_defer(mm, pmd_pgtable(_pmd));
+
 	folio = NULL;

 	result = SCAN_SUCCEED;
@@ -1463,6 +1472,8 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long s
 		anon_vma_unlock_write(vma->anon_vma);
 	mmap_write_unlock(mm);
 out_nolock:
+	if (pgtable)
+		pte_free(mm, pgtable);
 	if (folio)
 		folio_put(folio);
 	trace_mm_collapse_huge_page(mm, result == SCAN_SUCCEED, result, order);


>
> --
> Cheers,
>
> David

--
Cheers, Lorenzo
David Hildenbrand (Arm) Sept. 21, 2026, 5:13 p.m. UTC | #3
On 9/16/26 18:59, Lorenzo Stoakes (ARM) wrote:
> On Wed, Sep 16, 2026 at 04:42:04PM +0200, David Hildenbrand (Arm) wrote:
>> On 9/11/26 21:36, Lorenzo Stoakes (ARM) wrote:
>>> When an anonymous mapping is collapsed for THP, a PTE page table is
>>> 'deposited' with the installed PMD entry.
>>
>> Right. Or when we allocate an anon THP.
> 
> That case doesn't interact with page walking though, as they're
> just-allocated right?

I was surprised to read that page tables deposited during collapse should behave
differently than page tables deposited during allocation :)

That's the semantic change you are adding (and that I didn't quite understand
from reading your description): a deposited page table could now get reused and
modified for another page table due to a PMD split, while there might still be
an RCU page table walker walking it. And PMD comparisons (+ proper memory
ordering) have to make sure that we didn't just walk  something that doesn't
belong to us.

[...]

>>
>> Wait, but wouldn't it be really problematic to punch a page table that is still
> 
> Punch? You mean deposit?

Yes!

> 
>> being walked into the deposited list where it can just be allocated from another
>> PMD->PTE split?
> 
> In general, RCU-only page table walkers are _only_ guaranteed that the page
> tables are not freed from underneath them, as per the cover letter:
> 
> 	"As a result, page table walks can now be performed safely under RCU without
> 	 any risk of page tables being freed underneath a walker.
> 
> 	However, this is the only guarantee that this work provides - page table
> 	walkers must still ensure that page table entries are as expected
> 	throughout."

So far page table walkers never walk deposited page tables. Just like they never
walk freed page tables.

For *current* lockless page table walkers (IRQs disabled),
tlb_remove_table_sync_one() takes care of that. So there is no such thing as
"walking a page table that was reused in different context (as a page table) in
the meantime".

> 
> So this series doesn't actually have to answer that :)
> 
> But a PTE PTL -> pmd_same() check will flag anything like that,
> a.k.a. pte_offset_map_lock().
> 
> The RCU lock replaces stablisation on stuff other than VMA/mmap or rmap
> lock.
> 
> If the walker needs to be sure the PTE is actually valid and belongs to the
> expected walk then further is required.
> 
> Alternatively (like GUP-fast) if you wanted to avoid locking at all, IRQs
> off would be required paired with the tlb_remove_table_sync_one() in
> collapse_huge_page().

Doing an RCU sync in tlb_remove_table_sync_one() would be the big hammer (I
thought that was discussed with Lance recently). There is also this option: just
allocate a new page table and deposit it, and free the existing page table
through RCU.

Long story short: allowing empty page tables to be freed by RCU is simple.
Allowing empty page tables to be reused while they are still getting walked is
obviously more complicated conceptually. Likely it can be made working, but I'd
hope that we can keep it simpler.

> 
> Another strategy could be used I guess with looped checks but it gets a bit
> sketchy with timing etc.
> 
> But actually maybe we can avoid all that...
> 
>>
>> Note that pgtable_trans_huge_withdraw() just dequeues *some* PTE page table in
>> the list attached to the PMD table.
>>
>> Something is odd here.
> 
> ...Since this code path already allocates (the huge folio), so allocation
> here isn't an issue (as long as done outside of lcosk), why not have it
> also allocate a new PTE to deposit and RCU-free the existing PTE instead?
> 
> That eliminates the one place in the kernel (afaik) where a
> page-table-walker-visible table can just get yoinked over somewhere else.
> 
> Then a lockless PMD check works.
> 
> E.g. something like the below?

Hah! Great minds think alike.
Lorenzo Stoakes (ARM) Sept. 22, 2026, 12:14 p.m. UTC | #4
On Mon, Sep 21, 2026 at 07:13:16PM +0200, David Hildenbrand (Arm) wrote:
> On 9/16/26 18:59, Lorenzo Stoakes (ARM) wrote:
> > On Wed, Sep 16, 2026 at 04:42:04PM +0200, David Hildenbrand (Arm) wrote:
> >> On 9/11/26 21:36, Lorenzo Stoakes (ARM) wrote:
> >>> When an anonymous mapping is collapsed for THP, a PTE page table is
> >>> 'deposited' with the installed PMD entry.
> >>
> >> Right. Or when we allocate an anon THP.
> >
> > That case doesn't interact with page walking though, as they're
> > just-allocated right?
>
> I was surprised to read that page tables deposited during collapse should behave
> differently than page tables deposited during allocation :)

You'll be super surprised by how VMAs behave when detached vs. attached then ;)

>
> That's the semantic change you are adding (and that I didn't quite understand
> from reading your description): a deposited page table could now get reused and
> modified for another page table due to a PMD split, while there might still be
> an RCU page table walker walking it. And PMD comparisons (+ proper memory
> ordering) have to make sure that we didn't just walk  something that doesn't
> belong to us.
>
> [...]
>
> >>
> >> Wait, but wouldn't it be really problematic to punch a page table that is still
> >
> > Punch? You mean deposit?
>
> Yes!

Ack.

>
> >
> >> being walked into the deposited list where it can just be allocated from another
> >> PMD->PTE split?
> >
> > In general, RCU-only page table walkers are _only_ guaranteed that the page
> > tables are not freed from underneath them, as per the cover letter:
> >
> > 	"As a result, page table walks can now be performed safely under RCU without
> > 	 any risk of page tables being freed underneath a walker.
> >
> > 	However, this is the only guarantee that this work provides - page table
> > 	walkers must still ensure that page table entries are as expected
> > 	throughout."
>
> So far page table walkers never walk deposited page tables. Just like they never
> walk freed page tables.
>
> For *current* lockless page table walkers (IRQs disabled),
> tlb_remove_table_sync_one() takes care of that. So there is no such thing as
> "walking a page table that was reused in different context (as a page table) in
> the meantime".

Yup I know. That describes what happens for RCU page table walkers _after_ the
changes of course.

>
> >
> > So this series doesn't actually have to answer that :)
> >
> > But a PTE PTL -> pmd_same() check will flag anything like that,
> > a.k.a. pte_offset_map_lock().
> >
> > The RCU lock replaces stablisation on stuff other than VMA/mmap or rmap
> > lock.
> >
> > If the walker needs to be sure the PTE is actually valid and belongs to the
> > expected walk then further is required.
> >
> > Alternatively (like GUP-fast) if you wanted to avoid locking at all, IRQs
> > off would be required paired with the tlb_remove_table_sync_one() in
> > collapse_huge_page().
>
> Doing an RCU sync in tlb_remove_table_sync_one() would be the big hammer (I
> thought that was discussed with Lance recently). There is also this option: just
> allocate a new page table and deposit it, and free the existing page table
> through RCU.

Hmm yeah big old lump hammer :) I mean I don't see how that could really
possibly fly as an option, we have tlb_remove_table_sync_rcu() for
!CONFIG_PT_RECLAIM arches (i.e. err... the more 'specialist' ones :) but for
those in actual use that sounds a wee bit nuclear.

>
> Long story short: allowing empty page tables to be freed by RCU is simple.
> Allowing empty page tables to be reused while they are still getting walked is
> obviously more complicated conceptually. Likely it can be made working, but I'd
> hope that we can keep it simpler.

I'm not sure it can really safely be guaranteed or it'd involve more lock
dependencies or something horrible anyway.

But... we don't need to care about any of that....

>
> >
> > Another strategy could be used I guess with looped checks but it gets a bit
> > sketchy with timing etc.
> >
> > But actually maybe we can avoid all that...
> >
> >>
> >> Note that pgtable_trans_huge_withdraw() just dequeues *some* PTE page table in
> >> the list attached to the PMD table.
> >>
> >> Something is odd here.
> >
> > ...Since this code path already allocates (the huge folio), so allocation
> > here isn't an issue (as long as done outside of lcosk), why not have it
> > also allocate a new PTE to deposit and RCU-free the existing PTE instead?
> >
> > That eliminates the one place in the kernel (afaik) where a
> > page-table-walker-visible table can just get yoinked over somewhere else.
> >
> > Then a lockless PMD check works.
> >
> > E.g. something like the below?
>
> Hah! Great minds think alike.

...Because great minds indeed <3

Will respin with that then, thanks! :)

>
> --
> Cheers,
>
> David

--
Cheers, Lorenzo
diff mbox series

Patch

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 54494c3fa983..505f7b62ff28 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2476,7 +2476,7 @@  static inline void zap_deposited_table(struct mm_struct *mm, pmd_t *pmd)
 	pgtable_t pgtable;
 
 	pgtable = pgtable_trans_huge_withdraw(mm, pmd);
-	pte_free(mm, pgtable);
+	pte_free_defer(mm, pgtable);
 	mm_dec_nr_ptes(mm);
 }