[PATCH v2 0/3] io_uring mm related abuses

[PATCH v2 0/3] io_uring mm related abuses

[Pavel Begunkov]

Patch 1 uses unpin_user_folio instead of the page variant.
Patches 2-3 make sure io_uring doesn't make any assumptions
about user pointer alignments.

v2: change patch 1 tags
    use folio_page_idx()

Pavel Begunkov (3):
  io_uring/rsrc: fix folio unpinning
  io_uring/rsrc: don't rely on user vaddr alignment
  io_uring: don't assume uaddr alignment in io_vec_fill_bvec

 io_uring/rsrc.c | 27 ++++++++++++++++++++-------
 io_uring/rsrc.h |  1 +
 2 files changed, 21 insertions(+), 7 deletions(-)

-- 
2.49.0

[PATCH v2 1/3] io_uring/rsrc: fix folio unpinning

[Pavel Begunkov]

[  108.070381][   T14] kernel BUG at mm/gup.c:71!
[  108.070502][   T14] Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
[  108.123672][   T14] Hardware name: QEMU KVM Virtual Machine, BIOS edk2-20250221-8.fc42 02/21/2025
[  108.127458][   T14] Workqueue: iou_exit io_ring_exit_work
[  108.174205][   T14] Call trace:
[  108.175649][   T14]  sanity_check_pinned_pages+0x7cc/0x7d0 (P)
[  108.178138][   T14]  unpin_user_page+0x80/0x10c
[  108.180189][   T14]  io_release_ubuf+0x84/0xf8
[  108.182196][   T14]  io_free_rsrc_node+0x250/0x57c
[  108.184345][   T14]  io_rsrc_data_free+0x148/0x298
[  108.186493][   T14]  io_sqe_buffers_unregister+0x84/0xa0
[  108.188991][   T14]  io_ring_ctx_free+0x48/0x480
[  108.191057][   T14]  io_ring_exit_work+0x764/0x7d8
[  108.193207][   T14]  process_one_work+0x7e8/0x155c
[  108.195431][   T14]  worker_thread+0x958/0xed8
[  108.197561][   T14]  kthread+0x5fc/0x75c
[  108.199362][   T14]  ret_from_fork+0x10/0x20

We can pin a tail page of a folio, but then io_uring will try to unpin
the the head page of the folio. While it should be fine in terms of
keeping the page actually alive, but mm folks say it's wrong and
triggers a debug warning. Use unpin_user_folio() instead of
unpin_user_page*.

Cc: stable@vger.kernel.org
Debugged-by: David Hildenbrand <david@redhat.com>
Reported-by: syzbot+1d335893772467199ab6@syzkaller.appspotmail.com
Closes: https://lkml.kernel.org/r/683f1551.050a0220.55ceb.0017.GAE@google.com
Fixes: a8edbb424b139 ("io_uring/rsrc: enable multi-hugepage buffer coalescing")
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
 io_uring/rsrc.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index c592ceace97d..e83a294c718b 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -112,8 +112,11 @@ static void io_release_ubuf(void *priv)
 	struct io_mapped_ubuf *imu = priv;
 	unsigned int i;
 
-	for (i = 0; i < imu->nr_bvecs; i++)
-		unpin_user_page(imu->bvec[i].bv_page);
+	for (i = 0; i < imu->nr_bvecs; i++) {
+		struct folio *folio = page_folio(imu->bvec[i].bv_page);
+
+		unpin_user_folio(folio, 1);
+	}
 }
 
 static struct io_mapped_ubuf *io_alloc_imu(struct io_ring_ctx *ctx,
@@ -810,7 +813,8 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
 	imu->nr_bvecs = nr_pages;
 	ret = io_buffer_account_pin(ctx, pages, nr_pages, imu, last_hpage);
 	if (ret) {
-		unpin_user_pages(pages, nr_pages);
+		for (i = 0; i < nr_pages; i++)
+			unpin_user_folio(page_folio(pages[i]), 1);
 		goto done;
 	}
 
-- 
2.49.0

Re: [PATCH v2 1/3] io_uring/rsrc: fix folio unpinning

[David Hildenbrand]

On 24.06.25 15:40, Pavel Begunkov wrote:
> [  108.070381][   T14] kernel BUG at mm/gup.c:71!
> [  108.070502][   T14] Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
> [  108.123672][   T14] Hardware name: QEMU KVM Virtual Machine, BIOS edk2-20250221-8.fc42 02/21/2025
> [  108.127458][   T14] Workqueue: iou_exit io_ring_exit_work
> [  108.174205][   T14] Call trace:
> [  108.175649][   T14]  sanity_check_pinned_pages+0x7cc/0x7d0 (P)
> [  108.178138][   T14]  unpin_user_page+0x80/0x10c
> [  108.180189][   T14]  io_release_ubuf+0x84/0xf8
> [  108.182196][   T14]  io_free_rsrc_node+0x250/0x57c
> [  108.184345][   T14]  io_rsrc_data_free+0x148/0x298
> [  108.186493][   T14]  io_sqe_buffers_unregister+0x84/0xa0
> [  108.188991][   T14]  io_ring_ctx_free+0x48/0x480
> [  108.191057][   T14]  io_ring_exit_work+0x764/0x7d8
> [  108.193207][   T14]  process_one_work+0x7e8/0x155c
> [  108.195431][   T14]  worker_thread+0x958/0xed8
> [  108.197561][   T14]  kthread+0x5fc/0x75c
> [  108.199362][   T14]  ret_from_fork+0x10/0x20
> 
> We can pin a tail page of a folio, but then io_uring will try to unpin
> the the head page of the folio. While it should be fine in terms of
> keeping the page actually alive, but mm folks say it's wrong and
> triggers a debug warning. Use unpin_user_folio() instead of
> unpin_user_page*.
> 
> Cc: stable@vger.kernel.org
> Debugged-by: David Hildenbrand <david@redhat.com>
> Reported-by: syzbot+1d335893772467199ab6@syzkaller.appspotmail.com
> Closes: https://lkml.kernel.org/r/683f1551.050a0220.55ceb.0017.GAE@google.com
> Fixes: a8edbb424b139 ("io_uring/rsrc: enable multi-hugepage buffer coalescing")
> Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
> ---
>   io_uring/rsrc.c | 10 +++++++---
>   1 file changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
> index c592ceace97d..e83a294c718b 100644
> --- a/io_uring/rsrc.c
> +++ b/io_uring/rsrc.c
> @@ -112,8 +112,11 @@ static void io_release_ubuf(void *priv)
>   	struct io_mapped_ubuf *imu = priv;
>   	unsigned int i;
>   
> -	for (i = 0; i < imu->nr_bvecs; i++)
> -		unpin_user_page(imu->bvec[i].bv_page);
> +	for (i = 0; i < imu->nr_bvecs; i++) {
> +		struct folio *folio = page_folio(imu->bvec[i].bv_page);
> +
> +		unpin_user_folio(folio, 1);
> +	}
>   }
>   
>   static struct io_mapped_ubuf *io_alloc_imu(struct io_ring_ctx *ctx,
> @@ -810,7 +813,8 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
>   	imu->nr_bvecs = nr_pages;
>   	ret = io_buffer_account_pin(ctx, pages, nr_pages, imu, last_hpage);
>   	if (ret) {
> -		unpin_user_pages(pages, nr_pages);
> +		for (i = 0; i < nr_pages; i++)
> +			unpin_user_folio(page_folio(pages[i]), 1);
>   		goto done;
>   	}
>   

It should fix the issue, but it's a bit suboptimal in the case where we 
didn't coalesc, but there are folio ranges to coalesc:

unpin_user_pages() does a per-folio coalescing.

So in an ideal world, we would cleanly split both paths, and work with 
folios after we coalesced to use folios, and work with pages, when we 
didn't coalesc to use folios.

Then, we can just use unpin_folios() after we coalesced.

In any case, for a fix this is good enough, but probably we can do 
better later.

Acked-by: David Hildenbrand <david@redhat.com>

-- 
Cheers,

David / dhildenb

Re: [PATCH v2 1/3] io_uring/rsrc: fix folio unpinning

[Pavel Begunkov]

On 6/25/25 08:53, David Hildenbrand wrote:
...>>   static struct io_mapped_ubuf *io_alloc_imu(struct io_ring_ctx *ctx,
>> @@ -810,7 +813,8 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
>>       imu->nr_bvecs = nr_pages;
>>       ret = io_buffer_account_pin(ctx, pages, nr_pages, imu, last_hpage);
>>       if (ret) {
>> -        unpin_user_pages(pages, nr_pages);
>> +        for (i = 0; i < nr_pages; i++)
>> +            unpin_user_folio(page_folio(pages[i]), 1);
>>           goto done;
>>       }
> 
> It should fix the issue, but it's a bit suboptimal in the case where we didn't coalesc, but there are folio ranges to coalesc:
> 
> unpin_user_pages() does a per-folio coalescing.
> 
> So in an ideal world, we would cleanly split both paths, and work with folios after we coalesced to use folios, and work with pages, when we didn't coalesc to use folios.

Agreed, but I'm not too much concerned, it's a slow path, users
should be mindful while registering memory. Localised hammering
on the refcount is not great but ultimately shouldn't matter much.

> Then, we can just use unpin_folios() after we coalesced.
> 
> In any case, for a fix this is good enough, but probably we can do better later.
> 
> Acked-by: David Hildenbrand <david@redhat.com>

Thanks for the report!

-- 
Pavel Begunkov

[PATCH v2 2/3] io_uring/rsrc: don't rely on user vaddr alignment

[Pavel Begunkov]

There is no guaranteed alignment for user pointers, however the
calculation of an offset of the first page into a folio after
coalescing uses some weird bit mask logic, get rid of it.

Cc: stable@vger.kernel.org
Reported-by: David Hildenbrand <david@redhat.com>
Fixes: a8edbb424b139 ("io_uring/rsrc: enable multi-hugepage buffer coalescing")
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
 io_uring/rsrc.c | 7 ++++++-
 io_uring/rsrc.h | 1 +
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index e83a294c718b..8b06c732d136 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -734,6 +734,7 @@ bool io_check_coalesce_buffer(struct page **page_array, int nr_pages,
 
 	data->nr_pages_mid = folio_nr_pages(folio);
 	data->folio_shift = folio_shift(folio);
+	data->first_folio_page_idx = folio_page_idx(folio, page_array[0]);
 
 	/*
 	 * Check if pages are contiguous inside a folio, and all folios have
@@ -830,7 +831,11 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
 	if (coalesced)
 		imu->folio_shift = data.folio_shift;
 	refcount_set(&imu->refs, 1);
-	off = (unsigned long) iov->iov_base & ((1UL << imu->folio_shift) - 1);
+
+	off = (unsigned long)iov->iov_base & ~PAGE_MASK;
+	if (coalesced)
+		off += data.first_folio_page_idx << PAGE_SHIFT;
+
 	node->buf = imu;
 	ret = 0;
 
diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h
index 0d2138f16322..25e7e998dcfd 100644
--- a/io_uring/rsrc.h
+++ b/io_uring/rsrc.h
@@ -49,6 +49,7 @@ struct io_imu_folio_data {
 	unsigned int	nr_pages_mid;
 	unsigned int	folio_shift;
 	unsigned int	nr_folios;
+	unsigned long	first_folio_page_idx;
 };
 
 bool io_rsrc_cache_init(struct io_ring_ctx *ctx);
-- 
2.49.0

Re: [PATCH v2 2/3] io_uring/rsrc: don't rely on user vaddr alignment

[David Hildenbrand]

On 24.06.25 15:40, Pavel Begunkov wrote:
> There is no guaranteed alignment for user pointers, however the
> calculation of an offset of the first page into a folio after
> coalescing uses some weird bit mask logic, get rid of it.
> 
> Cc: stable@vger.kernel.org
> Reported-by: David Hildenbrand <david@redhat.com>
> Fixes: a8edbb424b139 ("io_uring/rsrc: enable multi-hugepage buffer coalescing")
> Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
> ---
>   io_uring/rsrc.c | 7 ++++++-
>   io_uring/rsrc.h | 1 +
>   2 files changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
> index e83a294c718b..8b06c732d136 100644
> --- a/io_uring/rsrc.c
> +++ b/io_uring/rsrc.c
> @@ -734,6 +734,7 @@ bool io_check_coalesce_buffer(struct page **page_array, int nr_pages,
>   
>   	data->nr_pages_mid = folio_nr_pages(folio);
>   	data->folio_shift = folio_shift(folio);
> +	data->first_folio_page_idx = folio_page_idx(folio, page_array[0]);
>   
>   	/*
>   	 * Check if pages are contiguous inside a folio, and all folios have
> @@ -830,7 +831,11 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
>   	if (coalesced)
>   		imu->folio_shift = data.folio_shift;
>   	refcount_set(&imu->refs, 1);
> -	off = (unsigned long) iov->iov_base & ((1UL << imu->folio_shift) - 1);
> +
> +	off = (unsigned long)iov->iov_base & ~PAGE_MASK;
> +	if (coalesced)
> +		off += data.first_folio_page_idx << PAGE_SHIFT;
> +
>   	node->buf = imu;
>   	ret = 0;
>   
> diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h
> index 0d2138f16322..25e7e998dcfd 100644
> --- a/io_uring/rsrc.h
> +++ b/io_uring/rsrc.h
> @@ -49,6 +49,7 @@ struct io_imu_folio_data {
>   	unsigned int	nr_pages_mid;
>   	unsigned int	folio_shift;
>   	unsigned int	nr_folios;
> +	unsigned long	first_folio_page_idx;
>   };
>   
>   bool io_rsrc_cache_init(struct io_ring_ctx *ctx);

Acked-by: David Hildenbrand <david@redhat.com>

-- 
Cheers,

David / dhildenb

[PATCH v2 3/3] io_uring: don't assume uaddr alignment in io_vec_fill_bvec

[Pavel Begunkov]

There is no guaranteed alignment for user pointers. Don't use mask
trickery and adjust the offset by bv_offset.

Cc: stable@vger.kernel.org
Reported-by: David Hildenbrand <david@redhat.com>
Fixes: 9ef4cbbcb4ac3 ("io_uring: add infra for importing vectored reg buffers")
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
 io_uring/rsrc.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index 8b06c732d136..c58fe736f297 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -1336,7 +1336,6 @@ static int io_vec_fill_bvec(int ddir, struct iov_iter *iter,
 {
 	unsigned long folio_size = 1 << imu->folio_shift;
 	unsigned long folio_mask = folio_size - 1;
-	u64 folio_addr = imu->ubuf & ~folio_mask;
 	struct bio_vec *res_bvec = vec->bvec;
 	size_t total_len = 0;
 	unsigned bvec_idx = 0;
@@ -1358,8 +1357,13 @@ static int io_vec_fill_bvec(int ddir, struct iov_iter *iter,
 		if (unlikely(check_add_overflow(total_len, iov_len, &total_len)))
 			return -EOVERFLOW;
 
-		/* by using folio address it also accounts for bvec offset */
-		offset = buf_addr - folio_addr;
+		offset = buf_addr - imu->ubuf;
+		/*
+		 * Only the first bvec can have non zero bv_offset, account it
+		 * here and work with full folios below.
+		 */
+		offset += imu->bvec[0].bv_offset;
+
 		src_bvec = imu->bvec + (offset >> imu->folio_shift);
 		offset &= folio_mask;
 
-- 
2.49.0

Re: [PATCH v2 0/3] io_uring mm related abuses

[Jens Axboe]

On 6/24/25 7:40 AM, Pavel Begunkov wrote:
> Patch 1 uses unpin_user_folio instead of the page variant.
> Patches 2-3 make sure io_uring doesn't make any assumptions
> about user pointer alignments.
> 
> v2: change patch 1 tags
>     use folio_page_idx()
> 
> Pavel Begunkov (3):
>   io_uring/rsrc: fix folio unpinning
>   io_uring/rsrc: don't rely on user vaddr alignment
>   io_uring: don't assume uaddr alignment in io_vec_fill_bvec
> 
>  io_uring/rsrc.c | 27 ++++++++++++++++++++-------
>  io_uring/rsrc.h |  1 +
>  2 files changed, 21 insertions(+), 7 deletions(-)

Hand applied, as this is against an older tree. Please check patch 1
in the current tree. Thanks!

-- 
Jens Axboe

Re: [PATCH v2 0/3] io_uring mm related abuses

[Pavel Begunkov]

On 6/25/25 03:52, Jens Axboe wrote:
> On 6/24/25 7:40 AM, Pavel Begunkov wrote:
>> Patch 1 uses unpin_user_folio instead of the page variant.
>> Patches 2-3 make sure io_uring doesn't make any assumptions
>> about user pointer alignments.
>>
>> v2: change patch 1 tags
>>      use folio_page_idx()
>>
>> Pavel Begunkov (3):
>>    io_uring/rsrc: fix folio unpinning
>>    io_uring/rsrc: don't rely on user vaddr alignment
>>    io_uring: don't assume uaddr alignment in io_vec_fill_bvec
>>
>>   io_uring/rsrc.c | 27 ++++++++++++++++++++-------
>>   io_uring/rsrc.h |  1 +
>>   2 files changed, 21 insertions(+), 7 deletions(-)
> 
> Hand applied, as this is against an older tree. Please check patch 1
> in the current tree. Thanks!

Turned to be for-next from a couple of days ago. Patch 1
looks the same, should be fine.

-- 
Pavel Begunkov

Re: [PATCH v2 0/3] io_uring mm related abuses

[Jens Axboe]

On 6/25/25 2:24 PM, Pavel Begunkov wrote:
> On 6/25/25 03:52, Jens Axboe wrote:
>> On 6/24/25 7:40 AM, Pavel Begunkov wrote:
>>> Patch 1 uses unpin_user_folio instead of the page variant.
>>> Patches 2-3 make sure io_uring doesn't make any assumptions
>>> about user pointer alignments.
>>>
>>> v2: change patch 1 tags
>>>      use folio_page_idx()
>>>
>>> Pavel Begunkov (3):
>>>    io_uring/rsrc: fix folio unpinning
>>>    io_uring/rsrc: don't rely on user vaddr alignment
>>>    io_uring: don't assume uaddr alignment in io_vec_fill_bvec
>>>
>>>   io_uring/rsrc.c | 27 ++++++++++++++++++++-------
>>>   io_uring/rsrc.h |  1 +
>>>   2 files changed, 21 insertions(+), 7 deletions(-)
>>
>> Hand applied, as this is against an older tree. Please check patch 1
>> in the current tree. Thanks!
> 
> Turned to be for-next from a couple of days ago. Patch 1
> looks the same, should be fine.

I don't always put current fixes in for-next, though I've tried to do it
consistently more recently. But it's conflicting with the error path
cleanup from about a week ago:

commit e1c75831f682eef0f68b35723437146ed86070b1 (tag: io_uring-6.16-20250619)
Author: Penglei Jiang <superman.xpt@gmail.com>
Date:   Tue Jun 17 09:56:44 2025 -0700

    io_uring: fix potential page leak in io_sqe_buffer_register()

and my hand-edit just put your hunk 2 of patch 1 into that cleanup path
too. Thanks for checking!

-- 
Jens Axboe

Re: [PATCH v2 0/3] io_uring mm related abuses

[Jens Axboe]

On Tue, 24 Jun 2025 14:40:32 +0100, Pavel Begunkov wrote:
> Patch 1 uses unpin_user_folio instead of the page variant.
> Patches 2-3 make sure io_uring doesn't make any assumptions
> about user pointer alignments.
> 
> v2: change patch 1 tags
>     use folio_page_idx()
> 
> [...]

Applied, thanks!

[1/3] io_uring/rsrc: fix folio unpinning
      commit: 5afb4bf9fc62d828647647ec31745083637132e4
[2/3] io_uring/rsrc: don't rely on user vaddr alignment
      commit: 3a3c6d61577dbb23c09df3e21f6f9eda1ecd634b
[3/3] io_uring: don't assume uaddr alignment in io_vec_fill_bvec
      commit: e1d7727b73a1f78035316ac35ee184d477059f0b

Best regards,
-- 
Jens Axboe