RCU warning off ublk_buf_cleanup() -> mas_for_each()

RCU warning off ublk_buf_cleanup() -> mas_for_each()

[Jens Axboe]

Hi Ming,

Ran into the below running tests on the current tree:

=============================
WARNING: suspicious RCU usage
7.0.0+ #16 Tainted: G                 N 
-----------------------------
lib/maple_tree.c:759 suspicious rcu_dereference_check() usage!

other info that might help us debug this:


rcu_scheduler_active = 2, debug_locks = 1
1 lock held by iou-wrk-55535/55536:
 #0: ffff800085a451a0 (ublk_ctl_mutex){+.+.}-{4:4}, at: ublk_ctrl_del_dev+0xdc/0x2f8

stack backtrace:
CPU: 4 UID: 0 PID: 55536 Comm: iou-wrk-55535 Tainted: G                 N  7.0.0+ #16 PREEMPT 
Tainted: [N]=TEST
Hardware name: linux,dummy-virt (DT)
Call trace:
 show_stack+0x1c/0x30 (C)
 dump_stack_lvl+0x68/0x90
 dump_stack+0x18/0x20
 lockdep_rcu_suspicious+0x170/0x200
 mas_walk+0x3f0/0x6a0
 mas_find+0x1b4/0x6b0
 ublk_buf_cleanup+0xe0/0x240
 ublk_cdev_rel+0x34/0x1b0
 device_release+0xa4/0x350
 kobject_put+0x138/0x250
 put_device+0x18/0x30
 ublk_put_device+0x18/0x28
 ublk_ctrl_del_dev+0x120/0x2f8
 ublk_ctrl_uring_cmd+0x598/0x29b8
 io_uring_cmd+0x1e0/0x468
 __io_issue_sqe+0xa4/0x748
 io_issue_sqe+0x80/0xf68
 io_wq_submit_work+0x26c/0xdc8
 io_worker_handle_work+0x334/0xf20
 io_wq_worker+0x278/0x9e8
 ret_from_fork+0x10/0x20
Buffer I/O error on dev ublkb0, logical block 0, async page read
Buffer I/O error on dev ublkb0, logical block 0, async page read
 ublkb0: unable to read partition table
Buffer I/O error on dev ublkb0, logical block 0, async page read
Buffer I/O error on dev ublkb0, logical block 0, async page read
Buffer I/O error on dev ublkb0, logical block 512, async page read
Buffer I/O error on dev ublkb0, logical block 512, async page read
Buffer I/O error on dev ublkb0, logical block 0, async page read
Buffer I/O error on dev ublkb0, logical block 512, async page read

and I briefly looked at it, but then just gave up as a) the maple tree
documentation is not that detailed, and b) other in-tree users also just
call mas_for_each() without either a lock held or RCU read side locked.

Adding Liam for shedding some light on this...

-- 
Jens Axboe

Re: RCU warning off ublk_buf_cleanup() -> mas_for_each()

[Jens Axboe]

On 4/21/26 11:47 AM, Jens Axboe wrote:
> Hi Ming,
> 
> Ran into the below running tests on the current tree:
> 
> =============================
> WARNING: suspicious RCU usage
> 7.0.0+ #16 Tainted: G                 N 
> -----------------------------
> lib/maple_tree.c:759 suspicious rcu_dereference_check() usage!

FWIW, here's what claude spits out on the maple tree usage for
ublk. Simply passing it on...

  Issue 1: ublk_buf_cleanup — missing RCU/lock for mas_for_each (line 5489)          
                                                                                     
  ublk_buf_cleanup iterates the tree using mas_for_each without holding either       
  rcu_read_lock or mas_lock. The mas_find() documentation explicitly states: "Must   
  hold rcu_read_lock or the write lock." Internally, mas_find → mas_next_slot →      
  mt_slot → rcu_dereference_check(slots[offset], mt_locked(mt)). With neither RCU nor
   the tree lock held, this will fire a lockdep splat on CONFIG_PROVE_RCU=y kernels. 

  While functionally safe (exclusive access during device release), the API contract 
  is violated. Fix: wrap the iteration in rcu_read_lock/unlock, or use
  mas_lock/unlock.                                                                   
                                                            
  Issue 2: __ublk_ctrl_unreg_buf — unpin_user_pages under spinlock (line 5431-5455)  
  
  This function holds mas_lock (a spinlock — atomic context) while unpinning         
  potentially many pages in a loop. For a large registered buffer with many disjoint
  PFN ranges, this holds the spinlock for an extended period. unpin_user_pages →     
  gup_put_folio → folio_put could also grab additional locks if the folio's refcount
  drops to zero.

  A cleaner pattern would be to collect entries, drop mas_lock, then unpin. Compare  
  with ublk_buf_cleanup which does the same unpinning work outside any lock — the
  asymmetry is notable.                                                              
                                                            
  Issue 3: ublk_buf_cleanup — kfree without mas_erase (line 5504)                    
   
  The cleanup function does kfree(range) during iteration without first calling      
  mas_erase(). This leaves dangling pointers in the tree nodes until mtree_destroy is
   called on line 5506. Not a bug (no concurrent access, mtree_destroy doesn't       
  dereference stored entries), but it's inconsistent with ublk_buf_erase_ranges and
  __ublk_ctrl_unreg_buf which both properly erase before freeing.


-- 
Jens Axboe

Re: RCU warning off ublk_buf_cleanup() -> mas_for_each()

[Liam R. Howlett]

* Jens Axboe <axboe@kernel.dk> [260421 13:47]:
> Hi Ming,
> 
> Ran into the below running tests on the current tree:
> 
> =============================
> WARNING: suspicious RCU usage
> 7.0.0+ #16 Tainted: G                 N 
> -----------------------------
> lib/maple_tree.c:759 suspicious rcu_dereference_check() usage!
> 
> other info that might help us debug this:
> 
> 
> rcu_scheduler_active = 2, debug_locks = 1
> 1 lock held by iou-wrk-55535/55536:
>  #0: ffff800085a451a0 (ublk_ctl_mutex){+.+.}-{4:4}, at: ublk_ctrl_del_dev+0xdc/0x2f8
> 
> stack backtrace:
> CPU: 4 UID: 0 PID: 55536 Comm: iou-wrk-55535 Tainted: G                 N  7.0.0+ #16 PREEMPT 
> Tainted: [N]=TEST
> Hardware name: linux,dummy-virt (DT)
> Call trace:
>  show_stack+0x1c/0x30 (C)
>  dump_stack_lvl+0x68/0x90
>  dump_stack+0x18/0x20
>  lockdep_rcu_suspicious+0x170/0x200
>  mas_walk+0x3f0/0x6a0
>  mas_find+0x1b4/0x6b0
>  ublk_buf_cleanup+0xe0/0x240
>  ublk_cdev_rel+0x34/0x1b0
>  device_release+0xa4/0x350
>  kobject_put+0x138/0x250
>  put_device+0x18/0x30
>  ublk_put_device+0x18/0x28
>  ublk_ctrl_del_dev+0x120/0x2f8
>  ublk_ctrl_uring_cmd+0x598/0x29b8
>  io_uring_cmd+0x1e0/0x468
>  __io_issue_sqe+0xa4/0x748
>  io_issue_sqe+0x80/0xf68
>  io_wq_submit_work+0x26c/0xdc8
>  io_worker_handle_work+0x334/0xf20
>  io_wq_worker+0x278/0x9e8
>  ret_from_fork+0x10/0x20
> Buffer I/O error on dev ublkb0, logical block 0, async page read
> Buffer I/O error on dev ublkb0, logical block 0, async page read
>  ublkb0: unable to read partition table
> Buffer I/O error on dev ublkb0, logical block 0, async page read
> Buffer I/O error on dev ublkb0, logical block 0, async page read
> Buffer I/O error on dev ublkb0, logical block 512, async page read
> Buffer I/O error on dev ublkb0, logical block 512, async page read
> Buffer I/O error on dev ublkb0, logical block 0, async page read
> Buffer I/O error on dev ublkb0, logical block 512, async page read
> 
> and I briefly looked at it, but then just gave up as a) the maple tree
> documentation is not that detailed,

Which documentation is lacking?  I will fix it.

I have user documentation in the Documentation directory while
technical details are in the code.


>and b) other in-tree users also just
> call mas_for_each() without either a lock held or RCU read side locked.

mas_for_each() must hold a lock of some type.

> 
> Adding Liam for shedding some light on this...
> 
> -- 
> Jens Axboe
> 

Re: RCU warning off ublk_buf_cleanup() -> mas_for_each()

[Jens Axboe]

On 4/21/26 2:28 PM, Liam R. Howlett wrote:
> * Jens Axboe <axboe@kernel.dk> [260421 13:47]:
>> Hi Ming,
>>
>> Ran into the below running tests on the current tree:
>>
>> =============================
>> WARNING: suspicious RCU usage
>> 7.0.0+ #16 Tainted: G                 N 
>> -----------------------------
>> lib/maple_tree.c:759 suspicious rcu_dereference_check() usage!
>>
>> other info that might help us debug this:
>>
>>
>> rcu_scheduler_active = 2, debug_locks = 1
>> 1 lock held by iou-wrk-55535/55536:
>>  #0: ffff800085a451a0 (ublk_ctl_mutex){+.+.}-{4:4}, at: ublk_ctrl_del_dev+0xdc/0x2f8
>>
>> stack backtrace:
>> CPU: 4 UID: 0 PID: 55536 Comm: iou-wrk-55535 Tainted: G                 N  7.0.0+ #16 PREEMPT 
>> Tainted: [N]=TEST
>> Hardware name: linux,dummy-virt (DT)
>> Call trace:
>>  show_stack+0x1c/0x30 (C)
>>  dump_stack_lvl+0x68/0x90
>>  dump_stack+0x18/0x20
>>  lockdep_rcu_suspicious+0x170/0x200
>>  mas_walk+0x3f0/0x6a0
>>  mas_find+0x1b4/0x6b0
>>  ublk_buf_cleanup+0xe0/0x240
>>  ublk_cdev_rel+0x34/0x1b0
>>  device_release+0xa4/0x350
>>  kobject_put+0x138/0x250
>>  put_device+0x18/0x30
>>  ublk_put_device+0x18/0x28
>>  ublk_ctrl_del_dev+0x120/0x2f8
>>  ublk_ctrl_uring_cmd+0x598/0x29b8
>>  io_uring_cmd+0x1e0/0x468
>>  __io_issue_sqe+0xa4/0x748
>>  io_issue_sqe+0x80/0xf68
>>  io_wq_submit_work+0x26c/0xdc8
>>  io_worker_handle_work+0x334/0xf20
>>  io_wq_worker+0x278/0x9e8
>>  ret_from_fork+0x10/0x20
>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>>  ublkb0: unable to read partition table
>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>> Buffer I/O error on dev ublkb0, logical block 512, async page read
>> Buffer I/O error on dev ublkb0, logical block 512, async page read
>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>> Buffer I/O error on dev ublkb0, logical block 512, async page read
>>
>> and I briefly looked at it, but then just gave up as a) the maple tree
>> documentation is not that detailed,
> 
> Which documentation is lacking?  I will fix it.
> 
> I have user documentation in the Documentation directory while
> technical details are in the code.

I went into the core-api/ and leafed through that, didn't have anything
on mas_for_each() that pertained to locking. Was hoping I'd find a table
of which parts of the API requires what in terms of locking or RCU.
There arent a whole lot of in-kernel users of it yet, so looking at
other places in the kernel wasn't very useful.

Since this is a merge window regression, I really just passed it to Ming
with you on the CC just in case, and didn't spend any more time on it.
I'm not the one that's supposed to be finding issues like this...

>> and b) other in-tree users also just
>> call mas_for_each() without either a lock held or RCU read side locked.
> 
> mas_for_each() must hold a lock of some type.

That's what I assumed. I missed that the rcu dereference check
checks for an external lock too, which I guess you can register
with the maple tree. Funky... I guess it's just for lockdep
purposes, makes sense then.

Presumably the current use case is fine, as it's serialized
teardown. It just ends up triggering the rcu sanity checks.

-- 
Jens Axboe

Re: RCU warning off ublk_buf_cleanup() -> mas_for_each()

[Liam R. Howlett]

* Jens Axboe <axboe@kernel.dk> [260421 17:41]:
> On 4/21/26 2:28 PM, Liam R. Howlett wrote:
> > * Jens Axboe <axboe@kernel.dk> [260421 13:47]:
> >> Hi Ming,
> >>
> >> Ran into the below running tests on the current tree:
> >>
> >> =============================
> >> WARNING: suspicious RCU usage
> >> 7.0.0+ #16 Tainted: G                 N 
> >> -----------------------------
> >> lib/maple_tree.c:759 suspicious rcu_dereference_check() usage!
> >>
> >> other info that might help us debug this:
> >>
> >>
> >> rcu_scheduler_active = 2, debug_locks = 1
> >> 1 lock held by iou-wrk-55535/55536:
> >>  #0: ffff800085a451a0 (ublk_ctl_mutex){+.+.}-{4:4}, at: ublk_ctrl_del_dev+0xdc/0x2f8
> >>
> >> stack backtrace:
> >> CPU: 4 UID: 0 PID: 55536 Comm: iou-wrk-55535 Tainted: G                 N  7.0.0+ #16 PREEMPT 
> >> Tainted: [N]=TEST
> >> Hardware name: linux,dummy-virt (DT)
> >> Call trace:
> >>  show_stack+0x1c/0x30 (C)
> >>  dump_stack_lvl+0x68/0x90
> >>  dump_stack+0x18/0x20
> >>  lockdep_rcu_suspicious+0x170/0x200
> >>  mas_walk+0x3f0/0x6a0
> >>  mas_find+0x1b4/0x6b0
> >>  ublk_buf_cleanup+0xe0/0x240
> >>  ublk_cdev_rel+0x34/0x1b0
> >>  device_release+0xa4/0x350
> >>  kobject_put+0x138/0x250
> >>  put_device+0x18/0x30
> >>  ublk_put_device+0x18/0x28
> >>  ublk_ctrl_del_dev+0x120/0x2f8
> >>  ublk_ctrl_uring_cmd+0x598/0x29b8
> >>  io_uring_cmd+0x1e0/0x468
> >>  __io_issue_sqe+0xa4/0x748
> >>  io_issue_sqe+0x80/0xf68
> >>  io_wq_submit_work+0x26c/0xdc8
> >>  io_worker_handle_work+0x334/0xf20
> >>  io_wq_worker+0x278/0x9e8
> >>  ret_from_fork+0x10/0x20
> >> Buffer I/O error on dev ublkb0, logical block 0, async page read
> >> Buffer I/O error on dev ublkb0, logical block 0, async page read
> >>  ublkb0: unable to read partition table
> >> Buffer I/O error on dev ublkb0, logical block 0, async page read
> >> Buffer I/O error on dev ublkb0, logical block 0, async page read
> >> Buffer I/O error on dev ublkb0, logical block 512, async page read
> >> Buffer I/O error on dev ublkb0, logical block 512, async page read
> >> Buffer I/O error on dev ublkb0, logical block 0, async page read
> >> Buffer I/O error on dev ublkb0, logical block 512, async page read
> >>
> >> and I briefly looked at it, but then just gave up as a) the maple tree
> >> documentation is not that detailed,
> > 
> > Which documentation is lacking?  I will fix it.
> > 
> > I have user documentation in the Documentation directory while
> > technical details are in the code.
> 
> I went into the core-api/ and leafed through that, didn't have anything
> on mas_for_each() that pertained to locking. Was hoping I'd find a table
> of which parts of the API requires what in terms of locking or RCU.
> There arent a whole lot of in-kernel users of it yet, so looking at
> other places in the kernel wasn't very useful.

There's several users and the test code.  Too bad none of them helped.

> 
> Since this is a merge window regression, I really just passed it to Ming
> with you on the CC just in case, and didn't spend any more time on it.
> I'm not the one that's supposed to be finding issues like this...
> 

I appreciate the response as I'll try to cater the doc to what users
search for.

There are two APIs outlined in the documentation normal api with mtree_
and the advanced api with mas_.  The locking is outlined in each
section.

I guess a note stating to check your code with lockdep might be in order
in the documentation.

> >> and b) other in-tree users also just
> >> call mas_for_each() without either a lock held or RCU read side locked.
> > 
> > mas_for_each() must hold a lock of some type.
> 
> That's what I assumed. I missed that the rcu dereference check
> checks for an external lock too, which I guess you can register
> with the maple tree. Funky... I guess it's just for lockdep
> purposes, makes sense then.

The external lock is so that people can use the tree (which needs to
allocate) with sleeping locks and not just spinlocks.  Willy wants me to
kill it one day, though.

> 
> Presumably the current use case is fine, as it's serialized
> teardown. It just ends up triggering the rcu sanity checks.

On tear down, we should iterate through and free any resources then
destroy the tree.  If you erase each element one at a time you will
rebalance the tree - since it's a b-tree.

Thanks again for taking the time to tell me where/what you looked so I
can better answer the questions in the docs.  At least the LLM found it
for you.

Thanks,
Liam

Re: RCU warning off ublk_buf_cleanup() -> mas_for_each()

[Jens Axboe]

On 4/21/26 3:56 PM, Liam R. Howlett wrote:
> * Jens Axboe <axboe@kernel.dk> [260421 17:41]:
>> On 4/21/26 2:28 PM, Liam R. Howlett wrote:
>>> * Jens Axboe <axboe@kernel.dk> [260421 13:47]:
>>>> Hi Ming,
>>>>
>>>> Ran into the below running tests on the current tree:
>>>>
>>>> =============================
>>>> WARNING: suspicious RCU usage
>>>> 7.0.0+ #16 Tainted: G                 N 
>>>> -----------------------------
>>>> lib/maple_tree.c:759 suspicious rcu_dereference_check() usage!
>>>>
>>>> other info that might help us debug this:
>>>>
>>>>
>>>> rcu_scheduler_active = 2, debug_locks = 1
>>>> 1 lock held by iou-wrk-55535/55536:
>>>>  #0: ffff800085a451a0 (ublk_ctl_mutex){+.+.}-{4:4}, at: ublk_ctrl_del_dev+0xdc/0x2f8
>>>>
>>>> stack backtrace:
>>>> CPU: 4 UID: 0 PID: 55536 Comm: iou-wrk-55535 Tainted: G                 N  7.0.0+ #16 PREEMPT 
>>>> Tainted: [N]=TEST
>>>> Hardware name: linux,dummy-virt (DT)
>>>> Call trace:
>>>>  show_stack+0x1c/0x30 (C)
>>>>  dump_stack_lvl+0x68/0x90
>>>>  dump_stack+0x18/0x20
>>>>  lockdep_rcu_suspicious+0x170/0x200
>>>>  mas_walk+0x3f0/0x6a0
>>>>  mas_find+0x1b4/0x6b0
>>>>  ublk_buf_cleanup+0xe0/0x240
>>>>  ublk_cdev_rel+0x34/0x1b0
>>>>  device_release+0xa4/0x350
>>>>  kobject_put+0x138/0x250
>>>>  put_device+0x18/0x30
>>>>  ublk_put_device+0x18/0x28
>>>>  ublk_ctrl_del_dev+0x120/0x2f8
>>>>  ublk_ctrl_uring_cmd+0x598/0x29b8
>>>>  io_uring_cmd+0x1e0/0x468
>>>>  __io_issue_sqe+0xa4/0x748
>>>>  io_issue_sqe+0x80/0xf68
>>>>  io_wq_submit_work+0x26c/0xdc8
>>>>  io_worker_handle_work+0x334/0xf20
>>>>  io_wq_worker+0x278/0x9e8
>>>>  ret_from_fork+0x10/0x20
>>>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>>>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>>>>  ublkb0: unable to read partition table
>>>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>>>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>>>> Buffer I/O error on dev ublkb0, logical block 512, async page read
>>>> Buffer I/O error on dev ublkb0, logical block 512, async page read
>>>> Buffer I/O error on dev ublkb0, logical block 0, async page read
>>>> Buffer I/O error on dev ublkb0, logical block 512, async page read
>>>>
>>>> and I briefly looked at it, but then just gave up as a) the maple tree
>>>> documentation is not that detailed,
>>>
>>> Which documentation is lacking?  I will fix it.
>>>
>>> I have user documentation in the Documentation directory while
>>> technical details are in the code.
>>
>> I went into the core-api/ and leafed through that, didn't have anything
>> on mas_for_each() that pertained to locking. Was hoping I'd find a table
>> of which parts of the API requires what in terms of locking or RCU.
>> There arent a whole lot of in-kernel users of it yet, so looking at
>> other places in the kernel wasn't very useful.
> 
> There's several users and the test code.  Too bad none of them helped.

There are several, but that's not very much for a kernel API. I didn't
look at the test code. If I was writing the code I would dug deeper, but
this isn't what this is. I just found an issue and the bare basics of
poking at the API and docs to check my assumptions.

>> Since this is a merge window regression, I really just passed it to Ming
>> with you on the CC just in case, and didn't spend any more time on it.
>> I'm not the one that's supposed to be finding issues like this...
>>
> 
> I appreciate the response as I'll try to cater the doc to what users
> search for.

A great example is what the llist stuff does in terms of which parts of
the API needs what kind of exclusion against each other, if any. But
maybe not as appropriate here.

And obviously my use case may not be entirely representative in terms of
what a developer might normally want if they were writing new code and
looking to use it. Or convert existing code.

> There are two APIs outlined in the documentation normal api with mtree_
> and the advanced api with mas_.  The locking is outlined in each
> section.

If I was writing code using it, I'm sure it would've been fine. And yes
mas_find() lists it, but I'm just looking at a basic back trace and the
main user, which was mas_for_each().

> I guess a note stating to check your code with lockdep might be in order
> in the documentation.

That's always a good suggestion. And the rcu checking, as you'd probably
use both.

>>>> and b) other in-tree users also just
>>>> call mas_for_each() without either a lock held or RCU read side locked.
>>>
>>> mas_for_each() must hold a lock of some type.
>>
>> That's what I assumed. I missed that the rcu dereference check
>> checks for an external lock too, which I guess you can register
>> with the maple tree. Funky... I guess it's just for lockdep
>> purposes, makes sense then.
> 
> The external lock is so that people can use the tree (which needs to
> allocate) with sleeping locks and not just spinlocks.  Willy wants me to
> kill it one day, though.

Ah, so it's actually used. I'd have to agree with willy on that front.
Though I'm always annoyed at the forced locking that xarray imposes on
you, so...

>> Presumably the current use case is fine, as it's serialized
>> teardown. It just ends up triggering the rcu sanity checks.
> 
> On tear down, we should iterate through and free any resources then
> destroy the tree.  If you erase each element one at a time you will
> rebalance the tree - since it's a b-tree.
> 
> Thanks again for taking the time to tell me where/what you looked so I
> can better answer the questions in the docs.  At least the LLM found it
> for you.

LLM was just a way to save my time. I think I've spent longer writing
emails about it at this point ;-). Thanks for the quick replies, I'm
sure Ming will appreciate it and sort this out.

-- 
Jens Axboe