[bug] kernel panic when running ublk selftest on next-20250321

[bug] kernel panic when running ublk selftest on next-20250321

[Ming Lei]

Hello,

When running ublk selftest on today's next tree, the following kernel
panic is triggered immediately:

```
cd tools/testing/selftests/ublk/
make
./test_loop_01.sh
```

And not see this issue on next-0317.


ktest-40 login: [   35.406086] BUG: unable to handle page fault for address: fffffdf1ea10bc88
[   35.407360] #PF: supervisor read access in kernel mode
[   35.408259] #PF: error_code(0x0000) - not-present page
[   35.409138] PGD 0 P4D 0 
[   35.409596] Oops: Oops: 0000 [#1] SMP NOPTI
[   35.410620] CPU: 15 UID: 0 PID: 1661 Comm: kublk Not tainted 6.14.0-rc7_blk_dev-next-20250321+ #149 PREEMPT(full) 
[   35.412617] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-1.fc39 04/01/2014
[   35.414075] RIP: 0010:kfree+0x6f/0x300
[   35.414780] Code: 80 48 01 d8 0f 82 a1 02 00 00 48 c7 c2 00 00 00 80 48 2b 15 2b af a6 01 48 01 d0 48 c1 e8 0c 481
[   35.417874] RSP: 0018:ffffd0bbc358b880 EFLAGS: 00010282
[   35.418800] RAX: fffffdf1ea10bc80 RBX: fffffc35442f2740 RCX: ffff8d0bcc5e35a0
[   35.420100] RDX: 000072f4c0000000 RSI: ffffffff96a9f61a RDI: fffffc35442f2740
[   35.421372] RBP: ffffd0bbc358b8c8 R08: 0000000000000001 R09: 0000000000000014
[   35.422603] R10: 0000000000000001 R11: 0000000000000000 R12: ffff8d0bd18b60c0
[   35.423887] R13: 0000000000000000 R14: 00000000ffffffff R15: ffff8d0bcc5e2290
[   35.425166] FS:  00007fea81c526c0(0000) GS:ffff8d109219b000(0000) knlGS:0000000000000000
[   35.426568] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   35.427574] CR2: fffffdf1ea10bc88 CR3: 00000001083b6005 CR4: 0000000000772ef0
[   35.428714] PKRU: 55555554
[   35.429171] Call Trace:
[   35.429665]  <TASK>
[   35.430046]  ? __die_body.cold+0x19/0x29
[   35.430734]  ? page_fault_oops+0xc9/0x290
[   35.431442]  ? search_module_extables+0x44/0x60
[   35.432181]  ? kfree+0x6f/0x300
[   35.432707]  ? search_bpf_extables+0x5f/0x80
[   35.433405]  ? exc_page_fault+0x189/0x1a0
[   35.434083]  ? asm_exc_page_fault+0x26/0x30
[   35.434742]  ? io_vec_free+0x1a/0x30
[   35.435375]  ? kfree+0x6f/0x300
[   35.435885]  ? psi_group_change+0x13b/0x310
[   35.436578]  io_vec_free+0x1a/0x30
[   35.437163]  io_req_uring_cleanup+0xa6/0xb0
[   35.437836]  io_uring_cmd_done+0xca/0x1d0
[   35.438498]  io_handle_tw_list+0xe8/0xf0
[   35.439154]  tctx_task_work_run+0x51/0xe0
[   35.439917]  tctx_task_work+0x3a/0x60
[   35.440537]  task_work_run+0x59/0x90
[   35.441150]  io_run_task_work+0x4c/0x110
[   35.441816]  io_cqring_wait+0x36b/0x660
[   35.442503]  ? __pfx_io_wake_function+0x10/0x10
[   35.443259]  __do_sys_io_uring_enter+0x4f1/0x750
[   35.444023]  do_syscall_64+0x82/0x170
[   35.444644]  ? syscall_exit_to_user_mode+0x1d5/0x210
[   35.445939]  ? do_syscall_64+0x8e/0x170
[   35.446997]  ? __memcg_slab_post_alloc_hook+0x1f4/0x370
[   35.448272]  ? __kmalloc_noprof+0x456/0x490
[   35.449295]  ? io_cache_alloc_new+0x19/0x80
[   35.450432]  ? io_cache_alloc_new+0x19/0x80
[   35.451506]  ? io_cache_alloc_new+0x19/0x80
[   35.452602]  ? io_rsrc_node_alloc+0x5f/0x70
[   35.453613]  ? io_sqe_files_register+0x204/0x2f0
[   35.454719]  ? __do_sys_io_uring_register+0x129/0xa70
[   35.455901]  ? syscall_exit_to_user_mode+0x1d5/0x210
[   35.457036]  ? syscall_exit_to_user_mode+0x10/0x210
[   35.458236]  ? do_syscall_64+0x8e/0x170
[   35.459226]  ? handle_mm_fault+0x1b8/0x2c0
[   35.460319]  ? do_user_addr_fault+0x211/0x680
[   35.461399]  ? clear_bhb_loop+0x35/0x90
[   35.462373]  ? clear_bhb_loop+0x35/0x90
[   35.463315]  ? clear_bhb_loop+0x35/0x90
[   35.464281]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[   35.465474] RIP: 0033:0x7fea8264b6fc
[   35.466422] Code: 0f b6 c0 48 8b 79 20 8b 3f 83 e7 01 44 0f 45 d0 41 83 ca 01 8b b9 cc 00 00 00 45 31 c0 41 b9 081
[   35.470085] RSP: 002b:00007fea81c51dc8 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
[   35.471641] RAX: ffffffffffffffda RBX: 000000003e33b4a8 RCX: 00007fea8264b6fc
[   35.473077] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 0000000000000000
[   35.474524] RBP: 0000000000000080 R08: 0000000000000000 R09: 0000000000000008
[   35.475925] R10: 0000000000000011 R11: 0000000000000246 R12: 000000003e33b4a8
[   35.477837] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000020
[   35.479455]  </TASK>
[   35.480262] Modules linked in: ip6table_filter iptable_filter isofs intel_rapl_msr intel_rapl_common kvm_intel vfg
[   35.488958] Dumping ftrace buffer:
[   35.490009]    (ftrace buffer empty)
[   35.491043] CR2: fffffdf1ea10bc88
[   35.491946] ---[ end trace 0000000000000000 ]---
[   35.493618] RIP: 0010:kfree+0x6f/0x300
[   35.494707] Code: 80 48 01 d8 0f 82 a1 02 00 00 48 c7 c2 00 00 00 80 48 2b 15 2b af a6 01 48 01 d0 48 c1 e8 0c 481
[   35.498709] RSP: 0018:ffffd0bbc358b880 EFLAGS: 00010282
[   35.500093] RAX: fffffdf1ea10bc80 RBX: fffffc35442f2740 RCX: ffff8d0bcc5e35a0
[   35.501798] RDX: 000072f4c0000000 RSI: ffffffff96a9f61a RDI: fffffc35442f2740
[   35.503907] RBP: ffffd0bbc358b8c8 R08: 0000000000000001 R09: 0000000000000014
[   35.505605] R10: 0000000000000001 R11: 0000000000000000 R12: ffff8d0bd18b60c0
[   35.507362] R13: 0000000000000000 R14: 00000000ffffffff R15: ffff8d0bcc5e2290
[   35.509073] FS:  00007fea81c526c0(0000) GS:ffff8d109219b000(0000) knlGS:0000000000000000
[   35.511178] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   35.512891] CR2: fffffdf1ea10bc88 CR3: 00000001083b6005 CR4: 0000000000772ef0
[   35.514582] PKRU: 55555554
[   35.515538] Kernel panic - not syncing: Fatal exception
[   35.517350] Dumping ftrace buffer:
[   35.518403]    (ftrace buffer empty)
[   35.519900] Kernel Offset: 0x15000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbff)
[   35.522222] ---[ end Kernel panic - not syncing: Fatal exception ]---



Thanks,
Ming

Re: [bug] kernel panic when running ublk selftest on next-20250321

[Pavel Begunkov]

On 3/21/25 14:33, Ming Lei wrote:
> Hello,
> 
> When running ublk selftest on today's next tree, the following kernel
> panic is triggered immediately:

Jens already stumbled on that one, it's likely because the
cached structure is not zeroed on alloc. I believe the
troubled patch is removed from the tree for now.

-- 
Pavel Begunkov

Re: [bug] kernel panic when running ublk selftest on next-20250321

[Jens Axboe]

On 3/21/25 8:40 AM, Pavel Begunkov wrote:
> On 3/21/25 14:33, Ming Lei wrote:
>> Hello,
>>
>> When running ublk selftest on today's next tree, the following kernel
>> panic is triggered immediately:
> 
> Jens already stumbled on that one, it's likely because the
> cached structure is not zeroed on alloc. I believe the
> troubled patch is removed from the tree for now.

Yep, ran into the same thing this morning, and yep the two last patches
in that series got dropped until this is resolved. Ming, if you use my
current for-next branch, it should all be good again.

-- 
Jens Axboe

Re: [bug] kernel panic when running ublk selftest on next-20250321

[Ming Lei]

On Fri, Mar 21, 2025 at 08:40:57AM -0600, Jens Axboe wrote:
> On 3/21/25 8:40 AM, Pavel Begunkov wrote:
> > On 3/21/25 14:33, Ming Lei wrote:
> >> Hello,
> >>
> >> When running ublk selftest on today's next tree, the following kernel
> >> panic is triggered immediately:
> > 
> > Jens already stumbled on that one, it's likely because the
> > cached structure is not zeroed on alloc. I believe the
> > troubled patch is removed from the tree for now.
> 
> Yep, ran into the same thing this morning, and yep the two last patches
> in that series got dropped until this is resolved. Ming, if you use my
> current for-next branch, it should all be good again.

Yeah, looks everything is good after reverting the following two patches:

Revert "io_uring/cmd: add iovec cache for commands"
Revert "io_uring/cmd: introduce io_uring_cmd_import_fixed_vec"

BTW I feel io_uring_cmd_import_fixed_vec is useful for using ublk
zc to write stacking target, plain readv/writev works just
fine without zc.




thanks,
Ming

Re: [bug] kernel panic when running ublk selftest on next-20250321

[Jens Axboe]

On 3/21/25 9:03 AM, Ming Lei wrote:
> On Fri, Mar 21, 2025 at 08:40:57AM -0600, Jens Axboe wrote:
>> On 3/21/25 8:40 AM, Pavel Begunkov wrote:
>>> On 3/21/25 14:33, Ming Lei wrote:
>>>> Hello,
>>>>
>>>> When running ublk selftest on today's next tree, the following kernel
>>>> panic is triggered immediately:
>>>
>>> Jens already stumbled on that one, it's likely because the
>>> cached structure is not zeroed on alloc. I believe the
>>> troubled patch is removed from the tree for now.
>>
>> Yep, ran into the same thing this morning, and yep the two last patches
>> in that series got dropped until this is resolved. Ming, if you use my
>> current for-next branch, it should all be good again.
> 
> Yeah, looks everything is good after reverting the following two patches:
> 
> Revert "io_uring/cmd: add iovec cache for commands"
> Revert "io_uring/cmd: introduce io_uring_cmd_import_fixed_vec"
> 
> BTW I feel io_uring_cmd_import_fixed_vec is useful for using ublk
> zc to write stacking target, plain readv/writev works just
> fine without zc.

I think Pavel is looking at the issue right now, so both may get
reintroduced for the 6.15 merge window. That said, we don't strictly
need the caching, we can always pile that on later. The vectored import
is going to be useful, agree, and can stand on its own.

-- 
Jens Axboe