From: Jens Axboe <axboe@kernel.dk>
To: Diangang Li <lidiangang@bytedance.com>,
Fengnan Chang <fengnanchang@gmail.com>,
asml.silence@gmail.com, io-uring@vger.kernel.org
Cc: Fengnan Chang <changfengnan@bytedance.com>
Subject: Re: [RFC PATCH 2/2] io_uring: fix io may accumulation in poll mode
Date: Wed, 17 Dec 2025 09:25:59 -0700 [thread overview]
Message-ID: <9a8418d8-439f-4dd2-b3fe-33567129861e@kernel.dk> (raw)
In-Reply-To: <f2836fb8-9ad7-4277-948b-430dcd24d1b6@bytedance.com>
On 12/17/25 5:34 AM, Diangang Li wrote:
> Hi Jens,
>
> We?ve identified one critical panic issue here.
>
> [ 4504.422964] [ T63683] list_del corruption, ff2adc9b51d19a90->next is
> LIST_POISON1 (dead000000000100)
> [ 4504.422994] [ T63683] ------------[ cut here ]------------
> [ 4504.422995] [ T63683] kernel BUG at lib/list_debug.c:56!
> [ 4504.423006] [ T63683] Oops: invalid opcode: 0000 [#1] SMP NOPTI
> [ 4504.423017] [ T63683] CPU: 38 UID: 0 PID: 63683 Comm: io_uring
> Kdump: loaded Tainted: G S E 6.19.0-rc1+ #1
> PREEMPT(voluntary)
> [ 4504.423032] [ T63683] Tainted: [S]=CPU_OUT_OF_SPEC, [E]=UNSIGNED_MODULE
> [ 4504.423040] [ T63683] Hardware name: Inventec S520-A6/Nanping MLB,
> BIOS 01.01.01.06.03 03/03/2023
> [ 4504.423050] [ T63683] RIP:
> 0010:__list_del_entry_valid_or_report+0x94/0x100
> [ 4504.423064] [ T63683] Code: 89 fe 48 c7 c7 f0 78 87 b5 e8 38 07 ae
> ff 0f 0b 48 89 ef e8 6e 40 cd ff 48 89 ea 48 89 de 48 c7 c7 20 79 87 b5
> e8 1c 07 ae ff <0f> 0b 4c 89 e7 e8 52 40 cd ff 4c 89 e2 48 89 de 48 c7
> c7 58 79 87
> [ 4504.423085] [ T63683] RSP: 0018:ff4efd9f3838fdb0 EFLAGS: 00010246
> [ 4504.423093] [ T63683] RAX: 000000000000004e RBX: ff2adc9b51d19a90
> RCX: 0000000000000027
> [ 4504.423103] [ T63683] RDX: 0000000000000000 RSI: 0000000000000001
> RDI: ff2add151cf99580
> [ 4504.423112] [ T63683] RBP: dead000000000100 R08: 0000000000000000
> R09: 0000000000000003
> [ 4504.423120] [ T63683] R10: ff4efd9f3838fc60 R11: ff2add151cdfffe8
> R12: dead000000000122
> [ 4504.423130] [ T63683] R13: ff2adc9b51d19a00 R14: 0000000000000000
> R15: 0000000000000000
> [ 4504.423139] [ T63683] FS: 00007fae4f7ff6c0(0000)
> GS:ff2add15665f5000(0000) knlGS:0000000000000000
> [ 4504.423148] [ T63683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 4504.423157] [ T63683] CR2: 000055aa8afe5000 CR3: 00000083037ee006
> CR4: 0000000000773ef0
> [ 4504.423166] [ T63683] PKRU: 55555554
> [ 4504.423171] [ T63683] Call Trace:
> [ 4504.423178] [ T63683] <TASK>
> [ 4504.423184] [ T63683] io_do_iopoll+0x298/0x330
> [ 4504.423193] [ T63683] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
> [ 4504.423204] [ T63683] __do_sys_io_uring_enter+0x421/0x770
> [ 4504.423214] [ T63683] do_syscall_64+0x67/0xf00
> [ 4504.423223] [ T63683] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 4504.423232] [ T63683] RIP: 0033:0x55aa707e99c3
>
> It can be reproduced in three ways:
> - Running iopoll tests while switching the block scheduler
> - A split IO scenario in iopoll (e.g., bs=512k with max_sectors_kb=256k)
> - Multi poll queues with multi threads
>
> All cases appear related to IO completions occurring outside the
> io_do_iopoll() loop. The root cause remains unclear.
Ah I see what it is - we can get multiple completions on the iopoll
side, if you have multiple bio's per request. This didn't matter before
the patch that uses a lockless list to collect them, as it just marked
the request completed by writing to ->iopoll_complete and letting the
reaper find them. But it matters with the llist change, as then we're
adding the request to the llist more than once.
--
Jens Axboe
next prev parent reply other threads:[~2025-12-17 16:26 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-12-10 8:54 [RFC PATCH 0/2] io_uring: fix io may accumulation in poll mode Fengnan Chang
2025-12-10 8:55 ` [RFC PATCH 1/2] blk-mq: delete task running check in blk_hctx_poll Fengnan Chang
2025-12-10 9:19 ` Jens Axboe
2025-12-10 9:53 ` Jens Axboe
2025-12-10 8:55 ` [RFC PATCH 2/2] io_uring: fix io may accumulation in poll mode Fengnan Chang
2025-12-11 2:15 ` Jens Axboe
2025-12-11 4:10 ` Jens Axboe
2025-12-11 7:38 ` Fengnan
2025-12-11 10:22 ` Jens Axboe
2025-12-11 10:33 ` Jens Axboe
2025-12-11 11:13 ` Fengnan Chang
2025-12-11 11:19 ` Jens Axboe
2025-12-12 1:41 ` Fengnan Chang
2025-12-12 1:53 ` Jens Axboe
2025-12-12 2:12 ` Fengnan Chang
2025-12-12 5:11 ` Jens Axboe
2025-12-12 8:58 ` Jens Axboe
2025-12-12 9:49 ` Fengnan Chang
2025-12-12 20:22 ` Jens Axboe
2025-12-12 13:32 ` Diangang Li
2025-12-12 20:09 ` Jens Axboe
2025-12-15 6:25 ` Diangang Li
2025-12-17 12:34 ` Diangang Li
2025-12-17 16:25 ` Jens Axboe [this message]
2025-12-19 5:43 ` Diangang Li
2025-12-10 9:53 ` (subset) [RFC PATCH 0/2] " Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9a8418d8-439f-4dd2-b3fe-33567129861e@kernel.dk \
--to=axboe@kernel.dk \
--cc=asml.silence@gmail.com \
--cc=changfengnan@bytedance.com \
--cc=fengnanchang@gmail.com \
--cc=io-uring@vger.kernel.org \
--cc=lidiangang@bytedance.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox