public inbox for io-uring@vger.kernel.org
 help / color / mirror / Atom feed
From: Jens Axboe <axboe@kernel.dk>
To: Diangang Li <lidiangang@bytedance.com>,
	Fengnan Chang <fengnanchang@gmail.com>,
	asml.silence@gmail.com, io-uring@vger.kernel.org
Cc: Fengnan Chang <changfengnan@bytedance.com>
Subject: Re: [RFC PATCH 2/2] io_uring: fix io may accumulation in poll mode
Date: Wed, 17 Dec 2025 09:25:59 -0700	[thread overview]
Message-ID: <9a8418d8-439f-4dd2-b3fe-33567129861e@kernel.dk> (raw)
In-Reply-To: <f2836fb8-9ad7-4277-948b-430dcd24d1b6@bytedance.com>

On 12/17/25 5:34 AM, Diangang Li wrote:
> Hi Jens,
> 
> We?ve identified one critical panic issue here.
> 
> [ 4504.422964] [  T63683] list_del corruption, ff2adc9b51d19a90->next is 
> LIST_POISON1 (dead000000000100)
> [ 4504.422994] [  T63683] ------------[ cut here ]------------
> [ 4504.422995] [  T63683] kernel BUG at lib/list_debug.c:56!
> [ 4504.423006] [  T63683] Oops: invalid opcode: 0000 [#1] SMP NOPTI
> [ 4504.423017] [  T63683] CPU: 38 UID: 0 PID: 63683 Comm: io_uring 
> Kdump: loaded Tainted: G S          E       6.19.0-rc1+ #1 
> PREEMPT(voluntary)
> [ 4504.423032] [  T63683] Tainted: [S]=CPU_OUT_OF_SPEC, [E]=UNSIGNED_MODULE
> [ 4504.423040] [  T63683] Hardware name: Inventec S520-A6/Nanping MLB, 
> BIOS 01.01.01.06.03 03/03/2023
> [ 4504.423050] [  T63683] RIP: 
> 0010:__list_del_entry_valid_or_report+0x94/0x100
> [ 4504.423064] [  T63683] Code: 89 fe 48 c7 c7 f0 78 87 b5 e8 38 07 ae 
> ff 0f 0b 48 89 ef e8 6e 40 cd ff 48 89 ea 48 89 de 48 c7 c7 20 79 87 b5 
> e8 1c 07 ae ff <0f> 0b 4c 89 e7 e8 52 40 cd ff 4c 89 e2 48 89 de 48 c7 
> c7 58 79 87
> [ 4504.423085] [  T63683] RSP: 0018:ff4efd9f3838fdb0 EFLAGS: 00010246
> [ 4504.423093] [  T63683] RAX: 000000000000004e RBX: ff2adc9b51d19a90 
> RCX: 0000000000000027
> [ 4504.423103] [  T63683] RDX: 0000000000000000 RSI: 0000000000000001 
> RDI: ff2add151cf99580
> [ 4504.423112] [  T63683] RBP: dead000000000100 R08: 0000000000000000 
> R09: 0000000000000003
> [ 4504.423120] [  T63683] R10: ff4efd9f3838fc60 R11: ff2add151cdfffe8 
> R12: dead000000000122
> [ 4504.423130] [  T63683] R13: ff2adc9b51d19a00 R14: 0000000000000000 
> R15: 0000000000000000
> [ 4504.423139] [  T63683] FS:  00007fae4f7ff6c0(0000) 
> GS:ff2add15665f5000(0000) knlGS:0000000000000000
> [ 4504.423148] [  T63683] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 4504.423157] [  T63683] CR2: 000055aa8afe5000 CR3: 00000083037ee006 
> CR4: 0000000000773ef0
> [ 4504.423166] [  T63683] PKRU: 55555554
> [ 4504.423171] [  T63683] Call Trace:
> [ 4504.423178] [  T63683]  <TASK>
> [ 4504.423184] [  T63683]  io_do_iopoll+0x298/0x330
> [ 4504.423193] [  T63683]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
> [ 4504.423204] [  T63683]  __do_sys_io_uring_enter+0x421/0x770
> [ 4504.423214] [  T63683]  do_syscall_64+0x67/0xf00
> [ 4504.423223] [  T63683]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 4504.423232] [  T63683] RIP: 0033:0x55aa707e99c3
> 
> It can be reproduced in three ways:
> - Running iopoll tests while switching the block scheduler
> - A split IO scenario in iopoll (e.g., bs=512k with max_sectors_kb=256k)
> - Multi poll queues with multi threads
> 
> All cases appear related to IO completions occurring outside the 
> io_do_iopoll() loop. The root cause remains unclear.

Ah I see what it is - we can get multiple completions on the iopoll
side, if you have multiple bio's per request. This didn't matter before
the patch that uses a lockless list to collect them, as it just marked
the request completed by writing to ->iopoll_complete and letting the
reaper find them. But it matters with the llist change, as then we're
adding the request to the llist more than once.


-- 
Jens Axboe

  reply	other threads:[~2025-12-17 16:26 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-12-10  8:54 [RFC PATCH 0/2] io_uring: fix io may accumulation in poll mode Fengnan Chang
2025-12-10  8:55 ` [RFC PATCH 1/2] blk-mq: delete task running check in blk_hctx_poll Fengnan Chang
2025-12-10  9:19   ` Jens Axboe
2025-12-10  9:53   ` Jens Axboe
2025-12-10  8:55 ` [RFC PATCH 2/2] io_uring: fix io may accumulation in poll mode Fengnan Chang
2025-12-11  2:15   ` Jens Axboe
2025-12-11  4:10     ` Jens Axboe
2025-12-11  7:38       ` Fengnan
2025-12-11 10:22         ` Jens Axboe
2025-12-11 10:33           ` Jens Axboe
2025-12-11 11:13             ` Fengnan Chang
2025-12-11 11:19               ` Jens Axboe
2025-12-12  1:41             ` Fengnan Chang
2025-12-12  1:53               ` Jens Axboe
2025-12-12  2:12                 ` Fengnan Chang
2025-12-12  5:11                   ` Jens Axboe
2025-12-12  8:58                     ` Jens Axboe
2025-12-12  9:49                       ` Fengnan Chang
2025-12-12 20:22                         ` Jens Axboe
2025-12-12 13:32                     ` Diangang Li
2025-12-12 20:09                       ` Jens Axboe
2025-12-15  6:25                         ` Diangang Li
2025-12-17 12:34                     ` Diangang Li
2025-12-17 16:25                       ` Jens Axboe [this message]
2025-12-19  5:43                         ` Diangang Li
2025-12-10  9:53 ` (subset) [RFC PATCH 0/2] " Jens Axboe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9a8418d8-439f-4dd2-b3fe-33567129861e@kernel.dk \
    --to=axboe@kernel.dk \
    --cc=asml.silence@gmail.com \
    --cc=changfengnan@bytedance.com \
    --cc=fengnanchang@gmail.com \
    --cc=io-uring@vger.kernel.org \
    --cc=lidiangang@bytedance.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox