public inbox for [email protected]
 help / color / mirror / Atom feed
* [syzbot] possible deadlock in io_disarm_next
@ 2022-04-20 10:00 syzbot
  2022-04-20 12:38 ` Pavel Begunkov
  0 siblings, 1 reply; 3+ messages in thread
From: syzbot @ 2022-04-20 10:00 UTC (permalink / raw)
  To: asml.silence, axboe, io-uring, linux-kernel, syzkaller-bugs

Hello,

syzbot found the following issue on:

HEAD commit:    634de1db0e9b Add linux-next specific files for 20220419
git tree:       linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=10c92db8f00000
kernel config:  https://syzkaller.appspot.com/x/.config?x=bbd6f9b0a89865b0
dashboard link: https://syzkaller.appspot.com/bug?extid=57e67273f92d7f5f1931
compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=10a02f68f00000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=138e3008f00000

The issue was bisected to:

commit 78bfbdd1a4977df1dded20f9783a6ec174e67ef8
Author: Pavel Begunkov <[email protected]>
Date:   Fri Apr 15 21:08:23 2022 +0000

    io_uring: kill io_put_req_deferred()

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=13039c0cf00000
final oops:     https://syzkaller.appspot.com/x/report.txt?x=10839c0cf00000
console output: https://syzkaller.appspot.com/x/log.txt?x=17039c0cf00000

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: [email protected]
Fixes: 78bfbdd1a497 ("io_uring: kill io_put_req_deferred()")

============================================
WARNING: possible recursive locking detected
5.18.0-rc3-next-20220419-syzkaller #0 Not tainted
--------------------------------------------
syz-executor162/3588 is trying to acquire lock:
ffff888011a453d8 (&ctx->timeout_lock){....}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:379 [inline]
ffff888011a453d8 (&ctx->timeout_lock){....}-{2:2}, at: io_disarm_next+0x545/0xaa0 fs/io_uring.c:2452

but task is already holding lock:
ffff888011a453d8 (&ctx->timeout_lock){....}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:379 [inline]
ffff888011a453d8 (&ctx->timeout_lock){....}-{2:2}, at: io_kill_timeouts+0x4c/0x227 fs/io_uring.c:10432

other info that might help us debug this:
 Possible unsafe locking scenario:

       CPU0
       ----
  lock(&ctx->timeout_lock);
  lock(&ctx->timeout_lock);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

2 locks held by syz-executor162/3588:
 #0: ffff888011a45398 (&ctx->completion_lock){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
 #0: ffff888011a45398 (&ctx->completion_lock){+.+.}-{2:2}, at: io_kill_timeouts+0x38/0x227 fs/io_uring.c:10431
 #1: ffff888011a453d8 (&ctx->timeout_lock){....}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:379 [inline]
 #1: ffff888011a453d8 (&ctx->timeout_lock){....}-{2:2}, at: io_kill_timeouts+0x4c/0x227 fs/io_uring.c:10432

stack backtrace:
CPU: 1 PID: 3588 Comm: syz-executor162 Not tainted 5.18.0-rc3-next-20220419-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 print_deadlock_bug kernel/locking/lockdep.c:2988 [inline]
 check_deadlock kernel/locking/lockdep.c:3031 [inline]
 validate_chain kernel/locking/lockdep.c:3816 [inline]
 __lock_acquire.cold+0x1f5/0x3b4 kernel/locking/lockdep.c:5053
 lock_acquire kernel/locking/lockdep.c:5665 [inline]
 lock_acquire+0x1ab/0x570 kernel/locking/lockdep.c:5630
 __raw_spin_lock_irq include/linux/spinlock_api_smp.h:119 [inline]
 _raw_spin_lock_irq+0x32/0x50 kernel/locking/spinlock.c:170
 spin_lock_irq include/linux/spinlock.h:379 [inline]
 io_disarm_next+0x545/0xaa0 fs/io_uring.c:2452
 __io_req_complete_post+0x794/0xd90 fs/io_uring.c:2200
 io_kill_timeout fs/io_uring.c:1815 [inline]
 io_kill_timeout+0x210/0x21d fs/io_uring.c:1803
 io_kill_timeouts+0xe2/0x227 fs/io_uring.c:10435
 io_ring_ctx_wait_and_kill+0x1eb/0x360 fs/io_uring.c:10462
 io_uring_release+0x42/0x46 fs/io_uring.c:10483
 __fput+0x277/0x9d0 fs/file_table.c:317
 task_work_run+0xdd/0x1a0 kernel/task_work.c:164
 exit_task_work include/linux/task_work.h:37 [inline]
 do_exit+0xaff/0x2a00 kernel/exit.c:796
 do_group_exit+0xd2/0x2f0 kernel/exit.c:926
 __do_sys_exit_group kernel/exit.c:937 [inline]
 __se_sys_exit_group kernel/exit.c:935 [inline]
 __x64_sys_exit_group+0x3a/0x50 kernel/exit.c:935
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f786cb8ccb9
Code: Unable to access opcode bytes at RIP 0x7f786cb8cc8f.
RSP: 002b:00007ffcf6b5b088 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 00007f786cc01350 RCX: 00007f786cb8ccb9
RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007f786cc01350
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001
 </TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at [email protected].

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
syzbot can test patches for this issue, for details see:
https://goo.gl/tpsmEJ#testing-patches

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [syzbot] possible deadlock in io_disarm_next
  2022-04-20 10:00 [syzbot] possible deadlock in io_disarm_next syzbot
@ 2022-04-20 12:38 ` Pavel Begunkov
  2022-04-20 13:18   ` syzbot
  0 siblings, 1 reply; 3+ messages in thread
From: Pavel Begunkov @ 2022-04-20 12:38 UTC (permalink / raw)
  To: syzbot, axboe, io-uring, linux-kernel, syzkaller-bugs

On 4/20/22 11:00, syzbot wrote:
> Hello,
> 
> syzbot found the following issue on:
> 
> HEAD commit:    634de1db0e9b Add linux-next specific files for 20220419
> git tree:       linux-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=10c92db8f00000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=bbd6f9b0a89865b0
> dashboard link: https://syzkaller.appspot.com/bug?extid=57e67273f92d7f5f1931
> compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=10a02f68f00000
> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=138e3008f00000
> 
> The issue was bisected to:
> 
> commit 78bfbdd1a4977df1dded20f9783a6ec174e67ef8
> Author: Pavel Begunkov <[email protected]>
> Date:   Fri Apr 15 21:08:23 2022 +0000
> 
>      io_uring: kill io_put_req_deferred()

#syz test: https://github.com/isilence/linux.git syz_timeout_deadlock



> bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=13039c0cf00000
> final oops:     https://syzkaller.appspot.com/x/report.txt?x=10839c0cf00000
> console output: https://syzkaller.appspot.com/x/log.txt?x=17039c0cf00000
> 
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: [email protected]
> Fixes: 78bfbdd1a497 ("io_uring: kill io_put_req_deferred()")
> 
> ============================================
> WARNING: possible recursive locking detected
> 5.18.0-rc3-next-20220419-syzkaller #0 Not tainted
> --------------------------------------------
> syz-executor162/3588 is trying to acquire lock:
> ffff888011a453d8 (&ctx->timeout_lock){....}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:379 [inline]
> ffff888011a453d8 (&ctx->timeout_lock){....}-{2:2}, at: io_disarm_next+0x545/0xaa0 fs/io_uring.c:2452
> 
> but task is already holding lock:
> ffff888011a453d8 (&ctx->timeout_lock){....}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:379 [inline]
> ffff888011a453d8 (&ctx->timeout_lock){....}-{2:2}, at: io_kill_timeouts+0x4c/0x227 fs/io_uring.c:10432
> 
> other info that might help us debug this:
>   Possible unsafe locking scenario:
> 
>         CPU0
>         ----
>    lock(&ctx->timeout_lock);
>    lock(&ctx->timeout_lock);
> 
>   *** DEADLOCK ***
> 
>   May be due to missing lock nesting notation
> 
> 2 locks held by syz-executor162/3588:
>   #0: ffff888011a45398 (&ctx->completion_lock){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
>   #0: ffff888011a45398 (&ctx->completion_lock){+.+.}-{2:2}, at: io_kill_timeouts+0x38/0x227 fs/io_uring.c:10431
>   #1: ffff888011a453d8 (&ctx->timeout_lock){....}-{2:2}, at: spin_lock_irq include/linux/spinlock.h:379 [inline]
>   #1: ffff888011a453d8 (&ctx->timeout_lock){....}-{2:2}, at: io_kill_timeouts+0x4c/0x227 fs/io_uring.c:10432
> 
> stack backtrace:
> CPU: 1 PID: 3588 Comm: syz-executor162 Not tainted 5.18.0-rc3-next-20220419-syzkaller #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> Call Trace:
>   <TASK>
>   __dump_stack lib/dump_stack.c:88 [inline]
>   dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
>   print_deadlock_bug kernel/locking/lockdep.c:2988 [inline]
>   check_deadlock kernel/locking/lockdep.c:3031 [inline]
>   validate_chain kernel/locking/lockdep.c:3816 [inline]
>   __lock_acquire.cold+0x1f5/0x3b4 kernel/locking/lockdep.c:5053
>   lock_acquire kernel/locking/lockdep.c:5665 [inline]
>   lock_acquire+0x1ab/0x570 kernel/locking/lockdep.c:5630
>   __raw_spin_lock_irq include/linux/spinlock_api_smp.h:119 [inline]
>   _raw_spin_lock_irq+0x32/0x50 kernel/locking/spinlock.c:170
>   spin_lock_irq include/linux/spinlock.h:379 [inline]
>   io_disarm_next+0x545/0xaa0 fs/io_uring.c:2452
>   __io_req_complete_post+0x794/0xd90 fs/io_uring.c:2200
>   io_kill_timeout fs/io_uring.c:1815 [inline]
>   io_kill_timeout+0x210/0x21d fs/io_uring.c:1803
>   io_kill_timeouts+0xe2/0x227 fs/io_uring.c:10435
>   io_ring_ctx_wait_and_kill+0x1eb/0x360 fs/io_uring.c:10462
>   io_uring_release+0x42/0x46 fs/io_uring.c:10483
>   __fput+0x277/0x9d0 fs/file_table.c:317
>   task_work_run+0xdd/0x1a0 kernel/task_work.c:164
>   exit_task_work include/linux/task_work.h:37 [inline]
>   do_exit+0xaff/0x2a00 kernel/exit.c:796
>   do_group_exit+0xd2/0x2f0 kernel/exit.c:926
>   __do_sys_exit_group kernel/exit.c:937 [inline]
>   __se_sys_exit_group kernel/exit.c:935 [inline]
>   __x64_sys_exit_group+0x3a/0x50 kernel/exit.c:935
>   do_syscall_x64 arch/x86/entry/common.c:50 [inline]
>   do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
>   entry_SYSCALL_64_after_hwframe+0x44/0xae
> RIP: 0033:0x7f786cb8ccb9
> Code: Unable to access opcode bytes at RIP 0x7f786cb8cc8f.
> RSP: 002b:00007ffcf6b5b088 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
> RAX: ffffffffffffffda RBX: 00007f786cc01350 RCX: 00007f786cb8ccb9
> RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
> RBP: 0000000000000000 R08: ffffffffffffffc0 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 00007f786cc01350
> R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001
>   </TASK>
> 
> 
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at [email protected].
> 
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> For information about bisection process see: https://goo.gl/tpsmEJ#bisection
> syzbot can test patches for this issue, for details see:
> https://goo.gl/tpsmEJ#testing-patches

-- 
Pavel Begunkov

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [syzbot] possible deadlock in io_disarm_next
  2022-04-20 12:38 ` Pavel Begunkov
@ 2022-04-20 13:18   ` syzbot
  0 siblings, 0 replies; 3+ messages in thread
From: syzbot @ 2022-04-20 13:18 UTC (permalink / raw)
  To: asml.silence, axboe, io-uring, linux-kernel, syzkaller-bugs

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: [email protected]

Tested on:

commit:         0f00d115 io_uring: refactor io_disarm_next() locking
git tree:       https://github.com/isilence/linux.git syz_timeout_deadlock
kernel config:  https://syzkaller.appspot.com/x/.config?x=c01066a8395ef6d7
dashboard link: https://syzkaller.appspot.com/bug?extid=57e67273f92d7f5f1931
compiler:       gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2

Note: no patches were applied.
Note: testing is done by a robot and is best-effort only.

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2022-04-20 13:18 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-04-20 10:00 [syzbot] possible deadlock in io_disarm_next syzbot
2022-04-20 12:38 ` Pavel Begunkov
2022-04-20 13:18   ` syzbot

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox