public inbox for [email protected]
 help / color / mirror / Atom feed
* possible deadlock in io_link_timeout_fn
@ 2021-02-23 13:35 syzbot
  2021-02-23 13:50 ` Pavel Begunkov
  0 siblings, 1 reply; 2+ messages in thread
From: syzbot @ 2021-02-23 13:35 UTC (permalink / raw)
  To: asml.silence, axboe, io-uring, linux-kernel, syzkaller-bugs

Hello,

syzbot found the following issue on:

HEAD commit:    31caf8b2 Merge branch 'linus' of git://git.kernel.org/pub/..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=11c64f12d00000
kernel config:  https://syzkaller.appspot.com/x/.config?x=5a8f3a57fabb4015
dashboard link: https://syzkaller.appspot.com/bug?extid=9a512c5bdc15635eab70

Unfortunately, I don't have any reproducer for this issue yet.

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: [email protected]

=====================================================
WARNING: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected
5.11.0-syzkaller #0 Not tainted
-----------------------------------------------------
syz-executor.0/12185 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
ffff888013db4820 (&fs->lock){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
ffff888013db4820 (&fs->lock){+.+.}-{2:2}, at: io_req_clean_work fs/io_uring.c:1405 [inline]
ffff888013db4820 (&fs->lock){+.+.}-{2:2}, at: io_dismantle_req+0x90f/0xf90 fs/io_uring.c:2051

and this task is already holding:
ffff88806bea6718 (&ctx->completion_lock){-...}-{2:2}, at: io_req_complete_post+0x4e/0x920 fs/io_uring.c:1923
which would create a new lock dependency:
 (&ctx->completion_lock){-...}-{2:2} -> (&fs->lock){+.+.}-{2:2}

but this new dependency connects a HARDIRQ-irq-safe lock:
 (&ctx->completion_lock){-...}-{2:2}

... which became HARDIRQ-irq-safe at:
  lock_acquire kernel/locking/lockdep.c:5510 [inline]
  lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
  __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
  _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:159
  io_link_timeout_fn+0xbf/0x720 fs/io_uring.c:6495
  __run_hrtimer kernel/time/hrtimer.c:1519 [inline]
  __hrtimer_run_queues+0x609/0xe40 kernel/time/hrtimer.c:1583
  hrtimer_interrupt+0x334/0x940 kernel/time/hrtimer.c:1645
  local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
  __sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1106
  asm_call_irq_on_stack+0xf/0x20
  __run_sysvec_on_irqstack arch/x86/include/asm/irq_stack.h:37 [inline]
  run_sysvec_on_irqstack_cond arch/x86/include/asm/irq_stack.h:89 [inline]
  sysvec_apic_timer_interrupt+0xbd/0x100 arch/x86/kernel/apic/apic.c:1100
  asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:635
  __raw_spin_unlock_irq include/linux/spinlock_api_smp.h:169 [inline]
  _raw_spin_unlock_irq+0x25/0x40 kernel/locking/spinlock.c:199
  spin_unlock_irq include/linux/spinlock.h:404 [inline]
  io_queue_linked_timeout+0x194/0x1f0 fs/io_uring.c:6541
  __io_queue_sqe+0x32f/0xdb0 fs/io_uring.c:6607
  __io_req_task_submit+0x18e/0x240 fs/io_uring.c:2344
  __tctx_task_work fs/io_uring.c:2204 [inline]
  tctx_task_work+0x12b/0x330 fs/io_uring.c:2230
  task_work_run+0xdd/0x1a0 kernel/task_work.c:140
  tracehook_notify_signal include/linux/tracehook.h:212 [inline]
  handle_signal_work kernel/entry/common.c:145 [inline]
  exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
  exit_to_user_mode_prepare+0x221/0x250 kernel/entry/common.c:208
  __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
  syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
  entry_SYSCALL_64_after_hwframe+0x44/0xae

to a HARDIRQ-irq-unsafe lock:
 (&fs->lock){+.+.}-{2:2}

... which became HARDIRQ-irq-unsafe at:
...
  lock_acquire kernel/locking/lockdep.c:5510 [inline]
  lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
  __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
  _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
  spin_lock include/linux/spinlock.h:354 [inline]
  set_fs_pwd+0x85/0x2a0 fs/fs_struct.c:39
  init_chdir+0x106/0x14e fs/init.c:54
  devtmpfs_setup drivers/base/devtmpfs.c:415 [inline]
  devtmpfsd+0x76/0x333 drivers/base/devtmpfs.c:430
  kthread+0x3b1/0x4a0 kernel/kthread.c:292
  ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294

other info that might help us debug this:

 Possible interrupt unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&fs->lock);
                               local_irq_disable();
                               lock(&ctx->completion_lock);
                               lock(&fs->lock);
  <Interrupt>
    lock(&ctx->completion_lock);

 *** DEADLOCK ***

1 lock held by syz-executor.0/12185:
 #0: ffff88806bea6718 (&ctx->completion_lock){-...}-{2:2}, at: io_req_complete_post+0x4e/0x920 fs/io_uring.c:1923

the dependencies between HARDIRQ-irq-safe lock and the holding lock:
-> (&ctx->completion_lock){-...}-{2:2} {
   IN-HARDIRQ-W at:
                    lock_acquire kernel/locking/lockdep.c:5510 [inline]
                    lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
                    __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
                    _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:159
                    io_link_timeout_fn+0xbf/0x720 fs/io_uring.c:6495
                    __run_hrtimer kernel/time/hrtimer.c:1519 [inline]
                    __hrtimer_run_queues+0x609/0xe40 kernel/time/hrtimer.c:1583
                    hrtimer_interrupt+0x334/0x940 kernel/time/hrtimer.c:1645
                    local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
                    __sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1106
                    asm_call_irq_on_stack+0xf/0x20
                    __run_sysvec_on_irqstack arch/x86/include/asm/irq_stack.h:37 [inline]
                    run_sysvec_on_irqstack_cond arch/x86/include/asm/irq_stack.h:89 [inline]
                    sysvec_apic_timer_interrupt+0xbd/0x100 arch/x86/kernel/apic/apic.c:1100
                    asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:635
                    __raw_spin_unlock_irq include/linux/spinlock_api_smp.h:169 [inline]
                    _raw_spin_unlock_irq+0x25/0x40 kernel/locking/spinlock.c:199
                    spin_unlock_irq include/linux/spinlock.h:404 [inline]
                    io_queue_linked_timeout+0x194/0x1f0 fs/io_uring.c:6541
                    __io_queue_sqe+0x32f/0xdb0 fs/io_uring.c:6607
                    __io_req_task_submit+0x18e/0x240 fs/io_uring.c:2344
                    __tctx_task_work fs/io_uring.c:2204 [inline]
                    tctx_task_work+0x12b/0x330 fs/io_uring.c:2230
                    task_work_run+0xdd/0x1a0 kernel/task_work.c:140
                    tracehook_notify_signal include/linux/tracehook.h:212 [inline]
                    handle_signal_work kernel/entry/common.c:145 [inline]
                    exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
                    exit_to_user_mode_prepare+0x221/0x250 kernel/entry/common.c:208
                    __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
                    syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
                    entry_SYSCALL_64_after_hwframe+0x44/0xae
   INITIAL USE at:
                   lock_acquire kernel/locking/lockdep.c:5510 [inline]
                   lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
                   __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
                   _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:159
                   io_req_complete_post+0x4e/0x920 fs/io_uring.c:1923
                   __io_req_complete fs/io_uring.c:1963 [inline]
                   io_req_complete fs/io_uring.c:1968 [inline]
                   io_queue_sqe+0xa3b/0xfa0 fs/io_uring.c:6620
                   io_submit_sqe fs/io_uring.c:6707 [inline]
                   io_submit_sqes+0x15f5/0x2b60 fs/io_uring.c:6939
                   __do_sys_io_uring_enter+0x1154/0x1f50 fs/io_uring.c:9454
                   do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
                   entry_SYSCALL_64_after_hwframe+0x44/0xae
 }
 ... key      at: [<ffffffff8fe69a80>] __key.9+0x0/0x40
 ... acquired at:
   lock_acquire kernel/locking/lockdep.c:5510 [inline]
   lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
   __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
   _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
   spin_lock include/linux/spinlock.h:354 [inline]
   io_req_clean_work fs/io_uring.c:1405 [inline]
   io_dismantle_req+0x90f/0xf90 fs/io_uring.c:2051
   io_req_complete_post+0xf6/0x920 fs/io_uring.c:1933
   __io_req_complete fs/io_uring.c:1963 [inline]
   io_req_complete fs/io_uring.c:1968 [inline]
   io_cancel_defer_files fs/io_uring.c:8945 [inline]
   io_uring_cancel_task_requests+0x67e/0xea0 fs/io_uring.c:9052
   __io_uring_files_cancel+0x110/0x1b0 fs/io_uring.c:9140
   io_uring_files_cancel include/linux/io_uring.h:65 [inline]
   do_exit+0x2fe/0x2ae0 kernel/exit.c:780
   do_group_exit+0x125/0x310 kernel/exit.c:922
   get_signal+0x42c/0x2100 kernel/signal.c:2773
   arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
   handle_signal_work kernel/entry/common.c:147 [inline]
   exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
   exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:208
   __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
   syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
   entry_SYSCALL_64_after_hwframe+0x44/0xae


the dependencies between the lock to be acquired
 and HARDIRQ-irq-unsafe lock:
-> (&fs->lock){+.+.}-{2:2} {
   HARDIRQ-ON-W at:
                    lock_acquire kernel/locking/lockdep.c:5510 [inline]
                    lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
                    __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
                    _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
                    spin_lock include/linux/spinlock.h:354 [inline]
                    set_fs_pwd+0x85/0x2a0 fs/fs_struct.c:39
                    init_chdir+0x106/0x14e fs/init.c:54
                    devtmpfs_setup drivers/base/devtmpfs.c:415 [inline]
                    devtmpfsd+0x76/0x333 drivers/base/devtmpfs.c:430
                    kthread+0x3b1/0x4a0 kernel/kthread.c:292
                    ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
   SOFTIRQ-ON-W at:
                    lock_acquire kernel/locking/lockdep.c:5510 [inline]
                    lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
                    __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
                    _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
                    spin_lock include/linux/spinlock.h:354 [inline]
                    set_fs_pwd+0x85/0x2a0 fs/fs_struct.c:39
                    init_chdir+0x106/0x14e fs/init.c:54
                    devtmpfs_setup drivers/base/devtmpfs.c:415 [inline]
                    devtmpfsd+0x76/0x333 drivers/base/devtmpfs.c:430
                    kthread+0x3b1/0x4a0 kernel/kthread.c:292
                    ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
   INITIAL USE at:
                   lock_acquire kernel/locking/lockdep.c:5510 [inline]
                   lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
                   __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
                   _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
                   spin_lock include/linux/spinlock.h:354 [inline]
                   set_fs_pwd+0x85/0x2a0 fs/fs_struct.c:39
                   init_chdir+0x106/0x14e fs/init.c:54
                   devtmpfs_setup drivers/base/devtmpfs.c:415 [inline]
                   devtmpfsd+0x76/0x333 drivers/base/devtmpfs.c:430
                   kthread+0x3b1/0x4a0 kernel/kthread.c:292
                   ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
 }
 ... key      at: [<ffffffff8fe68260>] __key.1+0x0/0x40
 ... acquired at:
   lock_acquire kernel/locking/lockdep.c:5510 [inline]
   lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
   __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
   _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
   spin_lock include/linux/spinlock.h:354 [inline]
   io_req_clean_work fs/io_uring.c:1405 [inline]
   io_dismantle_req+0x90f/0xf90 fs/io_uring.c:2051
   io_req_complete_post+0xf6/0x920 fs/io_uring.c:1933
   __io_req_complete fs/io_uring.c:1963 [inline]
   io_req_complete fs/io_uring.c:1968 [inline]
   io_cancel_defer_files fs/io_uring.c:8945 [inline]
   io_uring_cancel_task_requests+0x67e/0xea0 fs/io_uring.c:9052
   __io_uring_files_cancel+0x110/0x1b0 fs/io_uring.c:9140
   io_uring_files_cancel include/linux/io_uring.h:65 [inline]
   do_exit+0x2fe/0x2ae0 kernel/exit.c:780
   do_group_exit+0x125/0x310 kernel/exit.c:922
   get_signal+0x42c/0x2100 kernel/signal.c:2773
   arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
   handle_signal_work kernel/entry/common.c:147 [inline]
   exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
   exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:208
   __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
   syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
   entry_SYSCALL_64_after_hwframe+0x44/0xae


stack backtrace:
CPU: 3 PID: 12185 Comm: syz-executor.0 Not tainted 5.11.0-syzkaller #0
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-2 04/01/2014
Call Trace:
 __dump_stack lib/dump_stack.c:79 [inline]
 dump_stack+0xfa/0x151 lib/dump_stack.c:120
 print_bad_irq_dependency kernel/locking/lockdep.c:2460 [inline]
 check_irq_usage.cold+0x50d/0x744 kernel/locking/lockdep.c:2689
 check_prev_add kernel/locking/lockdep.c:2940 [inline]
 check_prevs_add kernel/locking/lockdep.c:3059 [inline]
 validate_chain kernel/locking/lockdep.c:3674 [inline]
 __lock_acquire+0x2b2c/0x54c0 kernel/locking/lockdep.c:4900
 lock_acquire kernel/locking/lockdep.c:5510 [inline]
 lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
 spin_lock include/linux/spinlock.h:354 [inline]
 io_req_clean_work fs/io_uring.c:1405 [inline]
 io_dismantle_req+0x90f/0xf90 fs/io_uring.c:2051
 io_req_complete_post+0xf6/0x920 fs/io_uring.c:1933
 __io_req_complete fs/io_uring.c:1963 [inline]
 io_req_complete fs/io_uring.c:1968 [inline]
 io_cancel_defer_files fs/io_uring.c:8945 [inline]
 io_uring_cancel_task_requests+0x67e/0xea0 fs/io_uring.c:9052
 __io_uring_files_cancel+0x110/0x1b0 fs/io_uring.c:9140
 io_uring_files_cancel include/linux/io_uring.h:65 [inline]
 do_exit+0x2fe/0x2ae0 kernel/exit.c:780
 do_group_exit+0x125/0x310 kernel/exit.c:922
 get_signal+0x42c/0x2100 kernel/signal.c:2773
 arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
 handle_signal_work kernel/entry/common.c:147 [inline]
 exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
 exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:208
 __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
 syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x465ef9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffb56aa0218 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: 0000000000000000 RBX: 000000000056bf68 RCX: 0000000000465ef9
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 000000000056bf68
RBP: 000000000056bf60 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056bf6c
R13: 00007fff198147ff R14: 00007ffb56aa0300 R15: 0000000000022000


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at [email protected].

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: possible deadlock in io_link_timeout_fn
  2021-02-23 13:35 possible deadlock in io_link_timeout_fn syzbot
@ 2021-02-23 13:50 ` Pavel Begunkov
  0 siblings, 0 replies; 2+ messages in thread
From: Pavel Begunkov @ 2021-02-23 13:50 UTC (permalink / raw)
  To: syzbot, axboe, io-uring, linux-kernel, syzkaller-bugs

On 23/02/2021 13:35, syzbot wrote:
> Hello,
> 
> syzbot found the following issue on:
> 
> HEAD commit:    31caf8b2 Merge branch 'linus' of git://git.kernel.org/pub/..
> git tree:       upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=11c64f12d00000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=5a8f3a57fabb4015
> dashboard link: https://syzkaller.appspot.com/bug?extid=9a512c5bdc15635eab70
> 
> Unfortunately, I don't have any reproducer for this issue yet.
> 
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: [email protected]

Ok, that IRQ recycling stuff kicks in. I guess we can take
work_clean out of spin, I'll try it out this evening.

Either, Jen's io-wq patches go 5.12 in the end, and it would probably be
solved naturally.

> 
> =====================================================
> WARNING: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected
> 5.11.0-syzkaller #0 Not tainted
> -----------------------------------------------------
> syz-executor.0/12185 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
> ffff888013db4820 (&fs->lock){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
> ffff888013db4820 (&fs->lock){+.+.}-{2:2}, at: io_req_clean_work fs/io_uring.c:1405 [inline]
> ffff888013db4820 (&fs->lock){+.+.}-{2:2}, at: io_dismantle_req+0x90f/0xf90 fs/io_uring.c:2051
> 
> and this task is already holding:
> ffff88806bea6718 (&ctx->completion_lock){-...}-{2:2}, at: io_req_complete_post+0x4e/0x920 fs/io_uring.c:1923
> which would create a new lock dependency:
>  (&ctx->completion_lock){-...}-{2:2} -> (&fs->lock){+.+.}-{2:2}
> 
> but this new dependency connects a HARDIRQ-irq-safe lock:
>  (&ctx->completion_lock){-...}-{2:2}
> 
> ... which became HARDIRQ-irq-safe at:
>   lock_acquire kernel/locking/lockdep.c:5510 [inline]
>   lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
>   __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
>   _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:159
>   io_link_timeout_fn+0xbf/0x720 fs/io_uring.c:6495
>   __run_hrtimer kernel/time/hrtimer.c:1519 [inline]
>   __hrtimer_run_queues+0x609/0xe40 kernel/time/hrtimer.c:1583
>   hrtimer_interrupt+0x334/0x940 kernel/time/hrtimer.c:1645
>   local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
>   __sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1106
>   asm_call_irq_on_stack+0xf/0x20
>   __run_sysvec_on_irqstack arch/x86/include/asm/irq_stack.h:37 [inline]
>   run_sysvec_on_irqstack_cond arch/x86/include/asm/irq_stack.h:89 [inline]
>   sysvec_apic_timer_interrupt+0xbd/0x100 arch/x86/kernel/apic/apic.c:1100
>   asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:635
>   __raw_spin_unlock_irq include/linux/spinlock_api_smp.h:169 [inline]
>   _raw_spin_unlock_irq+0x25/0x40 kernel/locking/spinlock.c:199
>   spin_unlock_irq include/linux/spinlock.h:404 [inline]
>   io_queue_linked_timeout+0x194/0x1f0 fs/io_uring.c:6541
>   __io_queue_sqe+0x32f/0xdb0 fs/io_uring.c:6607
>   __io_req_task_submit+0x18e/0x240 fs/io_uring.c:2344
>   __tctx_task_work fs/io_uring.c:2204 [inline]
>   tctx_task_work+0x12b/0x330 fs/io_uring.c:2230
>   task_work_run+0xdd/0x1a0 kernel/task_work.c:140
>   tracehook_notify_signal include/linux/tracehook.h:212 [inline]
>   handle_signal_work kernel/entry/common.c:145 [inline]
>   exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
>   exit_to_user_mode_prepare+0x221/0x250 kernel/entry/common.c:208
>   __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
>   syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
>   entry_SYSCALL_64_after_hwframe+0x44/0xae
> 
> to a HARDIRQ-irq-unsafe lock:
>  (&fs->lock){+.+.}-{2:2}
> 
> ... which became HARDIRQ-irq-unsafe at:
> ...
>   lock_acquire kernel/locking/lockdep.c:5510 [inline]
>   lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
>   __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
>   _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
>   spin_lock include/linux/spinlock.h:354 [inline]
>   set_fs_pwd+0x85/0x2a0 fs/fs_struct.c:39
>   init_chdir+0x106/0x14e fs/init.c:54
>   devtmpfs_setup drivers/base/devtmpfs.c:415 [inline]
>   devtmpfsd+0x76/0x333 drivers/base/devtmpfs.c:430
>   kthread+0x3b1/0x4a0 kernel/kthread.c:292
>   ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
> 
> other info that might help us debug this:
> 
>  Possible interrupt unsafe locking scenario:
> 
>        CPU0                    CPU1
>        ----                    ----
>   lock(&fs->lock);
>                                local_irq_disable();
>                                lock(&ctx->completion_lock);
>                                lock(&fs->lock);
>   <Interrupt>
>     lock(&ctx->completion_lock);
> 
>  *** DEADLOCK ***
> 
> 1 lock held by syz-executor.0/12185:
>  #0: ffff88806bea6718 (&ctx->completion_lock){-...}-{2:2}, at: io_req_complete_post+0x4e/0x920 fs/io_uring.c:1923
> 
> the dependencies between HARDIRQ-irq-safe lock and the holding lock:
> -> (&ctx->completion_lock){-...}-{2:2} {
>    IN-HARDIRQ-W at:
>                     lock_acquire kernel/locking/lockdep.c:5510 [inline]
>                     lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
>                     __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
>                     _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:159
>                     io_link_timeout_fn+0xbf/0x720 fs/io_uring.c:6495
>                     __run_hrtimer kernel/time/hrtimer.c:1519 [inline]
>                     __hrtimer_run_queues+0x609/0xe40 kernel/time/hrtimer.c:1583
>                     hrtimer_interrupt+0x334/0x940 kernel/time/hrtimer.c:1645
>                     local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
>                     __sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1106
>                     asm_call_irq_on_stack+0xf/0x20
>                     __run_sysvec_on_irqstack arch/x86/include/asm/irq_stack.h:37 [inline]
>                     run_sysvec_on_irqstack_cond arch/x86/include/asm/irq_stack.h:89 [inline]
>                     sysvec_apic_timer_interrupt+0xbd/0x100 arch/x86/kernel/apic/apic.c:1100
>                     asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:635
>                     __raw_spin_unlock_irq include/linux/spinlock_api_smp.h:169 [inline]
>                     _raw_spin_unlock_irq+0x25/0x40 kernel/locking/spinlock.c:199
>                     spin_unlock_irq include/linux/spinlock.h:404 [inline]
>                     io_queue_linked_timeout+0x194/0x1f0 fs/io_uring.c:6541
>                     __io_queue_sqe+0x32f/0xdb0 fs/io_uring.c:6607
>                     __io_req_task_submit+0x18e/0x240 fs/io_uring.c:2344
>                     __tctx_task_work fs/io_uring.c:2204 [inline]
>                     tctx_task_work+0x12b/0x330 fs/io_uring.c:2230
>                     task_work_run+0xdd/0x1a0 kernel/task_work.c:140
>                     tracehook_notify_signal include/linux/tracehook.h:212 [inline]
>                     handle_signal_work kernel/entry/common.c:145 [inline]
>                     exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
>                     exit_to_user_mode_prepare+0x221/0x250 kernel/entry/common.c:208
>                     __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
>                     syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
>                     entry_SYSCALL_64_after_hwframe+0x44/0xae
>    INITIAL USE at:
>                    lock_acquire kernel/locking/lockdep.c:5510 [inline]
>                    lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
>                    __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
>                    _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:159
>                    io_req_complete_post+0x4e/0x920 fs/io_uring.c:1923
>                    __io_req_complete fs/io_uring.c:1963 [inline]
>                    io_req_complete fs/io_uring.c:1968 [inline]
>                    io_queue_sqe+0xa3b/0xfa0 fs/io_uring.c:6620
>                    io_submit_sqe fs/io_uring.c:6707 [inline]
>                    io_submit_sqes+0x15f5/0x2b60 fs/io_uring.c:6939
>                    __do_sys_io_uring_enter+0x1154/0x1f50 fs/io_uring.c:9454
>                    do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
>                    entry_SYSCALL_64_after_hwframe+0x44/0xae
>  }
>  ... key      at: [<ffffffff8fe69a80>] __key.9+0x0/0x40
>  ... acquired at:
>    lock_acquire kernel/locking/lockdep.c:5510 [inline]
>    lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
>    __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
>    _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
>    spin_lock include/linux/spinlock.h:354 [inline]
>    io_req_clean_work fs/io_uring.c:1405 [inline]
>    io_dismantle_req+0x90f/0xf90 fs/io_uring.c:2051
>    io_req_complete_post+0xf6/0x920 fs/io_uring.c:1933
>    __io_req_complete fs/io_uring.c:1963 [inline]
>    io_req_complete fs/io_uring.c:1968 [inline]
>    io_cancel_defer_files fs/io_uring.c:8945 [inline]
>    io_uring_cancel_task_requests+0x67e/0xea0 fs/io_uring.c:9052
>    __io_uring_files_cancel+0x110/0x1b0 fs/io_uring.c:9140
>    io_uring_files_cancel include/linux/io_uring.h:65 [inline]
>    do_exit+0x2fe/0x2ae0 kernel/exit.c:780
>    do_group_exit+0x125/0x310 kernel/exit.c:922
>    get_signal+0x42c/0x2100 kernel/signal.c:2773
>    arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
>    handle_signal_work kernel/entry/common.c:147 [inline]
>    exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
>    exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:208
>    __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
>    syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
>    entry_SYSCALL_64_after_hwframe+0x44/0xae
> 
> 
> the dependencies between the lock to be acquired
>  and HARDIRQ-irq-unsafe lock:
> -> (&fs->lock){+.+.}-{2:2} {
>    HARDIRQ-ON-W at:
>                     lock_acquire kernel/locking/lockdep.c:5510 [inline]
>                     lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
>                     __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
>                     _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
>                     spin_lock include/linux/spinlock.h:354 [inline]
>                     set_fs_pwd+0x85/0x2a0 fs/fs_struct.c:39
>                     init_chdir+0x106/0x14e fs/init.c:54
>                     devtmpfs_setup drivers/base/devtmpfs.c:415 [inline]
>                     devtmpfsd+0x76/0x333 drivers/base/devtmpfs.c:430
>                     kthread+0x3b1/0x4a0 kernel/kthread.c:292
>                     ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
>    SOFTIRQ-ON-W at:
>                     lock_acquire kernel/locking/lockdep.c:5510 [inline]
>                     lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
>                     __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
>                     _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
>                     spin_lock include/linux/spinlock.h:354 [inline]
>                     set_fs_pwd+0x85/0x2a0 fs/fs_struct.c:39
>                     init_chdir+0x106/0x14e fs/init.c:54
>                     devtmpfs_setup drivers/base/devtmpfs.c:415 [inline]
>                     devtmpfsd+0x76/0x333 drivers/base/devtmpfs.c:430
>                     kthread+0x3b1/0x4a0 kernel/kthread.c:292
>                     ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
>    INITIAL USE at:
>                    lock_acquire kernel/locking/lockdep.c:5510 [inline]
>                    lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
>                    __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
>                    _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
>                    spin_lock include/linux/spinlock.h:354 [inline]
>                    set_fs_pwd+0x85/0x2a0 fs/fs_struct.c:39
>                    init_chdir+0x106/0x14e fs/init.c:54
>                    devtmpfs_setup drivers/base/devtmpfs.c:415 [inline]
>                    devtmpfsd+0x76/0x333 drivers/base/devtmpfs.c:430
>                    kthread+0x3b1/0x4a0 kernel/kthread.c:292
>                    ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
>  }
>  ... key      at: [<ffffffff8fe68260>] __key.1+0x0/0x40
>  ... acquired at:
>    lock_acquire kernel/locking/lockdep.c:5510 [inline]
>    lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
>    __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
>    _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
>    spin_lock include/linux/spinlock.h:354 [inline]
>    io_req_clean_work fs/io_uring.c:1405 [inline]
>    io_dismantle_req+0x90f/0xf90 fs/io_uring.c:2051
>    io_req_complete_post+0xf6/0x920 fs/io_uring.c:1933
>    __io_req_complete fs/io_uring.c:1963 [inline]
>    io_req_complete fs/io_uring.c:1968 [inline]
>    io_cancel_defer_files fs/io_uring.c:8945 [inline]
>    io_uring_cancel_task_requests+0x67e/0xea0 fs/io_uring.c:9052
>    __io_uring_files_cancel+0x110/0x1b0 fs/io_uring.c:9140
>    io_uring_files_cancel include/linux/io_uring.h:65 [inline]
>    do_exit+0x2fe/0x2ae0 kernel/exit.c:780
>    do_group_exit+0x125/0x310 kernel/exit.c:922
>    get_signal+0x42c/0x2100 kernel/signal.c:2773
>    arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
>    handle_signal_work kernel/entry/common.c:147 [inline]
>    exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
>    exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:208
>    __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
>    syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
>    entry_SYSCALL_64_after_hwframe+0x44/0xae
> 
> 
> stack backtrace:
> CPU: 3 PID: 12185 Comm: syz-executor.0 Not tainted 5.11.0-syzkaller #0
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-2 04/01/2014
> Call Trace:
>  __dump_stack lib/dump_stack.c:79 [inline]
>  dump_stack+0xfa/0x151 lib/dump_stack.c:120
>  print_bad_irq_dependency kernel/locking/lockdep.c:2460 [inline]
>  check_irq_usage.cold+0x50d/0x744 kernel/locking/lockdep.c:2689
>  check_prev_add kernel/locking/lockdep.c:2940 [inline]
>  check_prevs_add kernel/locking/lockdep.c:3059 [inline]
>  validate_chain kernel/locking/lockdep.c:3674 [inline]
>  __lock_acquire+0x2b2c/0x54c0 kernel/locking/lockdep.c:4900
>  lock_acquire kernel/locking/lockdep.c:5510 [inline]
>  lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
>  __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
>  _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
>  spin_lock include/linux/spinlock.h:354 [inline]
>  io_req_clean_work fs/io_uring.c:1405 [inline]
>  io_dismantle_req+0x90f/0xf90 fs/io_uring.c:2051
>  io_req_complete_post+0xf6/0x920 fs/io_uring.c:1933
>  __io_req_complete fs/io_uring.c:1963 [inline]
>  io_req_complete fs/io_uring.c:1968 [inline]
>  io_cancel_defer_files fs/io_uring.c:8945 [inline]
>  io_uring_cancel_task_requests+0x67e/0xea0 fs/io_uring.c:9052
>  __io_uring_files_cancel+0x110/0x1b0 fs/io_uring.c:9140
>  io_uring_files_cancel include/linux/io_uring.h:65 [inline]
>  do_exit+0x2fe/0x2ae0 kernel/exit.c:780
>  do_group_exit+0x125/0x310 kernel/exit.c:922
>  get_signal+0x42c/0x2100 kernel/signal.c:2773
>  arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
>  handle_signal_work kernel/entry/common.c:147 [inline]
>  exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
>  exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:208
>  __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
>  syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
>  entry_SYSCALL_64_after_hwframe+0x44/0xae
> RIP: 0033:0x465ef9
> Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
> RSP: 002b:00007ffb56aa0218 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
> RAX: 0000000000000000 RBX: 000000000056bf68 RCX: 0000000000465ef9
> RDX: 0000000000000000 RSI: 0000000000000080 RDI: 000000000056bf68
> RBP: 000000000056bf60 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056bf6c
> R13: 00007fff198147ff R14: 00007ffb56aa0300 R15: 0000000000022000
> 
> 
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at [email protected].
> 
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> 

-- 
Pavel Begunkov

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2021-02-23 13:54 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-02-23 13:35 possible deadlock in io_link_timeout_fn syzbot
2021-02-23 13:50 ` Pavel Begunkov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox