Re: [PATCH next] io_uring: fix task hung in io_uring_setup

Re: [PATCH next] io_uring: fix task hung in io_uring_setup

[Jens Axboe]

On 9/3/20 7:21 AM, Hillf Danton wrote:
> 
> The smart syzbot found the following issue:
> 
> INFO: task syz-executor047:6853 blocked for more than 143 seconds.
>       Not tainted 5.9.0-rc3-next-20200902-syzkaller #0
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> task:syz-executor047 state:D stack:28104 pid: 6853 ppid:  6847 flags:0x00004000
> Call Trace:
>  context_switch kernel/sched/core.c:3777 [inline]
>  __schedule+0xea9/0x2230 kernel/sched/core.c:4526
>  schedule+0xd0/0x2a0 kernel/sched/core.c:4601
>  schedule_timeout+0x1d8/0x250 kernel/time/timer.c:1855
>  do_wait_for_common kernel/sched/completion.c:85 [inline]
>  __wait_for_common kernel/sched/completion.c:106 [inline]
>  wait_for_common kernel/sched/completion.c:117 [inline]
>  wait_for_completion+0x163/0x260 kernel/sched/completion.c:138
>  io_sq_thread_stop fs/io_uring.c:6906 [inline]
>  io_finish_async fs/io_uring.c:6920 [inline]
>  io_sq_offload_create fs/io_uring.c:7595 [inline]
>  io_uring_create fs/io_uring.c:8671 [inline]
>  io_uring_setup+0x1495/0x29a0 fs/io_uring.c:8744
>  do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
>  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> 
> because the sqo_thread kthread is created in io_sq_offload_create() without
> being waked up. Then in the error branch of that function we will wait for
> the sqo kthread that never runs. It's fixed by waking it up before waiting.

Looks good - applied, thanks.

-- 
Jens Axboe

Re: [PATCH next] io_uring: fix task hung in io_uring_setup

[Pavel Begunkov]

On 03/09/2020 17:04, Jens Axboe wrote:
> On 9/3/20 7:21 AM, Hillf Danton wrote:
>>
>> The smart syzbot found the following issue:
>>
>> INFO: task syz-executor047:6853 blocked for more than 143 seconds.
>>       Not tainted 5.9.0-rc3-next-20200902-syzkaller #0
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> task:syz-executor047 state:D stack:28104 pid: 6853 ppid:  6847 flags:0x00004000
>> Call Trace:
>>  context_switch kernel/sched/core.c:3777 [inline]
>>  __schedule+0xea9/0x2230 kernel/sched/core.c:4526
>>  schedule+0xd0/0x2a0 kernel/sched/core.c:4601
>>  schedule_timeout+0x1d8/0x250 kernel/time/timer.c:1855
>>  do_wait_for_common kernel/sched/completion.c:85 [inline]
>>  __wait_for_common kernel/sched/completion.c:106 [inline]
>>  wait_for_common kernel/sched/completion.c:117 [inline]
>>  wait_for_completion+0x163/0x260 kernel/sched/completion.c:138
>>  io_sq_thread_stop fs/io_uring.c:6906 [inline]
>>  io_finish_async fs/io_uring.c:6920 [inline]
>>  io_sq_offload_create fs/io_uring.c:7595 [inline]
>>  io_uring_create fs/io_uring.c:8671 [inline]
>>  io_uring_setup+0x1495/0x29a0 fs/io_uring.c:8744
>>  do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
>>  entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>
>> because the sqo_thread kthread is created in io_sq_offload_create() without
>> being waked up. Then in the error branch of that function we will wait for
>> the sqo kthread that never runs. It's fixed by waking it up before waiting.
> 
> Looks good - applied, thanks.

BTW, I don't see the patch itself, and it's neither in io_uring, block
nor fs mailing lists. Hillf, could you please CC proper lists next time?

-- 
Pavel Begunkov

Re: [PATCH next] io_uring: fix task hung in io_uring_setup

[Jens Axboe]

On 9/7/20 2:50 AM, Pavel Begunkov wrote:
> On 03/09/2020 17:04, Jens Axboe wrote:
>> On 9/3/20 7:21 AM, Hillf Danton wrote:
>>>
>>> The smart syzbot found the following issue:
>>>
>>> INFO: task syz-executor047:6853 blocked for more than 143 seconds.
>>>       Not tainted 5.9.0-rc3-next-20200902-syzkaller #0
>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> task:syz-executor047 state:D stack:28104 pid: 6853 ppid:  6847 flags:0x00004000
>>> Call Trace:
>>>  context_switch kernel/sched/core.c:3777 [inline]
>>>  __schedule+0xea9/0x2230 kernel/sched/core.c:4526
>>>  schedule+0xd0/0x2a0 kernel/sched/core.c:4601
>>>  schedule_timeout+0x1d8/0x250 kernel/time/timer.c:1855
>>>  do_wait_for_common kernel/sched/completion.c:85 [inline]
>>>  __wait_for_common kernel/sched/completion.c:106 [inline]
>>>  wait_for_common kernel/sched/completion.c:117 [inline]
>>>  wait_for_completion+0x163/0x260 kernel/sched/completion.c:138
>>>  io_sq_thread_stop fs/io_uring.c:6906 [inline]
>>>  io_finish_async fs/io_uring.c:6920 [inline]
>>>  io_sq_offload_create fs/io_uring.c:7595 [inline]
>>>  io_uring_create fs/io_uring.c:8671 [inline]
>>>  io_uring_setup+0x1495/0x29a0 fs/io_uring.c:8744
>>>  do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
>>>  entry_SYSCALL_64_after_hwframe+0x44/0xa9
>>>
>>> because the sqo_thread kthread is created in io_sq_offload_create() without
>>> being waked up. Then in the error branch of that function we will wait for
>>> the sqo kthread that never runs. It's fixed by waking it up before waiting.
>>
>> Looks good - applied, thanks.
> 
> BTW, I don't see the patch itself, and it's neither in io_uring, block
> nor fs mailing lists. Hillf, could you please CC proper lists next time?

He did, but I'm guessing that vger didn't like the email for whatever
reason. Hillf, did you get an error back from vger when sending the patch?

-- 
Jens Axboe

Re: [PATCH next] io_uring: fix task hung in io_uring_setup

[Pavel Begunkov]

On 08/09/2020 03:03, Hillf Danton wrote:
> 
> On Mon, 7 Sep 2020 06:55:04 Jens Axboe wrote:
>> On 9/7/20 2:50 AM, Pavel Begunkov wrote:
>>>
>>> BTW, I don't see the patch itself, and it's neither in io_uring, block
>>> nor fs mailing lists. Hillf, could you please CC proper lists next time?
> 
> Yes, I can. So will I send io_uring patches with Pavel Cced.

Thanks

>>
>> He did, but I'm guessing that vger didn't like the email for whatever
>> reason. Hillf, did you get an error back from vger when sending the pat	
> 
> My inbox, a simple free mail, is perhaps blocked as spam at the vger end.

-- 
Pavel Begunkov

Re: [PATCH next] io_uring: fix task hung in io_uring_setup

[Jens Axboe]

On 9/7/20 6:03 PM, Hillf Danton wrote:
> 
> On Mon, 7 Sep 2020 06:55:04 Jens Axboe wrote:
>> On 9/7/20 2:50 AM, Pavel Begunkov wrote:
>>>
>>> BTW, I don't see the patch itself, and it's neither in io_uring, block
>>> nor fs mailing lists. Hillf, could you please CC proper lists next time?
> 
> Yes, I can. So will I send io_uring patches with Pavel Cced.

While that is nice, it should not be necessary. We need to ensure that your
emails reach the list, that's more important than needing to CC a specific
person, because it still means that everyone else doesn't see it.

Do you get an error from vger, or does it simply not show up?

-- 
Jens Axboe

Re: [PATCH next] io_uring: fix task hung in io_uring_setup

[Jens Axboe]

On 9/8/20 6:19 PM, Hillf Danton wrote:
> 
> On Tue, 8 Sep 2020 17:34:26 -0600 Jens Axboe wrote:
>> On 9/7/20 6:03 PM, Hillf Danton wrote:
>>> On Mon, 7 Sep 2020 06:55:04 Jens Axboe wrote:
>>>> On 9/7/20 2:50 AM, Pavel Begunkov wrote:
>>>>>
>>>>> BTW, I don't see the patch itself, and it's neither in io_uring, block
>>>>> nor fs mailing lists. Hillf, could you please CC proper lists next time?
>>>
>>> Yes, I can. So will I send io_uring patches with Pavel Cced.
>>
>> While that is nice, it should not be necessary. We need to ensure that your
>> emails reach the list, that's more important than needing to CC a specific
>> person, because it still means that everyone else doesn't see it.
>>
>> Do you get an error from vger, or does it simply not show up?
> 
> After tapping the send button for this message, I will receive a message
> from the sina mail server saying it failed to deliver it to one of the
> targets (abc@vger.kernel.org), which has been happing over the past a
> couple of years. One of the redhat guys, I can't remmenber his name,
> once tryied to help me solve the problem, by sending somebody@vger a
> message explaining what was going on, but failed. AFAIC there's a
> glitch in exchanging info between the sina server and the server at the
> vger end, and it seems it would take more time than thought to figure
> it out. So let it be for now.

Might be worthwhile to just have a gmail account for sending patches
and replying to list emails?

-- 
Jens Axboe

Re: [PATCH next] io_uring: fix task hung in io_uring_setup

[Stefano Garzarella]

On Thu, Sep 03, 2020 at 09:21:19PM +0800, Hillf Danton wrote:
> 
> The smart syzbot found the following issue:
> 
> INFO: task syz-executor047:6853 blocked for more than 143 seconds.
>       Not tainted 5.9.0-rc3-next-20200902-syzkaller #0
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> task:syz-executor047 state:D stack:28104 pid: 6853 ppid:  6847 flags:0x00004000
> Call Trace:
>  context_switch kernel/sched/core.c:3777 [inline]
>  __schedule+0xea9/0x2230 kernel/sched/core.c:4526
>  schedule+0xd0/0x2a0 kernel/sched/core.c:4601
>  schedule_timeout+0x1d8/0x250 kernel/time/timer.c:1855
>  do_wait_for_common kernel/sched/completion.c:85 [inline]
>  __wait_for_common kernel/sched/completion.c:106 [inline]
>  wait_for_common kernel/sched/completion.c:117 [inline]
>  wait_for_completion+0x163/0x260 kernel/sched/completion.c:138
>  io_sq_thread_stop fs/io_uring.c:6906 [inline]
>  io_finish_async fs/io_uring.c:6920 [inline]
>  io_sq_offload_create fs/io_uring.c:7595 [inline]
>  io_uring_create fs/io_uring.c:8671 [inline]
>  io_uring_setup+0x1495/0x29a0 fs/io_uring.c:8744
>  do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
>  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> 
> because the sqo_thread kthread is created in io_sq_offload_create() without
> being waked up. Then in the error branch of that function we will wait for
> the sqo kthread that never runs. It's fixed by waking it up before waiting.
> 
> Reported-by: syzbot+107dd59d1efcaf3ffca4@syzkaller.appspotmail.com
> Fixes: dfe127799f8e ("io_uring: allow disabling rings during the creation")
> Cc: Stefano Garzarella <sgarzare@redhat.com>
> Cc: Kees Cook <keescook@chromium.org>
> Signed-off-by: Hillf Danton <hdanton@sina.com>
> ---
> 
> --- a/fs/io_uring.c
> +++ b/fs/io_uring.c
> @@ -6903,6 +6903,13 @@ static int io_sqe_files_unregister(struc
>  static void io_sq_thread_stop(struct io_ring_ctx *ctx)
>  {
>  	if (ctx->sqo_thread) {
> +		/*
> +		 * We may arrive here from the error branch in
> +		 * io_sq_offload_create() where the kthread is created without
> +		 * being waked up, thus wake it up now to make sure the wait will
> +		 * complete.
> +		 */
> +		wake_up_process(ctx->sqo_thread);
>  		wait_for_completion(&ctx->sq_thread_comp);
>  		/*
>  		 * The park is a bit of a work-around, without it we get
> --
> 

Thanks for fixing this issue!
Jens already queued this, but just for recording:

Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>


Thanks,
Stefano