public inbox for [email protected]
 help / color / mirror / Atom feed
From: Xiaoguang Wang <[email protected]>
To: Jens Axboe <[email protected]>, [email protected]
Subject: Re: [PATCH] io_uring: fix poll_list race for SETUP_IOPOLL|SETUP_SQPOLL
Date: Mon, 17 Feb 2020 16:35:59 +0800	[thread overview]
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>

hi,

> On 2/14/20 6:11 AM, Xiaoguang Wang wrote:
>> After making ext4 support iopoll method:
>>    let ext4_file_operations's iopoll method be iomap_dio_iopoll(),
>> we found fio can easily hang in fio_ioring_getevents() with below fio
>> job:
>>      rm -f testfile; sync;
>>      sudo fio -name=fiotest -filename=testfile -iodepth=128 -thread
>> -rw=write -ioengine=io_uring  -hipri=1 -sqthread_poll=1 -direct=1
>> -bs=4k -size=10G -numjobs=8 -runtime=2000 -group_reporting
>> with IORING_SETUP_SQPOLL and IORING_SETUP_IOPOLL enabled.
>>
>> There are two issues that results in this hang, one reason is that
>> when IORING_SETUP_SQPOLL and IORING_SETUP_IOPOLL are enabled, fio
>> does not use io_uring_enter to get completed events, it relies on
>> kernel io_sq_thread to poll for completed events.
>>
>> Another reason is that there is a race: when io_submit_sqes() in
>> io_sq_thread() submits a batch of sqes, variable 'inflight' will
>> record the number of submitted reqs, then io_sq_thread will poll for
>> reqs which have been added to poll_list. But note, if some previous
>> reqs have been punted to io worker, these reqs will won't be in
>> poll_list timely. io_sq_thread() will only poll for a part of previous
>> submitted reqs, and then find poll_list is empty, reset variable
>> 'inflight' to be zero. If app just waits these deferred reqs and does
>> not wake up io_sq_thread again, then hang happens.
>>
>> For app that entirely relies on io_sq_thread to poll completed requests,
>> let io_iopoll_req_issued() wake up io_sq_thread properly when adding new
>> element to poll_list.
> 
> I think your analysis is correct, but the various conditional locking
> and unlocking in io_sq_thread() is not easy to follow. When I see
> things like:
> 
> @@ -5101,16 +5095,22 @@ static int io_sq_thread(void *data)
>   			if (!to_submit || ret == -EBUSY) {
>   				if (kthread_should_park()) {
>   					finish_wait(&ctx->sqo_wait, &wait);
> +					if (iopoll)
> +						mutex_unlock(&ctx->uring_lock);
>   					break;
>   				}
>   				if (signal_pending(current))
>   					flush_signals(current);
> +				if (iopoll)
> +					mutex_unlock(&ctx->uring_lock);
>   				schedule();
>   				finish_wait(&ctx->sqo_wait, &wait);
>   
>   				ctx->rings->sq_flags &= ~IORING_SQ_NEED_WAKEUP;
>   				continue;
>   			}
> +			if (iopoll)
> +				mutex_unlock(&ctx->uring_lock);
>   			finish_wait(&ctx->sqo_wait, &wait);
>   
>   			ctx->rings->sq_flags &= ~IORING_SQ_NEED_WAKEUP;
> 
> it triggers the taste senses a bit. Any chance you could take another
> look at that part and see if we can clean it up a bit?
Ok, I'll try to make a better version, thanks.

Regards,
Xiaoguang Wang

> 
> Even if that isn't possible, then I think it'd help to rename 'iopoll'
> to something related to the lock, and have a comment when you first do:
> 
> 	/* If we're doing polled IO, we need to bla bla */
> 	if (ctx->flags & IORING_SETUP_IOPOLL)
> 		needs_uring_lock = true;
> 
> 

      reply	other threads:[~2020-02-17  8:36 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-02-14 13:11 [PATCH] io_uring: fix poll_list race for SETUP_IOPOLL|SETUP_SQPOLL Xiaoguang Wang
2020-02-16 17:13 ` Jens Axboe
2020-02-17  8:35   ` Xiaoguang Wang [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=6a8e4bcd-a817-8dea-c265-4d4a2108c6b2@linux.alibaba.com \
    [email protected] \
    [email protected] \
    [email protected] \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox