From: Pavel Begunkov <[email protected]>
To: Hao Xu <[email protected]>, Jens Axboe <[email protected]>
Cc: [email protected], Joseph Qi <[email protected]>
Subject: Re: [PATCH RFC 5.13 2/2] io_uring: submit sqes in the original context when waking up sqthread
Date: Thu, 29 Apr 2021 23:02:20 +0100 [thread overview]
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
On 4/28/21 2:32 PM, Hao Xu wrote:
> sqes are submitted by sqthread when it is leveraged, which means there
> is IO latency when waking up sqthread. To wipe it out, submit limited
> number of sqes in the original task context.
> Tests result below:
Synchronising needs some patching in case of IOPOLL, as
io_iopoll_req_issued() won't wake up sqpoll task. Added in this patch
wake_up() doesn't help with that because it's placed before submission
>
> 99th latency:
> iops\idle 10us 60us 110us 160us 210us 260us 310us 360us 410us 460us 510us
> with this patch:
> 2k 13 13 12 13 13 12 12 11 11 10.304 11.84
> without this patch:
> 2k 15 14 15 15 15 14 15 14 14 13 11.84
>
> fio config:
> ./run_fio.sh
> fio \
> --ioengine=io_uring --sqthread_poll=1 --hipri=1 --thread=1 --bs=4k \
> --direct=1 --rw=randread --time_based=1 --runtime=300 \
> --group_reporting=1 --filename=/dev/nvme1n1 --sqthread_poll_cpu=30 \
> --randrepeat=0 --cpus_allowed=35 --iodepth=128 --rate_iops=${1} \
> --io_sq_thread_idle=${2}
>
> Signed-off-by: Hao Xu <[email protected]>
> ---
> fs/io_uring.c | 29 +++++++++++++++++++++++------
> include/uapi/linux/io_uring.h | 1 +
> 2 files changed, 24 insertions(+), 6 deletions(-)
>
> diff --git a/fs/io_uring.c b/fs/io_uring.c
> index 1871fad48412..f0a01232671e 100644
> --- a/fs/io_uring.c
> +++ b/fs/io_uring.c
> @@ -1252,7 +1252,12 @@ static void io_queue_async_work(struct io_kiocb *req)
> {
> struct io_ring_ctx *ctx = req->ctx;
> struct io_kiocb *link = io_prep_linked_timeout(req);
> - struct io_uring_task *tctx = req->task->io_uring;
> + struct io_uring_task *tctx = NULL;
> +
> + if (ctx->sq_data && ctx->sq_data->thread)
> + tctx = ctx->sq_data->thread->io_uring;
> + else
> + tctx = req->task->io_uring;
>
> BUG_ON(!tctx);
> BUG_ON(!tctx->io_wq);
> @@ -9063,9 +9068,10 @@ static void io_uring_try_cancel(struct files_struct *files)
> xa_for_each(&tctx->xa, index, node) {
> struct io_ring_ctx *ctx = node->ctx;
>
> - /* sqpoll task will cancel all its requests */
> - if (!ctx->sq_data)
> - io_uring_try_cancel_requests(ctx, current, files);
> + /*
> + * for sqpoll ctx, there may be requests in task_works etc.
> + */
> + io_uring_try_cancel_requests(ctx, current, files);
> }
> }
>
> @@ -9271,7 +9277,8 @@ static int io_get_ext_arg(unsigned flags, const void __user *argp, size_t *argsz
> io_run_task_work();
>
> if (unlikely(flags & ~(IORING_ENTER_GETEVENTS | IORING_ENTER_SQ_WAKEUP |
> - IORING_ENTER_SQ_WAIT | IORING_ENTER_EXT_ARG)))
> + IORING_ENTER_SQ_WAIT | IORING_ENTER_EXT_ARG |
> + IORING_ENTER_SQ_DEPUTY)))
> return -EINVAL;
>
> f = fdget(fd);
> @@ -9304,8 +9311,18 @@ static int io_get_ext_arg(unsigned flags, const void __user *argp, size_t *argsz
> if (unlikely(ctx->sq_data->thread == NULL)) {
> goto out;
> }
> - if (flags & IORING_ENTER_SQ_WAKEUP)
> + if (flags & IORING_ENTER_SQ_WAKEUP) {
> wake_up(&ctx->sq_data->wait);
> + if ((flags & IORING_ENTER_SQ_DEPUTY) &&
> + !(ctx->flags & IORING_SETUP_IOPOLL)) {
> + ret = io_uring_add_task_file(ctx);
> + if (unlikely(ret))
> + goto out;
> + mutex_lock(&ctx->uring_lock);
> + io_submit_sqes(ctx, min(to_submit, 8U));
> + mutex_unlock(&ctx->uring_lock);
> + }
> + }
> if (flags & IORING_ENTER_SQ_WAIT) {
> ret = io_sqpoll_wait_sq(ctx);
> if (ret)
> diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h
> index 311532ff6ce3..b1130fec2b7d 100644
> --- a/include/uapi/linux/io_uring.h
> +++ b/include/uapi/linux/io_uring.h
> @@ -251,6 +251,7 @@ struct io_cqring_offsets {
> #define IORING_ENTER_SQ_WAKEUP (1U << 1)
> #define IORING_ENTER_SQ_WAIT (1U << 2)
> #define IORING_ENTER_EXT_ARG (1U << 3)
> +#define IORING_ENTER_SQ_DEPUTY (1U << 4)
>
> /*
> * Passed in for io_uring_setup(2). Copied back with updated info on success
>
--
Pavel Begunkov
prev parent reply other threads:[~2021-04-29 22:02 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-28 13:32 [PATCH RFC 5.13 0/2] adaptive sqpoll and its wakeup optimization Hao Xu
2021-04-28 13:32 ` [PATCH RFC 5.13 1/2] io_uring: add support for ns granularity of io_sq_thread_idle Hao Xu
2021-04-28 14:07 ` Pavel Begunkov
2021-04-28 14:16 ` Jens Axboe
2021-04-28 14:53 ` Pavel Begunkov
2021-04-28 14:54 ` Jens Axboe
2021-04-29 3:41 ` Hao Xu
2021-04-29 9:11 ` Pavel Begunkov
2021-05-05 14:07 ` Hao Xu
2021-05-05 17:40 ` Pavel Begunkov
2021-04-29 3:28 ` Hao Xu
2021-04-29 22:15 ` Pavel Begunkov
2021-09-26 10:00 ` Hao Xu
2021-09-28 10:51 ` Pavel Begunkov
2021-09-29 7:52 ` Hao Xu
2021-09-29 9:24 ` Hao Xu
2021-09-29 11:37 ` Pavel Begunkov
2021-09-29 12:13 ` Hao Xu
2021-09-30 8:51 ` Pavel Begunkov
2021-09-30 12:04 ` Pavel Begunkov
2021-10-05 15:00 ` Hao Xu
2021-04-28 13:32 ` [PATCH RFC 5.13 2/2] io_uring: submit sqes in the original context when waking up sqthread Hao Xu
2021-04-28 14:12 ` Jens Axboe
2021-04-29 4:12 ` Hao Xu
2021-04-28 14:34 ` Pavel Begunkov
2021-04-28 14:37 ` Pavel Begunkov
2021-04-29 4:37 ` Hao Xu
2021-04-29 9:28 ` Pavel Begunkov
2021-05-05 11:20 ` Hao Xu
2021-04-28 14:39 ` Jens Axboe
2021-04-28 14:50 ` Pavel Begunkov
2021-04-28 14:53 ` Jens Axboe
2021-04-28 14:56 ` Pavel Begunkov
2021-04-28 15:09 ` Jens Axboe
2021-04-29 4:43 ` Hao Xu
2021-04-29 8:44 ` Hao Xu
2021-04-29 22:10 ` Pavel Begunkov
2021-05-05 13:10 ` Hao Xu
2021-05-05 17:44 ` Pavel Begunkov
2021-04-29 22:02 ` Pavel Begunkov [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox