From: Hao Xu <[email protected]>
To: Jens Axboe <[email protected]>
Cc: [email protected], Pavel Begunkov <[email protected]>,
Joseph Qi <[email protected]>
Subject: [PATCH RFC 5.13 2/2] io_uring: submit sqes in the original context when waking up sqthread
Date: Wed, 28 Apr 2021 21:32:28 +0800 [thread overview]
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
sqes are submitted by sqthread when it is leveraged, which means there
is IO latency when waking up sqthread. To wipe it out, submit limited
number of sqes in the original task context.
Tests result below:
99th latency:
iops\idle 10us 60us 110us 160us 210us 260us 310us 360us 410us 460us 510us
with this patch:
2k 13 13 12 13 13 12 12 11 11 10.304 11.84
without this patch:
2k 15 14 15 15 15 14 15 14 14 13 11.84
fio config:
./run_fio.sh
fio \
--ioengine=io_uring --sqthread_poll=1 --hipri=1 --thread=1 --bs=4k \
--direct=1 --rw=randread --time_based=1 --runtime=300 \
--group_reporting=1 --filename=/dev/nvme1n1 --sqthread_poll_cpu=30 \
--randrepeat=0 --cpus_allowed=35 --iodepth=128 --rate_iops=${1} \
--io_sq_thread_idle=${2}
Signed-off-by: Hao Xu <[email protected]>
---
fs/io_uring.c | 29 +++++++++++++++++++++++------
include/uapi/linux/io_uring.h | 1 +
2 files changed, 24 insertions(+), 6 deletions(-)
diff --git a/fs/io_uring.c b/fs/io_uring.c
index 1871fad48412..f0a01232671e 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -1252,7 +1252,12 @@ static void io_queue_async_work(struct io_kiocb *req)
{
struct io_ring_ctx *ctx = req->ctx;
struct io_kiocb *link = io_prep_linked_timeout(req);
- struct io_uring_task *tctx = req->task->io_uring;
+ struct io_uring_task *tctx = NULL;
+
+ if (ctx->sq_data && ctx->sq_data->thread)
+ tctx = ctx->sq_data->thread->io_uring;
+ else
+ tctx = req->task->io_uring;
BUG_ON(!tctx);
BUG_ON(!tctx->io_wq);
@@ -9063,9 +9068,10 @@ static void io_uring_try_cancel(struct files_struct *files)
xa_for_each(&tctx->xa, index, node) {
struct io_ring_ctx *ctx = node->ctx;
- /* sqpoll task will cancel all its requests */
- if (!ctx->sq_data)
- io_uring_try_cancel_requests(ctx, current, files);
+ /*
+ * for sqpoll ctx, there may be requests in task_works etc.
+ */
+ io_uring_try_cancel_requests(ctx, current, files);
}
}
@@ -9271,7 +9277,8 @@ static int io_get_ext_arg(unsigned flags, const void __user *argp, size_t *argsz
io_run_task_work();
if (unlikely(flags & ~(IORING_ENTER_GETEVENTS | IORING_ENTER_SQ_WAKEUP |
- IORING_ENTER_SQ_WAIT | IORING_ENTER_EXT_ARG)))
+ IORING_ENTER_SQ_WAIT | IORING_ENTER_EXT_ARG |
+ IORING_ENTER_SQ_DEPUTY)))
return -EINVAL;
f = fdget(fd);
@@ -9304,8 +9311,18 @@ static int io_get_ext_arg(unsigned flags, const void __user *argp, size_t *argsz
if (unlikely(ctx->sq_data->thread == NULL)) {
goto out;
}
- if (flags & IORING_ENTER_SQ_WAKEUP)
+ if (flags & IORING_ENTER_SQ_WAKEUP) {
wake_up(&ctx->sq_data->wait);
+ if ((flags & IORING_ENTER_SQ_DEPUTY) &&
+ !(ctx->flags & IORING_SETUP_IOPOLL)) {
+ ret = io_uring_add_task_file(ctx);
+ if (unlikely(ret))
+ goto out;
+ mutex_lock(&ctx->uring_lock);
+ io_submit_sqes(ctx, min(to_submit, 8U));
+ mutex_unlock(&ctx->uring_lock);
+ }
+ }
if (flags & IORING_ENTER_SQ_WAIT) {
ret = io_sqpoll_wait_sq(ctx);
if (ret)
diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h
index 311532ff6ce3..b1130fec2b7d 100644
--- a/include/uapi/linux/io_uring.h
+++ b/include/uapi/linux/io_uring.h
@@ -251,6 +251,7 @@ struct io_cqring_offsets {
#define IORING_ENTER_SQ_WAKEUP (1U << 1)
#define IORING_ENTER_SQ_WAIT (1U << 2)
#define IORING_ENTER_EXT_ARG (1U << 3)
+#define IORING_ENTER_SQ_DEPUTY (1U << 4)
/*
* Passed in for io_uring_setup(2). Copied back with updated info on success
--
1.8.3.1
next prev parent reply other threads:[~2021-04-28 13:32 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-28 13:32 [PATCH RFC 5.13 0/2] adaptive sqpoll and its wakeup optimization Hao Xu
2021-04-28 13:32 ` [PATCH RFC 5.13 1/2] io_uring: add support for ns granularity of io_sq_thread_idle Hao Xu
2021-04-28 14:07 ` Pavel Begunkov
2021-04-28 14:16 ` Jens Axboe
2021-04-28 14:53 ` Pavel Begunkov
2021-04-28 14:54 ` Jens Axboe
2021-04-29 3:41 ` Hao Xu
2021-04-29 9:11 ` Pavel Begunkov
2021-05-05 14:07 ` Hao Xu
2021-05-05 17:40 ` Pavel Begunkov
2021-04-29 3:28 ` Hao Xu
2021-04-29 22:15 ` Pavel Begunkov
2021-09-26 10:00 ` Hao Xu
2021-09-28 10:51 ` Pavel Begunkov
2021-09-29 7:52 ` Hao Xu
2021-09-29 9:24 ` Hao Xu
2021-09-29 11:37 ` Pavel Begunkov
2021-09-29 12:13 ` Hao Xu
2021-09-30 8:51 ` Pavel Begunkov
2021-09-30 12:04 ` Pavel Begunkov
2021-10-05 15:00 ` Hao Xu
2021-04-28 13:32 ` Hao Xu [this message]
2021-04-28 14:12 ` [PATCH RFC 5.13 2/2] io_uring: submit sqes in the original context when waking up sqthread Jens Axboe
2021-04-29 4:12 ` Hao Xu
2021-04-28 14:34 ` Pavel Begunkov
2021-04-28 14:37 ` Pavel Begunkov
2021-04-29 4:37 ` Hao Xu
2021-04-29 9:28 ` Pavel Begunkov
2021-05-05 11:20 ` Hao Xu
2021-04-28 14:39 ` Jens Axboe
2021-04-28 14:50 ` Pavel Begunkov
2021-04-28 14:53 ` Jens Axboe
2021-04-28 14:56 ` Pavel Begunkov
2021-04-28 15:09 ` Jens Axboe
2021-04-29 4:43 ` Hao Xu
2021-04-29 8:44 ` Hao Xu
2021-04-29 22:10 ` Pavel Begunkov
2021-05-05 13:10 ` Hao Xu
2021-05-05 17:44 ` Pavel Begunkov
2021-04-29 22:02 ` Pavel Begunkov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1619616748-17149-3-git-send-email-haoxu@linux.alibaba.com \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox