From: Pavel Begunkov <[email protected]>
To: Jens Axboe <[email protected]>, [email protected]
Subject: [PATCH 20/28] io_uring: optimise putting task struct
Date: Mon, 9 Aug 2021 13:04:20 +0100 [thread overview]
Message-ID: <6fe9646b3cb70e46aca1f58426776e368c8926b3.1628471125.git.asml.silence@gmail.com> (raw)
In-Reply-To: <[email protected]>
We cache all the reference to task + tctx, so if io_put_task() is
called by the corresponding task itself, we can save on atomics and
return the refs right back into the cache.
It's beneficial for all inline completions, and also iopolling, when
polling and submissions are done by the same task, including
SQPOLL|IOPOLL.
Note: io_uring_cancel_generic() can return refs to the cache as well,
so those should be flushed in the loop for tctx_inflight() to work
right.
Signed-off-by: Pavel Begunkov <[email protected]>
---
fs/io_uring.c | 17 +++++++++++------
1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/fs/io_uring.c b/fs/io_uring.c
index 3c5c4cf73d1c..0982b0dba6b0 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -2099,10 +2099,12 @@ static inline void io_init_req_batch(struct req_batch *rb)
static void io_req_free_batch_finish(struct io_ring_ctx *ctx,
struct req_batch *rb)
{
- if (rb->task)
- io_put_task(rb->task, rb->task_refs);
if (rb->ctx_refs)
percpu_ref_put_many(&ctx->refs, rb->ctx_refs);
+ if (rb->task == current)
+ current->io_uring->cached_refs += rb->task_refs;
+ else if (rb->task)
+ io_put_task(rb->task, rb->task_refs);
}
static void io_req_free_batch(struct req_batch *rb, struct io_kiocb *req,
@@ -9143,9 +9145,11 @@ static void io_uring_drop_tctx_refs(struct task_struct *task)
struct io_uring_task *tctx = task->io_uring;
unsigned int refs = tctx->cached_refs;
- tctx->cached_refs = 0;
- percpu_counter_sub(&tctx->inflight, refs);
- put_task_struct_many(task, refs);
+ if (refs) {
+ tctx->cached_refs = 0;
+ percpu_counter_sub(&tctx->inflight, refs);
+ put_task_struct_many(task, refs);
+ }
}
/*
@@ -9166,9 +9170,9 @@ static void io_uring_cancel_generic(bool cancel_all, struct io_sq_data *sqd)
if (tctx->io_wq)
io_wq_exit_start(tctx->io_wq);
- io_uring_drop_tctx_refs(current);
atomic_inc(&tctx->in_idle);
do {
+ io_uring_drop_tctx_refs(current);
/* read completions before cancelations */
inflight = tctx_inflight(tctx, !cancel_all);
if (!inflight)
@@ -9192,6 +9196,7 @@ static void io_uring_cancel_generic(bool cancel_all, struct io_sq_data *sqd)
}
prepare_to_wait(&tctx->wait, &wait, TASK_UNINTERRUPTIBLE);
+ io_uring_drop_tctx_refs(current);
/*
* If we've seen completions, retry without waiting. This
* avoids a race where a completion comes in before we did
--
2.32.0
next prev parent reply other threads:[~2021-08-09 12:05 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-09 12:04 [PATCH v2 00/28] for-next patches Pavel Begunkov
2021-08-09 12:04 ` [PATCH 01/28] io_uring: use kvmalloc for fixed files Pavel Begunkov
2021-08-09 12:04 ` [PATCH 02/28] io_uring: inline fixed part of io_file_get() Pavel Begunkov
2021-08-09 12:04 ` [PATCH 03/28] io_uring: rename io_file_supports_async() Pavel Begunkov
2021-08-09 12:04 ` [PATCH 04/28] io_uring: avoid touching inode in rw prep Pavel Begunkov
2021-08-09 12:04 ` [PATCH 05/28] io_uring: clean io-wq callbacks Pavel Begunkov
2021-08-09 12:04 ` [PATCH 06/28] io_uring: remove unnecessary PF_EXITING check Pavel Begunkov
2021-08-09 12:04 ` [PATCH 07/28] io-wq: improve wq_list_add_tail() Pavel Begunkov
2021-08-09 12:04 ` [PATCH 08/28] io_uring: refactor io_alloc_req Pavel Begunkov
2021-08-09 12:04 ` [PATCH 09/28] io_uring: don't halt iopoll too early Pavel Begunkov
2021-08-09 12:04 ` [PATCH 10/28] io_uring: add more locking annotations for submit Pavel Begunkov
2021-08-09 12:04 ` [PATCH 11/28] io_uring: optimise io_cqring_wait() hot path Pavel Begunkov
2021-08-09 12:04 ` [PATCH 12/28] io_uring: extract a helper for ctx quiesce Pavel Begunkov
2021-08-09 12:04 ` [PATCH 13/28] io_uring: move io_put_task() definition Pavel Begunkov
2021-08-09 12:04 ` [PATCH 14/28] io_uring: move io_rsrc_node_alloc() definition Pavel Begunkov
2021-08-09 12:04 ` [PATCH 15/28] io_uring: inline io_free_req_deferred Pavel Begunkov
2021-08-09 12:04 ` [PATCH 16/28] io_uring: deduplicate open iopoll check Pavel Begunkov
2021-08-09 12:04 ` [PATCH 17/28] io_uring: improve ctx hang handling Pavel Begunkov
2021-08-09 12:04 ` [PATCH 18/28] io_uring: kill unused IO_IOPOLL_BATCH Pavel Begunkov
2021-08-09 12:04 ` [PATCH 19/28] io_uring: drop exec checks from io_req_task_submit Pavel Begunkov
2021-08-09 12:04 ` Pavel Begunkov [this message]
2021-08-09 12:04 ` [PATCH 21/28] io_uring: hide async dadta behind flags Pavel Begunkov
2021-08-09 17:30 ` Jens Axboe
2021-08-09 17:44 ` Pavel Begunkov
2021-08-09 12:04 ` [PATCH 22/28] io_uring: move io_fallback_req_func() Pavel Begunkov
2021-08-09 12:04 ` [PATCH 23/28] io_uring: cache __io_free_req()'d requests Pavel Begunkov
2021-08-09 12:04 ` [PATCH 24/28] io_uring: remove redundant args from cache_free Pavel Begunkov
2021-08-09 12:04 ` [PATCH 25/28] io_uring: use inflight_entry instead of compl.list Pavel Begunkov
2021-08-09 12:04 ` [PATCH 26/28] io_uring: inline struct io_comp_state Pavel Begunkov
2021-08-09 12:04 ` [PATCH 27/28] io_uring: remove extra argument for overflow flush Pavel Begunkov
2021-08-09 12:04 ` [PATCH 28/28] io_uring: inline io_poll_remove_waitqs Pavel Begunkov
2021-08-09 17:48 ` [PATCH v2 00/28] for-next patches Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6fe9646b3cb70e46aca1f58426776e368c8926b3.1628471125.git.asml.silence@gmail.com \
[email protected] \
[email protected] \
[email protected] \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox