From: Pavel Begunkov <[email protected]>
To: Jens Axboe <[email protected]>, [email protected]
Subject: [PATCH v2 1/9] io_uring: optimise iowq refcounting
Date: Sun, 15 Aug 2021 10:40:18 +0100 [thread overview]
Message-ID: <2d53f4449faaf73b4a4c5de667fc3c176d974860.1628981736.git.asml.silence@gmail.com> (raw)
In-Reply-To: <[email protected]>
If a requests is forwarded into io-wq, there is a good chance it hasn't
been refcounted yet and we can save one req_ref_get() by setting the
refcount number to the right value directly.
Signed-off-by: Pavel Begunkov <[email protected]>
---
fs/io_uring.c | 25 ++++++++++++++++---------
1 file changed, 16 insertions(+), 9 deletions(-)
diff --git a/fs/io_uring.c b/fs/io_uring.c
index 51c4166f68b5..761bfb56ed3b 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -1115,14 +1115,19 @@ static inline void req_ref_get(struct io_kiocb *req)
atomic_inc(&req->refs);
}
-static inline void io_req_refcount(struct io_kiocb *req)
+static inline void __io_req_set_refcount(struct io_kiocb *req, int nr)
{
if (!(req->flags & REQ_F_REFCOUNT)) {
req->flags |= REQ_F_REFCOUNT;
- atomic_set(&req->refs, 1);
+ atomic_set(&req->refs, nr);
}
}
+static inline void io_req_set_refcount(struct io_kiocb *req)
+{
+ __io_req_set_refcount(req, 1);
+}
+
static inline void io_req_set_rsrc_node(struct io_kiocb *req)
{
struct io_ring_ctx *ctx = req->ctx;
@@ -1304,8 +1309,8 @@ static struct io_kiocb *__io_prep_linked_timeout(struct io_kiocb *req)
return NULL;
/* linked timeouts should have two refs once prep'ed */
- io_req_refcount(req);
- io_req_refcount(nxt);
+ io_req_set_refcount(req);
+ io_req_set_refcount(nxt);
req_ref_get(nxt);
nxt->timeout.head = req;
@@ -5231,7 +5236,7 @@ static int io_arm_poll_handler(struct io_kiocb *req)
req->apoll = apoll;
req->flags |= REQ_F_POLLED;
ipt.pt._qproc = io_async_queue_proc;
- io_req_refcount(req);
+ io_req_set_refcount(req);
ret = __io_arm_poll_handler(req, &apoll->poll, &ipt, mask,
io_async_wake);
@@ -5419,7 +5424,7 @@ static int io_poll_add_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe
if (flags & ~IORING_POLL_ADD_MULTI)
return -EINVAL;
- io_req_refcount(req);
+ io_req_set_refcount(req);
poll->events = io_poll_parse_events(sqe, flags);
return 0;
}
@@ -6311,9 +6316,11 @@ static void io_wq_submit_work(struct io_wq_work *work)
struct io_kiocb *timeout;
int ret = 0;
- io_req_refcount(req);
- /* will be dropped by ->io_free_work() after returning to io-wq */
- req_ref_get(req);
+ /* one will be dropped by ->io_free_work() after returning to io-wq */
+ if (!(req->flags & REQ_F_REFCOUNT))
+ __io_req_set_refcount(req, 2);
+ else
+ req_ref_get(req);
timeout = io_prep_linked_timeout(req);
if (timeout)
--
2.32.0
next prev parent reply other threads:[~2021-08-15 9:41 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-15 9:40 [PATCH v2 for-next 0/5] 5.15 cleanups and optimisations Pavel Begunkov
2021-08-15 9:40 ` Pavel Begunkov [this message]
2021-08-15 9:40 ` [PATCH v2 2/9] io_uring: don't inflight-track linked timeouts Pavel Begunkov
2021-08-15 9:40 ` [PATCH v2 3/9] io_uring: optimise initial ltimeout refcounting Pavel Begunkov
2021-08-15 9:40 ` [PATCH v2 4/9] io_uring: kill not necessary resubmit switch Pavel Begunkov
2021-08-15 9:40 ` [PATCH v2 5/9] io_uring: deduplicate cancellation code Pavel Begunkov
2021-08-15 9:40 ` [PATCH v2 6/9] io_uring: kill REQ_F_LTIMEOUT_ACTIVE Pavel Begunkov
2021-08-15 9:40 ` [PATCH v2 7/9] io_uring: simplify io_prep_linked_timeout Pavel Begunkov
2021-08-15 9:40 ` [PATCH v2 8/9] io_uring: cancel not-armed linked touts separately Pavel Begunkov
2021-08-15 9:40 ` [PATCH v2 9/9] io_uring: optimise io_prep_linked_timeout() Pavel Begunkov
2021-08-15 15:17 ` [PATCH v2 for-next 0/5] 5.15 cleanups and optimisations Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2d53f4449faaf73b4a4c5de667fc3c176d974860.1628981736.git.asml.silence@gmail.com \
[email protected] \
[email protected] \
[email protected] \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox