From: Jens Axboe <[email protected]>
To: [email protected]
Cc: Pavel Begunkov <[email protected]>, Jens Axboe <[email protected]>
Subject: [PATCH 19/27] io_uring: add io_disarm_next() helper
Date: Wed, 10 Mar 2021 15:43:50 -0700 [thread overview]
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
From: Pavel Begunkov <[email protected]>
A preparation patch placing all preparations before extracting a next
request into a separate helper io_disarm_next().
Also, don't spuriously do ev_posted in a rare case where REQ_F_FAIL_LINK
is set but there are no requests linked (i.e. after cancelling a linked
timeout or setting IOSQE_IO_LINK on a last request of a submission
batch).
Signed-off-by: Pavel Begunkov <[email protected]>
Link: https://lore.kernel.org/r/44ecff68d6b47e1c4e6b891bdde1ddc08cfc3590.1615250156.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <[email protected]>
---
fs/io_uring.c | 68 ++++++++++++++++++++++++++-------------------------
1 file changed, 35 insertions(+), 33 deletions(-)
diff --git a/fs/io_uring.c b/fs/io_uring.c
index 3299807894ec..cc9a2cc95608 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -1705,15 +1705,11 @@ static inline void io_remove_next_linked(struct io_kiocb *req)
nxt->link = NULL;
}
-static void io_kill_linked_timeout(struct io_kiocb *req)
+static bool io_kill_linked_timeout(struct io_kiocb *req)
+ __must_hold(&req->ctx->completion_lock)
{
- struct io_ring_ctx *ctx = req->ctx;
- struct io_kiocb *link;
+ struct io_kiocb *link = req->link;
bool cancelled = false;
- unsigned long flags;
-
- spin_lock_irqsave(&ctx->completion_lock, flags);
- link = req->link;
/*
* Can happen if a linked timeout fired and link had been like
@@ -1728,50 +1724,48 @@ static void io_kill_linked_timeout(struct io_kiocb *req)
ret = hrtimer_try_to_cancel(&io->timer);
if (ret != -1) {
io_cqring_fill_event(link, -ECANCELED);
- io_commit_cqring(ctx);
+ io_put_req_deferred(link, 1);
cancelled = true;
}
}
req->flags &= ~REQ_F_LINK_TIMEOUT;
- spin_unlock_irqrestore(&ctx->completion_lock, flags);
-
- if (cancelled) {
- io_cqring_ev_posted(ctx);
- io_put_req(link);
- }
+ return cancelled;
}
-
static void io_fail_links(struct io_kiocb *req)
+ __must_hold(&req->ctx->completion_lock)
{
- struct io_kiocb *link, *nxt;
- struct io_ring_ctx *ctx = req->ctx;
- unsigned long flags;
+ struct io_kiocb *nxt, *link = req->link;
- spin_lock_irqsave(&ctx->completion_lock, flags);
- link = req->link;
req->link = NULL;
-
while (link) {
nxt = link->link;
link->link = NULL;
trace_io_uring_fail_link(req, link);
io_cqring_fill_event(link, -ECANCELED);
-
io_put_req_deferred(link, 2);
link = nxt;
}
- io_commit_cqring(ctx);
- spin_unlock_irqrestore(&ctx->completion_lock, flags);
+}
- io_cqring_ev_posted(ctx);
+static bool io_disarm_next(struct io_kiocb *req)
+ __must_hold(&req->ctx->completion_lock)
+{
+ bool posted = false;
+
+ if (likely(req->flags & REQ_F_LINK_TIMEOUT))
+ posted = io_kill_linked_timeout(req);
+ if (unlikely(req->flags & REQ_F_FAIL_LINK)) {
+ posted |= (req->link != NULL);
+ io_fail_links(req);
+ }
+ return posted;
}
static struct io_kiocb *__io_req_find_next(struct io_kiocb *req)
{
- if (req->flags & REQ_F_LINK_TIMEOUT)
- io_kill_linked_timeout(req);
+ struct io_kiocb *nxt;
/*
* If LINK is set, we have dependent requests in this chain. If we
@@ -1779,14 +1773,22 @@ static struct io_kiocb *__io_req_find_next(struct io_kiocb *req)
* dependencies to the next request. In case of failure, fail the rest
* of the chain.
*/
- if (likely(!(req->flags & REQ_F_FAIL_LINK))) {
- struct io_kiocb *nxt = req->link;
+ if (req->flags & (REQ_F_LINK_TIMEOUT | REQ_F_FAIL_LINK)) {
+ struct io_ring_ctx *ctx = req->ctx;
+ unsigned long flags;
+ bool posted;
- req->link = NULL;
- return nxt;
+ spin_lock_irqsave(&ctx->completion_lock, flags);
+ posted = io_disarm_next(req);
+ if (posted)
+ io_commit_cqring(req->ctx);
+ spin_unlock_irqrestore(&ctx->completion_lock, flags);
+ if (posted)
+ io_cqring_ev_posted(ctx);
}
- io_fail_links(req);
- return NULL;
+ nxt = req->link;
+ req->link = NULL;
+ return nxt;
}
static inline struct io_kiocb *io_req_find_next(struct io_kiocb *req)
--
2.30.2
next prev parent reply other threads:[~2021-03-10 22:45 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-10 22:43 [PATCHSET for 5.12] Patches queued up for 5.12 Jens Axboe
2021-03-10 22:43 ` [PATCH 01/27] io-wq: fix race in freeing 'wq' and worker access Jens Axboe
2021-03-10 22:43 ` [PATCH 02/27] io-wq: always track creds for async issue Jens Axboe
2021-03-10 22:43 ` [PATCH 03/27] io_uring: make del_task_file more forgiving Jens Axboe
2021-03-10 22:43 ` [PATCH 04/27] io_uring: introduce ctx to tctx back map Jens Axboe
2021-03-10 22:43 ` [PATCH 05/27] io_uring: do ctx initiated file note removal Jens Axboe
2021-03-10 22:43 ` [PATCH 06/27] io_uring: don't take task ring-file notes Jens Axboe
2021-03-10 22:43 ` [PATCH 07/27] io_uring: index io_uring->xa by ctx not file Jens Axboe
2021-03-10 22:43 ` [PATCH 08/27] io_uring: warn when ring exit takes too long Jens Axboe
2021-03-10 22:43 ` [PATCH 09/27] io_uring: cancel reqs of all iowq's on ring exit Jens Axboe
2021-03-10 22:43 ` [PATCH 10/27] io-wq: warn on creating manager while exiting Jens Axboe
2021-03-10 22:43 ` [PATCH 11/27] io_uring: run __io_sq_thread() with the initial creds from io_uring_setup() Jens Axboe
2021-03-10 22:43 ` [PATCH 12/27] io_uring: kill io_sq_thread_fork() and return -EOWNERDEAD if the sq_thread is gone Jens Axboe
2021-03-10 22:43 ` [PATCH 13/27] io_uring: SQPOLL parking fixes Jens Axboe
2021-03-10 22:43 ` [PATCH 14/27] io_uring: fix unrelated ctx reqs cancellation Jens Axboe
2021-03-10 22:43 ` [PATCH 15/27] io_uring: clean R_DISABLED startup mess Jens Axboe
2021-03-10 22:43 ` [PATCH 16/27] io_uring: Convert personality_idr to XArray Jens Axboe
2021-03-10 22:43 ` [PATCH 17/27] io-wq: remove unused 'user' member of io_wq Jens Axboe
2021-03-10 22:43 ` [PATCH 18/27] io_uring: fix io_sq_offload_create error handling Jens Axboe
2021-03-10 22:43 ` Jens Axboe [this message]
2021-03-10 22:43 ` [PATCH 20/27] io_uring: fix complete_post races for linked req Jens Axboe
2021-03-10 22:43 ` [PATCH 21/27] io-wq: fix ref leak for req in case of exit cancelations Jens Axboe
2021-03-10 22:43 ` [PATCH 22/27] io_uring: move all io_kiocb init early in io_init_req() Jens Axboe
2021-03-10 22:43 ` [PATCH 23/27] io_uring: remove unneeded variable 'ret' Jens Axboe
2021-03-10 22:43 ` [PATCH 24/27] io_uring: always wait for sqd exited when stopping SQPOLL thread Jens Axboe
2021-03-10 22:43 ` [PATCH 25/27] kernel: make IO threads unfreezable by default Jens Axboe
2021-03-10 22:43 ` [PATCH 26/27] io_uring: fix invalid ctx->sq_thread_idle Jens Axboe
2021-03-10 22:43 ` [PATCH 27/27] io_uring: remove indirect ctx into sqo injection Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox