public inbox for [email protected]
 help / color / mirror / Atom feed
From: Pavel Begunkov <[email protected]>
To: Jens Axboe <[email protected]>, [email protected]
Subject: [PATCH 3/3] io_uring: don't spinlock when not posting CQEs
Date: Sat, 11 Sep 2021 14:52:02 +0100	[thread overview]
Message-ID: <3a5f0436099b84f71fdc8c9bd9f21842581feaf9.1631367587.git.asml.silence@gmail.com> (raw)
In-Reply-To: <[email protected]>

When no of queued for the batch completion requests need to post an CQE,
see IOSQE_CQE_SKIP_SUCCESS, avoid grabbing ->completion_lock and other
commit/post.

Signed-off-by: Pavel Begunkov <[email protected]>
---
 fs/io_uring.c | 24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)

diff --git a/fs/io_uring.c b/fs/io_uring.c
index 172c857e8b3f..8983a5a6851a 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -317,6 +317,7 @@ struct io_submit_state {
 
 	bool			plug_started;
 	bool			need_plug;
+	bool			flush_cqes;
 
 	/*
 	 * Batch completion logic
@@ -1858,6 +1859,8 @@ static void io_req_complete_state(struct io_kiocb *req, long res,
 	req->result = res;
 	req->compl.cflags = cflags;
 	req->flags |= REQ_F_COMPLETE_INLINE;
+	if (!(req->flags & IOSQE_CQE_SKIP_SUCCESS))
+		req->ctx->submit_state.flush_cqes = true;
 }
 
 static inline void __io_req_complete(struct io_kiocb *req, unsigned issue_flags,
@@ -2354,17 +2357,19 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx)
 	int i, nr = state->compl_nr;
 	struct req_batch rb;
 
-	spin_lock(&ctx->completion_lock);
-	for (i = 0; i < nr; i++) {
-		struct io_kiocb *req = state->compl_reqs[i];
+	if (state->flush_cqes) {
+		spin_lock(&ctx->completion_lock);
+		for (i = 0; i < nr; i++) {
+			struct io_kiocb *req = state->compl_reqs[i];
 
-		if (!(req->flags & REQ_F_CQE_SKIP))
-			__io_fill_cqe(ctx, req->user_data, req->result,
-				      req->compl.cflags);
+			if (!(req->flags & REQ_F_CQE_SKIP))
+				__io_fill_cqe(ctx, req->user_data, req->result,
+					      req->compl.cflags);
+		}
+		io_commit_cqring(ctx);
+		spin_unlock(&ctx->completion_lock);
+		io_cqring_ev_posted(ctx);
 	}
-	io_commit_cqring(ctx);
-	spin_unlock(&ctx->completion_lock);
-	io_cqring_ev_posted(ctx);
 
 	io_init_req_batch(&rb);
 	for (i = 0; i < nr; i++) {
@@ -2376,6 +2381,7 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx)
 
 	io_req_free_batch_finish(ctx, &rb);
 	state->compl_nr = 0;
+	state->flush_cqes = false;
 }
 
 /*
-- 
2.33.0


  parent reply	other threads:[~2021-09-11 13:52 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-09-11 13:51 [RFC][PATCH 0/3] allow to skip CQE posting Pavel Begunkov
2021-09-11 13:52 ` [PATCH 1/3] io_uring: clean cqe filling functions Pavel Begunkov
2021-09-12 18:24   ` Hao Xu
2021-09-12 23:20     ` Pavel Begunkov
2021-09-11 13:52 ` [PATCH 2/3] io_uring: add option to skip CQE posting Pavel Begunkov
2021-09-11 13:52 ` Pavel Begunkov [this message]
2021-09-11 20:12   ` [PATCH 3/3] io_uring: don't spinlock when not posting CQEs Hao Xu
2021-09-11 21:10     ` Pavel Begunkov
2021-09-12 18:14       ` Hao Xu
2021-09-12 22:16         ` Pavel Begunkov
2021-09-12 23:49 ` [RFC][PATCH 0/3] allow to skip CQE posting Pavel Begunkov

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=3a5f0436099b84f71fdc8c9bd9f21842581feaf9.1631367587.git.asml.silence@gmail.com \
    [email protected] \
    [email protected] \
    [email protected] \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox