public inbox for [email protected]
 help / color / mirror / Atom feed
From: Jens Axboe <[email protected]>
To: [email protected], [email protected]
Cc: [email protected], [email protected], [email protected],
	Jens Axboe <[email protected]>
Subject: [PATCH 7/8] io_uring/rw: add write support for IOCB_DIO_DEFER
Date: Thu, 20 Jul 2023 12:13:09 -0600	[thread overview]
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>

If the filesystem dio handler understands IOCB_DIO_DEFER, we'll get
a kiocb->ki_complete() callback with kiocb->dio_complete set. In that
case, rather than complete the IO directly through task_work, queue
up an intermediate task_work handler that first processes this
callback and then immediately completes the request.

For XFS, this avoids a punt through a workqueue, which is a lot less
efficient and adds latency to lower queue depth (or sync) O_DIRECT
writes.

Only do this for non-polled IO, as polled IO doesn't need this kind
of deferral as it always completes within the task itself. This then
avoids a check for deferral in the polled IO completion handler.

Signed-off-by: Jens Axboe <[email protected]>
---
 io_uring/rw.c | 27 ++++++++++++++++++++++++---
 1 file changed, 24 insertions(+), 3 deletions(-)

diff --git a/io_uring/rw.c b/io_uring/rw.c
index 1bce2208b65c..f4f700383b4e 100644
--- a/io_uring/rw.c
+++ b/io_uring/rw.c
@@ -285,6 +285,14 @@ static inline int io_fixup_rw_res(struct io_kiocb *req, long res)
 
 void io_req_rw_complete(struct io_kiocb *req, struct io_tw_state *ts)
 {
+	struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw);
+
+	if (rw->kiocb.dio_complete) {
+		long res = rw->kiocb.dio_complete(rw->kiocb.private);
+
+		io_req_set_res(req, io_fixup_rw_res(req, res), 0);
+	}
+
 	io_req_io_end(req);
 
 	if (req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING)) {
@@ -300,9 +308,11 @@ static void io_complete_rw(struct kiocb *kiocb, long res)
 	struct io_rw *rw = container_of(kiocb, struct io_rw, kiocb);
 	struct io_kiocb *req = cmd_to_io_kiocb(rw);
 
-	if (__io_complete_rw_common(req, res))
-		return;
-	io_req_set_res(req, io_fixup_rw_res(req, res), 0);
+	if (!rw->kiocb.dio_complete) {
+		if (__io_complete_rw_common(req, res))
+			return;
+		io_req_set_res(req, io_fixup_rw_res(req, res), 0);
+	}
 	req->io_task_work.func = io_req_rw_complete;
 	__io_req_task_work_add(req, IOU_F_TWQ_LAZY_WAKE);
 }
@@ -916,6 +926,17 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags)
 	}
 	kiocb->ki_flags |= IOCB_WRITE;
 
+	/*
+	 * For non-polled IO, set IOCB_DIO_DEFER, stating that our handler
+	 * groks deferring the completion to task context. This isn't
+	 * necessary and useful for polled IO as that can always complete
+	 * directly.
+	 */
+	if (!(kiocb->ki_flags & IOCB_HIPRI)) {
+		kiocb->ki_flags |= IOCB_DIO_DEFER;
+		kiocb->dio_complete = NULL;
+	}
+
 	if (likely(req->file->f_op->write_iter))
 		ret2 = call_write_iter(req->file, kiocb, &s->iter);
 	else if (req->file->f_op->write)
-- 
2.40.1


  parent reply	other threads:[~2023-07-20 18:13 UTC|newest]

Thread overview: 38+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-07-20 18:13 [PATCHSET v4 0/8] Improve async iomap DIO performance Jens Axboe
2023-07-20 18:13 ` [PATCH 1/8] iomap: cleanup up iomap_dio_bio_end_io() Jens Axboe
2023-07-21  6:14   ` Christoph Hellwig
2023-07-21 15:13   ` Darrick J. Wong
2023-07-20 18:13 ` [PATCH 2/8] iomap: add IOMAP_DIO_INLINE_COMP Jens Axboe
2023-07-21  6:14   ` Christoph Hellwig
2023-07-21 15:16   ` Darrick J. Wong
2023-07-20 18:13 ` [PATCH 3/8] iomap: treat a write through cache the same as FUA Jens Axboe
2023-07-21  6:15   ` Christoph Hellwig
2023-07-21 14:04     ` Jens Axboe
2023-07-21 15:55       ` Darrick J. Wong
2023-07-21 16:03         ` Jens Axboe
2023-07-20 18:13 ` [PATCH 4/8] iomap: completed polled IO inline Jens Axboe
2023-07-21  6:16   ` Christoph Hellwig
2023-07-21 15:19   ` Darrick J. Wong
2023-07-21 21:43   ` Dave Chinner
2023-07-22  3:10     ` Jens Axboe
2023-07-22 23:05       ` Dave Chinner
2023-07-24 22:35         ` Jens Axboe
2023-07-22 16:54     ` Jens Axboe
2023-07-20 18:13 ` [PATCH 5/8] iomap: only set iocb->private for polled bio Jens Axboe
2023-07-21  6:18   ` Christoph Hellwig
2023-07-21 15:35   ` Darrick J. Wong
2023-07-21 15:37     ` Jens Axboe
2023-07-20 18:13 ` [PATCH 6/8] fs: add IOCB flags related to passing back dio completions Jens Axboe
2023-07-21  6:18   ` Christoph Hellwig
2023-07-21 15:48   ` Darrick J. Wong
2023-07-21 15:53     ` Jens Axboe
2023-07-20 18:13 ` Jens Axboe [this message]
2023-07-21  6:19   ` [PATCH 7/8] io_uring/rw: add write support for IOCB_DIO_DEFER Christoph Hellwig
2023-07-21 15:50   ` Darrick J. Wong
2023-07-21 15:53     ` Jens Axboe
2023-07-20 18:13 ` [PATCH 8/8] iomap: support IOCB_DIO_DEFER Jens Axboe
2023-07-21  6:19   ` Christoph Hellwig
2023-07-21 16:01   ` Darrick J. Wong
2023-07-21 16:30     ` Jens Axboe
2023-07-21 22:05   ` Dave Chinner
2023-07-22  3:12     ` Jens Axboe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox