[GIT PULL v2] Improve iomap async dio performance

[GIT PULL v2] Improve iomap async dio performance

[Jens Axboe]

Hi,

Here's the pull request for improving async dio performance with
iomap. Contains a few generic cleanups as well, but the meat of it
is described in the tagged commit message below.

Please pull for 6.6!


The following changes since commit ccff6d117d8dc8d8d86e8695a75e5f8b01e573bf:

  Merge tag 'perf-tools-fixes-for-v6.5-1-2023-07-18' of git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools (2023-07-18 14:51:29 -0700)

are available in the Git repository at:

  git://git.kernel.dk/linux.git tags/xfs-async-dio.6-2023-07-28

for you to fetch changes up to 634aae6c33338308f92eb137b0ea4833457c2ce7:

  iomap: support IOCB_DIO_CALLER_COMP (2023-07-25 07:47:20 -0600)

----------------------------------------------------------------
Improve iomap/xfs async dio write performance

iomap always punts async dio write completions to a workqueue, which has
a cost in terms of efficiency (now you need an unrelated worker to
process it) and latency (now you're bouncing a completion through an
async worker, which is a classic slowdown scenario).

io_uring handles IRQ completions via task_work, and for writes that
don't need to do extra IO at completion time, we can safely complete
them inline from that. This patchset adds IOCB_DIO_CALLER_COMP, which an
IO issuer can set to inform the completion side that any extra work that
needs doing for that completion can be punted to a safe task context.

The iomap dio completion will happen in hard/soft irq context, and we
need a saner context to process these completions. IOCB_DIO_CALLER_COMP
is added, which can be set in a struct kiocb->ki_flags by the issuer. If
the completion side of the iocb handling understands this flag, it can
choose to set a kiocb->dio_complete() handler and just call ki_complete
from IRQ context. The issuer must then ensure that this callback is
processed from a task. io_uring punts IRQ completions to task_work
already, so it's trivial wire it up to run more of the completion before
posting a CQE. This is good for up to a 37% improvement in
throughput/latency for low queue depth IO, patch 5 has the details.

If we need to do real work at completion time, iomap will clear the
IOMAP_DIO_CALLER_COMP flag.

This work came about when Andres tested low queue depth dio writes for
postgres and compared it to doing sync dio writes, showing that the
async processing slows us down a lot.

----------------------------------------------------------------
Jens Axboe (8):
      iomap: cleanup up iomap_dio_bio_end_io()
      iomap: use an unsigned type for IOMAP_DIO_* defines
      iomap: treat a write through cache the same as FUA
      iomap: only set iocb->private for polled bio
      iomap: add IOMAP_DIO_INLINE_COMP
      fs: add IOCB flags related to passing back dio completions
      io_uring/rw: add write support for IOCB_DIO_CALLER_COMP
      iomap: support IOCB_DIO_CALLER_COMP

 fs/iomap/direct-io.c | 163 ++++++++++++++++++++++++++++++++++++++-------------
 include/linux/fs.h   |  35 ++++++++++-
 io_uring/rw.c        |  27 ++++++++-
 3 files changed, 180 insertions(+), 45 deletions(-)

-- 
Jens Axboe

Re: [GIT PULL v2] Improve iomap async dio performance

[Jens Axboe]

On 7/28/23 7:47 AM, Jens Axboe wrote:
> Hi,
> 
> Here's the pull request for improving async dio performance with
> iomap. Contains a few generic cleanups as well, but the meat of it
> is described in the tagged commit message below.
> 
> Please pull for 6.6!
> 
> 
> The following changes since commit ccff6d117d8dc8d8d86e8695a75e5f8b01e573bf:
> 
>   Merge tag 'perf-tools-fixes-for-v6.5-1-2023-07-18' of git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools (2023-07-18 14:51:29 -0700)
> 
> are available in the Git repository at:
> 
>   git://git.kernel.dk/linux.git tags/xfs-async-dio.6-2023-07-28

Darrick, I collected Dave's reviewed-by on all the patches. Same pull
request as this one, only the commit meta data changed:

   git://git.kernel.dk/linux.git tags/xfs-async-dio.6-2023-08-01

Please pull!

-- 
Jens Axboe