[PATCHSET RFC 0/3] uring_cmd copy avoidance

[Jens Axboe <axboe@kernel.dk>]

Hi,

Currently uring_cmd unconditionally copies the SQE at prep time, as it
has no other choice - the SQE data must remain stable after submit.
This can lead to excessive memory bandwidth being used for that copy,
as passthrough will often use 128b SQEs, and efficiency concerns as
those copies will potentially use quite a lot of CPU cycles as well.

As a quick test, running the current -git kernel on a box with 23
NVMe drives doing passthrough IO, memcpy() is the highest cycle user
at 9.05%, which is all off the uring_cmd prep path. The test case is
a 512b random read, which runs at 91-92M IOPS.

With these patches, memcpy() is gone from the profiles, and it runs
at 98-99M IOPS, or about 7-8% faster.

Even for lower IOPS production testing, Caleb reports that memcpy()
overhead is in the realm of 1.1% of CPU time.

This patch series attempts to mark requests in the io_uring core with
REQ_F_ASYNC_ISSUE, if we know the issue will happen out-of-line. A
helper is added to check for this, as REQ_F_FORCE_ASYNC should be
factored in as well, and I did not feel like adding ASYNC_ISSUE to
those locations.

io_uring_cmd_sqe_verify() is added on the uring_cmd side to verify
that the core did not mess this up - if that's the case, then
-EFAULT is returned for this request and a warning is triggered.

Still not fully in love with stashing an io_uring_sqe pointer from
prep to issue, would be much nicer if we kept passing it around...
Suggestions certainly more than welcome!

 include/linux/io_uring_types.h |  3 ++
 io_uring/io_uring.c            |  1 +
 io_uring/io_uring.h            |  5 +++
 io_uring/uring_cmd.c           | 68 ++++++++++++++++++++++------------
 4 files changed, 53 insertions(+), 24 deletions(-)

-- 
Jens Axboe