Re: [PATCHSET v2 0/8] Add support for mixed sized CQEs

[Caleb Sander Mateos <csander@purestorage.com>]

On Thu, Aug 21, 2025 at 7:28 AM Jens Axboe <axboe@kernel.dk> wrote:
>
> Hi,
>
> Currently io_uring supports two modes for CQEs:
>
> 1) The standard mode, where 16b CQEs are used
> 2) Setting IORING_SETUP_CQE32, which makes all CQEs posted 32b
>
> Certain features need to pass more information back than just a single
> 32-bit res field, and hence mandate the use of CQE32 to be able to work.
> Examples of that include passthrough or other uses of ->uring_cmd() like
> socket option getting and setting, including timestamps.
>
> This patchset adds support for IORING_SETUP_CQE_MIXED, which allows
> posting both 16b and 32b CQEs on the same CQ ring. The idea here is that
> we need not waste twice the space for CQ rings, or use twice the space
> per CQE posted, if only some of the CQEs posted require the use of 32b
> CQEs. On a ring setup in CQE mixed mode, 32b posted CQEs will have
> IORING_CQE_F_32 set in cqe->flags to tell the application (or liburing)
> about this fact.

This makes a lot of sense. Have you considered something analogous for
SQEs? Requiring all SQEs to be 128 bytes when an io_uring is used for
a mix of 64-byte and 128-byte SQEs also wastes memory, probably even
more since SQEs are 4x larger than CQEs.

Best,
Caleb

>
> This is mostly trivial to support, with the corner case being attempting
> to post a 32b CQE when the ring is a single 16b CQE away from wrapping.
> As CQEs must be contigious in memory, that's simply not possible. The
> solution taken by this patchset is to add a special CQE type, which has
> IORING_CQE_F_SKIP set. This is a pad/nop CQE, which should simply be
> ignored, as it carries no information and serves no other purpose than
> to re-align the posted CQEs for ring wrap.
>
> If used with liburing, then both the 32b vs 16b postings and the skip
> are transparent.
>
> liburing support and a few basic test cases can be found here:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/axboe/liburing.git/log/?h=cqe-mixed
>
> including man page updates for the newly added setup and CQE flags, and
> the patches posted here can also be found at:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=io_uring-cqe-mix
>
> Patch 1 is just a prep patch, and patch 2 adds the cqe flags so that the
> core can be adapted before support is actually there. Patches 3 and 4
> are exactly that, and patch 5 finally adds support for the mixed mode.
> Patch 6 adds support for NOP testing of this, and patches 7/8 allow
> IORING_SETUP_CQE_MIXED for uring_cmd/zcrx which previously required
> IORING_SETUP_CQE32 to work.
>
>  Documentation/networking/iou-zcrx.rst |  2 +-
>  include/linux/io_uring_types.h        |  6 ---
>  include/trace/events/io_uring.h       |  4 +-
>  include/uapi/linux/io_uring.h         | 17 ++++++
>  io_uring/cmd_net.c                    |  3 +-
>  io_uring/fdinfo.c                     | 22 ++++----
>  io_uring/io_uring.c                   | 78 +++++++++++++++++++++------
>  io_uring/io_uring.h                   | 49 ++++++++++++-----
>  io_uring/nop.c                        | 17 +++++-
>  io_uring/register.c                   |  3 +-
>  io_uring/uring_cmd.c                  |  2 +-
>  io_uring/zcrx.c                       |  5 +-
>  12 files changed, 152 insertions(+), 56 deletions(-)
>
> Changes since v1:
> - Various little cleanups
> - Rebase on for-6.18/io_uring
>
> --
> Jens Axboe
>
>