Re: [PATCHSET v2 0/8] Add support for mixed sized CQEs

[Caleb Sander Mateos <csander@purestorage.com>]

On Thu, Aug 21, 2025 at 10:12 AM Jens Axboe <axboe@kernel.dk> wrote:
>
> On 8/21/25 11:02 AM, Caleb Sander Mateos wrote:
> > On Thu, Aug 21, 2025 at 7:28?AM Jens Axboe <axboe@kernel.dk> wrote:
> >>
> >> Hi,
> >>
> >> Currently io_uring supports two modes for CQEs:
> >>
> >> 1) The standard mode, where 16b CQEs are used
> >> 2) Setting IORING_SETUP_CQE32, which makes all CQEs posted 32b
> >>
> >> Certain features need to pass more information back than just a single
> >> 32-bit res field, and hence mandate the use of CQE32 to be able to work.
> >> Examples of that include passthrough or other uses of ->uring_cmd() like
> >> socket option getting and setting, including timestamps.
> >>
> >> This patchset adds support for IORING_SETUP_CQE_MIXED, which allows
> >> posting both 16b and 32b CQEs on the same CQ ring. The idea here is that
> >> we need not waste twice the space for CQ rings, or use twice the space
> >> per CQE posted, if only some of the CQEs posted require the use of 32b
> >> CQEs. On a ring setup in CQE mixed mode, 32b posted CQEs will have
> >> IORING_CQE_F_32 set in cqe->flags to tell the application (or liburing)
> >> about this fact.
> >
> > This makes a lot of sense. Have you considered something analogous for
> > SQEs? Requiring all SQEs to be 128 bytes when an io_uring is used for
> > a mix of 64-byte and 128-byte SQEs also wastes memory, probably even
> > more since SQEs are 4x larger than CQEs.
>
> Adding Keith, as he and I literally just talked about that. My answer
> was that the case is a bit different in that 32b CQEs can be useful in
> cases that are predominately 16b in the first place. For example,
> networking workload doing send/recv/etc and the occassional
> get/setsockopt kind of thing. Or maybe a mix of normal recv and zero
> copy rx.
>
> For the SQE case, I think it's a bit different. At least the cases I
> know of, it's mostly 100% 64b SQEs or 128b SQEs. I'm certainly willing
> to be told otherwise! Because that is kind of the key question that
> needs answering before even thinking about doing that kind of work.

We certainly have a use case that mixes the two on the same io_uring:
ublk commit/buffer register/unregister commands (64 byte SQEs) and
NVMe passthru commands (128 byte SQEs). I could also imagine an
application issuing both normal read/write commands and NVMe passthru
commands. But you're probably right that this isn't a super common use
case.

>
> But yes, it could be supported, and Keith (kind of) signed himself up to
> do that. One oddity I see on that side is that while with CQE32 the
> kernel can manage the potential wrap-around gap, for SQEs that's
> obviously on the application to do. That could just be a NOP or
> something like that, but you do need something to fill/skip that space.
> I guess that could be as simple as having an opcode that is simply "skip
> me", so on the kernel side it'd be easy as it'd just drop it on the
> floor. You still need to app side to fill one, however, and then deal
> with "oops SQ ring is now full" too.

Sure, of course userspace would need to handle a misaligned big SQE at
the end of the SQ analogously to mixed CQE sizes. I assume liburing
should be able to do that mostly transparently, that logic could all
be encapsulated by io_uring_get_sqe().

Best,
Caleb

>
> Probably won't be too bad at all, however.
>
> --
> Jens Axboe