Re: [PATCHSET v2 0/8] Add support for mixed sized CQEs

[Jens Axboe <axboe@kernel.dk>]

On 8/21/25 11:02 AM, Caleb Sander Mateos wrote:
> On Thu, Aug 21, 2025 at 7:28?AM Jens Axboe <axboe@kernel.dk> wrote:
>>
>> Hi,
>>
>> Currently io_uring supports two modes for CQEs:
>>
>> 1) The standard mode, where 16b CQEs are used
>> 2) Setting IORING_SETUP_CQE32, which makes all CQEs posted 32b
>>
>> Certain features need to pass more information back than just a single
>> 32-bit res field, and hence mandate the use of CQE32 to be able to work.
>> Examples of that include passthrough or other uses of ->uring_cmd() like
>> socket option getting and setting, including timestamps.
>>
>> This patchset adds support for IORING_SETUP_CQE_MIXED, which allows
>> posting both 16b and 32b CQEs on the same CQ ring. The idea here is that
>> we need not waste twice the space for CQ rings, or use twice the space
>> per CQE posted, if only some of the CQEs posted require the use of 32b
>> CQEs. On a ring setup in CQE mixed mode, 32b posted CQEs will have
>> IORING_CQE_F_32 set in cqe->flags to tell the application (or liburing)
>> about this fact.
> 
> This makes a lot of sense. Have you considered something analogous for
> SQEs? Requiring all SQEs to be 128 bytes when an io_uring is used for
> a mix of 64-byte and 128-byte SQEs also wastes memory, probably even
> more since SQEs are 4x larger than CQEs.

Adding Keith, as he and I literally just talked about that. My answer
was that the case is a bit different in that 32b CQEs can be useful in
cases that are predominately 16b in the first place. For example,
networking workload doing send/recv/etc and the occassional
get/setsockopt kind of thing. Or maybe a mix of normal recv and zero
copy rx.

For the SQE case, I think it's a bit different. At least the cases I
know of, it's mostly 100% 64b SQEs or 128b SQEs. I'm certainly willing
to be told otherwise! Because that is kind of the key question that
needs answering before even thinking about doing that kind of work.

But yes, it could be supported, and Keith (kind of) signed himself up to
do that. One oddity I see on that side is that while with CQE32 the
kernel can manage the potential wrap-around gap, for SQEs that's
obviously on the application to do. That could just be a NOP or
something like that, but you do need something to fill/skip that space.
I guess that could be as simple as having an opcode that is simply "skip
me", so on the kernel side it'd be easy as it'd just drop it on the
floor. You still need to app side to fill one, however, and then deal
with "oops SQ ring is now full" too.

Probably won't be too bad at all, however.

-- 
Jens Axboe