Re: CQE repeats the first item?

[Jens Axboe <axboe@kernel.dk>]

On 10/5/25 3:54 PM, Jacob Thompson wrote:
> On Sun, Oct 05, 2025 at 02:56:05PM -0600, Jens Axboe wrote:
>> On 10/5/25 2:21 PM, Jacob Thompson wrote:
>>> I'm doing something wrong and I wanted to know if anyone knows what I
>>> did wrong from the description I'm using syscalls to call
>>> io_uring_setup and io_uring_enter. I managed to submit 1 item without
>>> an issue but any more gets me the first item over and over again. In
>>> my test I did a memset -1 on cqes and sqes, I memset 0 the first ten
>>> sqes with different user_data (0x1234 + i), and I used the opcode
>>> IORING_OP_NOP. I called "io_uring_enter(fd, 10, 0,
>>> IORING_ENTER_SQ_WAKEUP, 0)" and looked at the cq. Item 11 has the
>>> user_data as '18446744073709551615' which is correct, but the first 10
>>> all has user_data be 0x1234 which is weird AF since only one item has
>>> that user_data and I submited 10 I considered maybe the debugger was
>>> giving me incorrect values so I tried printing the user data in a
>>> loop, I have no idea why the first one repeats 10 times. I only called
>>> enter once
>>>
>>> Id is 4660
>>> Id is 4660
>>> Id is 4660
>>> Id is 4660
>>> Id is 4660
>>> Id is 4660
>>> Id is 4660
>>> Id is 4660
>>> Id is 4660
>>> Id is 4660
>>> Id is 18446744073709551615
>>
>> You're presumably not updating your side of the CQ ring correctly, see
>> what liburing does when you call io_uring_cqe_seen(). If that's not it,
>> then you're probably mishandling something else and an example would be
>> useful as otherwise I'd just be guessing. There's really not much to go
>> from in this report.
>>
>> -- 
>> Jens Axboe
> 
> I tried reproducing it in a smaller file. Assume I did everything wrong but somehow I seem to get results and they're not correct.
> 
> The codebase I'd like to use this in has very little activity (could go seconds without a single syscall), then execute a few hundreds-thousand (which I like to be async).
> SQPOLL sounds like the one best for my usecase. You can see I updated the sq tail before enter and I used IORING_ENTER_SQ_WAKEUP + slept for a second.
> The sq tail isn't zero which means I have results? and you can see its 10 of the same user_data
> 
> cq head is 0 enter result was 10
> 1234 0
> 1234 0
> 1234 0
> 1234 0
> 1234 0
> 1234 0
> 1234 0
> 1234 0
> 1234 0
> 1234 0
> FFFFFFFF -1

I looked at your test code, and you're setting up 10 NOP requests with
userdata == 0x1234, and hence you get 10 completions with that userdata.
For some reason you iterate 11 CQEs, which means your last one is the one
that you already filled with -1.

In other words, it very much looks like it's working as it should. Any
reason why you're using the raw interface rather than liburing? All of
this seems to be not understanding how the ring works, and liburing
helps isolate you from that. The SQ ring doesn't tell you anything about
whether you have results (CQEs?), the difference between the SQ head and
tail just tell you if there's something to submit. The CQ ring head and
tail would tell you if there are CQEs to reap or not.

-- 
Jens Axboe