Re: Problem about sq->khead update and ring full judgement

[Gabriel Krisman Bertazi <krisman@suse.de>]

Ferry Meng <mengferry@linux.alibaba.com> writes:

> Hi all:
>     I'm using io_uring in a program, with SQPOLL feature enabled. The
> userspace program will actively count the queue status of urings, the
> programming model is similar to:
>     {
>         sqe = io_uring_get_sqe();
>         if(sqe){
>             /* prepare next request */
>             queue_count++;
>     }
>
>
>     {
>         cqe = io_uring_peek_cqe();
>         if(cqe){
>             queue_count--;
>         }
>     }
>
>     In this way, maybe we can assume that " sq_ring_size - queue_count =
> sqe_left_we_can_alloc "?


>     Now I'm currently coming into a situation where  I/O is very high ——
> Userspace program submit lots of sqes (around 2000) at the same time,
> meanwhile  sq_ring_size is 4096. In kernel,
> __io_sq_thread->io_submit_sqes(), I see that nr(to_submit) is also over
> 2000.   At a point, a strange point comes out: Userspace program find
> sq_ring is not full, but Kernel(in fact liburing::io_uring_get_sqe)
> think sq_ring is full.
>
>     After analyzing, I find the reason is: kernel update "sq->khead"
> after submitting "all" sqes. The running of my program is : Before
> kernel update khead, userspace program has received many cqes, causing
> queue_count-- . After decreasing queue_count, user-program thinks
> sq_ring is not full, and try to start new IO requeust. As sq->head is
> not updated, io_uring_get_sqe() returns NULL.
>
>     My questions are:
>
>     1. Is userspace 'queue_count' judgement reasonable? From
> 'traditional' point of view, if we want to find sq_ring full or not, we
> can just use io_uring_get_sqe() to check

Not really. IIUC, you cannot rely on the number of CQEs to account for
submission queue space.  The SQE is consumed and the SQ pointer advanced
before the request is completed, so you can have more inflight than SQ
entry space.  Maybe what you want is io_uring_sq_space_left(3)?

-- 
Gabriel Krisman Bertazi