Re: User questions: client code and SQE/CQE starvation

[Noah Goldstein <goldstein.w.n@gmail.com>]

On Wed, Jan 12, 2022 at 3:17 PM dormando <dormando@rydia.net> wrote:
>
> Hey,
>
> Been integrating io_uring in my stack which has been going well-ish.
> Wondering if you folks have seen implementations of client libraries that
> feel clean and user friendly?

libev: http://cvs.schmorp.de/libev/
has an io_uring backend: http://cvs.schmorp.de/libev/ev_iouring.c?view=markup

>
> IE: with poll/select/epoll/kqueue most client libraries (like libcurl)
> implement functions like "client_send_data(ctx, etc)", which returns
> -WANT_READ/-WANT_WRITE/etc and an fd if it needs more data to move
> forward. With the syscalls themselves externalized in io_uring I'm
> struggling to come up with abstractions I like and haven't found much
> public on a googlin'. Do any public ones exist yet?
>
> On implementing networked servers, it feels natural to do a core loop
> like:
>
>       while (1) {
>           io_uring_submit_and_wait(&t->ring, 1);
>
>           uint32_t head = 0;
>           uint32_t count = 0;
>
>           io_uring_for_each_cqe(&t->ring, head, cqe) {
>
>               event *pe = io_uring_cqe_get_data(cqe);
>               pe->callback(pe->udata, cqe);
>
>               count++;
>           }
>           io_uring_cq_advance(&t->ring, count);
>       }
>
> ... but A) you can run out of SQE's if they're generated from within
> callbacks()'s (retries, get further data, writes after reads, etc).
> B) Run out of CQE's with IORING_FEAT_NODROP and can no longer free up
> SQE's
>
> So this loop doesn't work under pressure :)
>
> I see that qemu's implementation walks an object queue, which calls
> io_uring_submit() if SQE's are exhausted. I don't recall it trying to do
> anything if submit returns EBUSY because of CQE exhaustion? I've not found
> other merged code implementing non-toy network servers and most examples
> are rewrites of CLI tooling which are much more constrained problems. Have
> I missed anything?
>
> I can make this work but a lot of solutions are double walking lists
> (fetch all CQE's into an array, advance them, then process), or not being
> able to take advantage of any of the batching API's. Hoping the
> community's got some better examples to untwist my brain a bit :)
>
> For now I have things working but want to do a cleanup pass before making
> my clients/server bits public facing.
>
> Thanks!
> -Dormando