Re: User questions: client code and SQE/CQE starvation

[Josef <josef.grieb@gmail.com>]

sorry i accidentally pressed send message...

run out of SQE should not be problem, when
io_uring_get_sqe(https://github.com/axboe/liburing/blob/master/src/queue.c#L409)
returns a null, you can run io_uring_submit
in netty we do that automatically when its full
https://github.com/netty/netty-incubator-transport-io_uring/blob/main/transport-classes-io_uring/src/main/java/io/netty/incubator/channel/uring/IOUringSubmissionQueue.java#L117

In theory you could run out of CQE, netty io_uring approach is a
little bit different.
https://github.com/netty/netty-incubator-transport-io_uring/blob/main/transport-classes-io_uring/src/main/java/io/netty/incubator/channel/uring/IOUringCompletionQueue.java#L86
(similar to io_uring_for_each_cqe) to make sure the kernel sees that
and the process function is called here
https://github.com/netty/netty-incubator-transport-io_uring/blob/main/transport-classes-io_uring/src/main/java/io/netty/incubator/channel/uring/IOUringEventLoop.java#L203



> On Wed, 12 Jan 2022 at 22:17, dormando <dormando@rydia.net> wrote:
> >
> > Hey,
> >
> > Been integrating io_uring in my stack which has been going well-ish.
> > Wondering if you folks have seen implementations of client libraries that
> > feel clean and user friendly?
> >
> > IE: with poll/select/epoll/kqueue most client libraries (like libcurl)
> > implement functions like "client_send_data(ctx, etc)", which returns
> > -WANT_READ/-WANT_WRITE/etc and an fd if it needs more data to move
> > forward. With the syscalls themselves externalized in io_uring I'm
> > struggling to come up with abstractions I like and haven't found much
> > public on a googlin'. Do any public ones exist yet?
> >
> > On implementing networked servers, it feels natural to do a core loop
> > like:
> >
> >       while (1) {
> >           io_uring_submit_and_wait(&t->ring, 1);
> >
> >           uint32_t head = 0;
> >           uint32_t count = 0;
> >
> >           io_uring_for_each_cqe(&t->ring, head, cqe) {
> >
> >               event *pe = io_uring_cqe_get_data(cqe);
> >               pe->callback(pe->udata, cqe);
> >
> >               count++;
> >           }
> >           io_uring_cq_advance(&t->ring, count);
> >       }
> >
> > ... but A) you can run out of SQE's if they're generated from within
> > callbacks()'s (retries, get further data, writes after reads, etc).
> > B) Run out of CQE's with IORING_FEAT_NODROP and can no longer free up
> > SQE's
> >
> > So this loop doesn't work under pressure :)
> >
> > I see that qemu's implementation walks an object queue, which calls
> > io_uring_submit() if SQE's are exhausted. I don't recall it trying to do
> > anything if submit returns EBUSY because of CQE exhaustion? I've not found
> > other merged code implementing non-toy network servers and most examples
> > are rewrites of CLI tooling which are much more constrained problems. Have
> > I missed anything?
> >
> > I can make this work but a lot of solutions are double walking lists
> > (fetch all CQE's into an array, advance them, then process), or not being
> > able to take advantage of any of the batching API's. Hoping the
> > community's got some better examples to untwist my brain a bit :)
> >
> > For now I have things working but want to do a cleanup pass before making
> > my clients/server bits public facing.
> >
> > Thanks!
> > -Dormando
>
>
>
> --
> Josef Grieb

--
Josef Grieb