User questions: client code and SQE/CQE starvation

[dormando <dormando@rydia.net>]

Hey,

Been integrating io_uring in my stack which has been going well-ish.
Wondering if you folks have seen implementations of client libraries that
feel clean and user friendly?

IE: with poll/select/epoll/kqueue most client libraries (like libcurl)
implement functions like "client_send_data(ctx, etc)", which returns
-WANT_READ/-WANT_WRITE/etc and an fd if it needs more data to move
forward. With the syscalls themselves externalized in io_uring I'm
struggling to come up with abstractions I like and haven't found much
public on a googlin'. Do any public ones exist yet?

On implementing networked servers, it feels natural to do a core loop
like:

      while (1) {
          io_uring_submit_and_wait(&t->ring, 1);

          uint32_t head = 0;
          uint32_t count = 0;

          io_uring_for_each_cqe(&t->ring, head, cqe) {

              event *pe = io_uring_cqe_get_data(cqe);
              pe->callback(pe->udata, cqe);

              count++;
          }
          io_uring_cq_advance(&t->ring, count);
      }

... but A) you can run out of SQE's if they're generated from within
callbacks()'s (retries, get further data, writes after reads, etc).
B) Run out of CQE's with IORING_FEAT_NODROP and can no longer free up
SQE's

So this loop doesn't work under pressure :)

I see that qemu's implementation walks an object queue, which calls
io_uring_submit() if SQE's are exhausted. I don't recall it trying to do
anything if submit returns EBUSY because of CQE exhaustion? I've not found
other merged code implementing non-toy network servers and most examples
are rewrites of CLI tooling which are much more constrained problems. Have
I missed anything?

I can make this work but a lot of solutions are double walking lists
(fetch all CQE's into an array, advance them, then process), or not being
able to take advantage of any of the batching API's. Hoping the
community's got some better examples to untwist my brain a bit :)

For now I have things working but want to do a cleanup pass before making
my clients/server bits public facing.

Thanks!
-Dormando