Re: Waiting for requests completions from multiple threads

[Dmitry Sychov <dmitry.sychov@gmail.com>]

Thank you for quick reply! Yes I understand that I need a sort of
serializable-level isolation
when accessing the rings - I hope this could be done with a simple
atomic cmp-add after optimistic write ring update.

Correct me if I'am wrong, but from my understanding the kernel can
start to pick up newly written Uring jobs
without waiting for the "io_uring_enter" user level call and that's
why we need a write barrier(so that
the ring state is always valid for the kernel), else "io_uring_enter"
could serve as a write barrier itself as well...


On Wed, Jan 22, 2020 at 5:51 AM Jens Axboe <axboe@kernel.dk> wrote:
>
> On 1/21/20 7:45 PM, Dmitry Sychov wrote:
> > Really nice work, I have a question though.
> >
> > It is possible to efficiently wait for request completions
> > from multiple threads?
> >
> > Like, two threads are entering
> > " io_uring_enter" both with min_complete=1 while the completion ring
> > holds 2 events - will the first one goes to thread 1 and the second
> > one to thread 2?
> >
> > I just do not understand exactly the best way to scale this api into
> > multiple threads... with IOCP for example is is perfectly clear.
>
> You can have two threads waiting on events, and yes, if they each ask to
> wait for 1 event and 2 completes, then they will both get woken up. But
> the wait side doesn't give you any events, it merely tells you of the
> availability of them. When each thread is woken up and goes back to
> userspace, it'll have to reap an event from the ring. If each thread
> reaps one event from the CQ ring, then you're done.
>
> You need synchronization on the CQ ring side in userspace if you want
> two rings to access the CQ ring. That is not needed for entering the
> kernel, only when the application reads a CQE (or modifies the ring), if
> you can have more than one thread modifying the CQ ring. The exact same
> is true on the SQ ring side.
>
> --
> Jens Axboe
>