Re: Waiting for requests completions from multiple threads

[Pavel Begunkov <asml.silence@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 1895 bytes --]

On 22/01/2020 06:16, Jens Axboe wrote:
> On 1/21/20 8:09 PM, Dmitry Sychov wrote:
>> Thank you for quick reply! Yes I understand that I need a sort of
>> serializable-level isolation
>> when accessing the rings - I hope this could be done with a simple
>> atomic cmp-add after optimistic write ring update.
> 
> That's not a bad idea, that could definitely work, and would be more
> efficient than just grabbing a lock.
> 

If I got it right, it still will spam the system with atomics.
There is another pattern to consider, (seen in the networking world a lot). Just
one thread gets completions (i.e. calls io_uring_enter()), and than distributes
jobs to a thread pool.
And for this distribution there are a lot of way to do it efficiently. E.g. see
internal techniques in java fork join merge.

That's for completion part.

> Could also be made to work quite nicely with restartable sequences. I'd
> love to see liburing grow support for smarter sharing of a ring, that's
> really where that belongs.
> 
>> Correct me if I'am wrong, but from my understanding the kernel can
>> start to pick up newly written Uring jobs
>> without waiting for the "io_uring_enter" user level call and that's
>> why we need a write barrier(so that
>> the ring state is always valid for the kernel), else "io_uring_enter"
>> could serve as a write barrier itself as well...
> 
> By uring jobs, you mean SQEs, or submission queue entries? The kernel
> only picks up what you ask it to, it won't randomly just grab entries
> from the SQ ring unless you do an io_uring_enter() and tell it to
> consume N entries. The exception is if you setup the ring with
> IORING_SETUP_SQPOLL, in which case the kernel will maintain a submission
> thread. For that case, yes, the kernel can pickup an entry as soon as
> the SQ tail is updated by the application.
> 

-- 
Pavel Begunkov


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]