Re: Any performance gains from using per thread(thread local) urings?

[Dmitry Sychov <dmitry.sychov@gmail.com>]

> E.g. 100+ cores hammering on a spinlock/mutex protecting an SQ wouldn't do any good.

Its possible to mitigate the hammering by using proxy buffer - instead
of spinning, the particular thread
could add the next entry into the buffer through XADD instead, and
another thread currently holding an exclusive
lock could in turn check this buffer and batch-submit all pending
entries to SQ before leasing SQ mutex.

> will be offloaded to an internal thread pool (aka io-wq), which is per io_uring by default, but can be shared if specified.

Well, thats sounds like mumbo jumbo to me, does this mean that the
kernel holds and internal pool of threads to
perform uring tasks independent to the number of user urings?

If there are multiple kernel work flows bound to corresponding uring
setups the issue with threads starvation could exist if they do not
actively steal from each other SQs.

And starvation costs could be greater than allowing for multiple
threads to dig into one uring queue, even under the exclusive lock.

> And there a lot of details, probably worth of a separate write-up.

I've reread io_uring.pdf and there are not much tech details on the
inner implementation of uring to try to apply best practices and to
avoid noob questions like mine.



On Wed, May 13, 2020 at 7:03 PM Pavel Begunkov <asml.silence@gmail.com> wrote:
>
> On 13/05/2020 17:22, Dmitry Sychov wrote:
> > Anyone could shed some light on the inner implementation of uring please? :)
>
> It really depends on the workload, hardware, etc.
>
> io_uring instances are intended to be independent, and each have one CQ and SQ.
> The main user's concern should be synchronisation (in userspace) on CQ+SQ. E.g.
> 100+ cores hammering on a spinlock/mutex protecting an SQ wouldn't do any good.
>
> Everything that can't be inline completed\submitted during io_urng_enter(), will
> be offloaded to an internal thread pool (aka io-wq), which is per io_uring by
> default, but can be shared if specified. There are pros and cons, but I'd
> recommend first to share a single io-wq, and then experiment and tune.
>
> Also, in-kernel submission is not instantaneous and done by only thread at any
> moment. Single io_uring may bottleneck you there or add high latency in some cases.
>
> And there a lot of details, probably worth of a separate write-up.
>
> >
> > Specifically how well kernel scales with the increased number of user
> > created urings?
>
> Should scale well, especially for rw. Just don't overthrow the kernel with
> threads from dozens of io-wqs.
>
> >
> >> If kernel implementation will change from single to multiple queues,
> >> user space is already prepared for this change.
> >
> > Thats +1 for per-thread urings. An expectation for the kernel to
> > become better and better in multiple urings scaling in the future.
> >
> > On Wed, May 13, 2020 at 4:52 PM Sergiy Yevtushenko
> > <sergiy.yevtushenko@gmail.com> wrote:
> >>
> >> Completely agree. Sharing state should be avoided as much as possible.
> >> Returning to original question: I believe that uring-per-thread scheme is better regardless from how queue is managed inside the kernel.
> >> - If there is only one queue inside the kernel, then it's more efficient to perform multiplexing/demultiplexing requests in kernel space
> >> - If there are several queues inside the kernel, then user space code better matches kernel-space code.
> >> - If kernel implementation will change from single to multiple queues, user space is already prepared for this change.
> >>
> >>
> >> On Wed, May 13, 2020 at 3:30 PM Mark Papadakis <markuspapadakis@icloud.com> wrote:
> >>>
> >>>
> >>>
> >>>> On 13 May 2020, at 4:15 PM, Dmitry Sychov <dmitry.sychov@gmail.com> wrote:
> >>>>
> >>>> Hey Mark,
> >>>>
> >>>> Or we could share one SQ and one CQ between multiple threads(bound by
> >>>> the max number of CPU cores) for direct read/write access using very
> >>>> light mutex to sync.
> >>>>
> >>>> This also solves threads starvation issue  - thread A submits the job
> >>>> into shared SQ while thread B both collects and _processes_ the result
> >>>> from the shared CQ instead of waiting on his own unique CQ for next
> >>>> completion event.
> >>>>
> >>>
> >>>
> >>> Well, if the SQ submitted by A and its matching CQ is consumed by B, and A will need access to that CQ because it is tightly coupled to state it owns exclusively(for example), or other reasons, then you’d still need to move that CQ from B back to A, or share it somehow, which seems expensive-is.
> >>>
> >>> It depends on what kind of roles your threads have though; I am personally very much against sharing state between threads unless there a really good reason for it.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> On Wed, May 13, 2020 at 2:56 PM Mark Papadakis
> >>>> <markuspapadakis@icloud.com> wrote:
> >>>>>
> >>>>> For what it’s worth, I am (also) using using multiple “reactor” (i.e event driven) cores, each associated with one OS thread, and each reactor core manages its own io_uring context/queues.
> >>>>>
> >>>>> Even if scheduling all SQEs through a single io_uring SQ — by e.g collecting all such SQEs in every OS thread and then somehow “moving” them to the one OS thread that manages the SQ so that it can enqueue them all -- is very cheap, you ‘d still need to drain the CQ from that thread and presumably process those CQEs in a single OS thread, which will definitely be more work than having each reactor/OS thread dequeue CQEs for SQEs that itself submitted.
> >>>>> You could have a single OS thread just for I/O and all other threads could do something else but you’d presumably need to serialize access/share state between them and the one OS thread for I/O which maybe a scalability bottleneck.
> >>>>>
> >>>>> ( if you are curious, you can read about it here https://medium.com/@markpapadakis/building-high-performance-services-in-2020-e2dea272f6f6 )
> >>>>>
> >>>>> If you experiment with the various possible designs though, I’d love it if you were to share your findings.
> >>>>>
> >>>>> —
> >>>>> @markpapapdakis
> >>>>>
> >>>>>
> >>>>>> On 13 May 2020, at 2:01 PM, Dmitry Sychov <dmitry.sychov@gmail.com> wrote:
> >>>>>>
> >>>>>> Hi Hielke,
> >>>>>>
> >>>>>>> If you want max performance, what you generally will see in non-blocking servers is one event loop per core/thread.
> >>>>>>> This means one ring per core/thread. Of course there is no simple answer to this.
> >>>>>>> See how thread-based servers work vs non-blocking servers. E.g. Apache vs Nginx or Tomcat vs Netty.
> >>>>>>
> >>>>>> I think a lot depends on the internal uring implementation. To what
> >>>>>> degree the kernel is able to handle multiple urings independently,
> >>>>>> without much congestion points(like updates of the same memory
> >>>>>> locations from multiple threads), thus taking advantage of one ring
> >>>>>> per CPU core.
> >>>>>>
> >>>>>> For example, if the tasks from multiple rings are later combined into
> >>>>>> single input kernel queue (effectively forming a congestion point) I
> >>>>>> see
> >>>>>> no reason to use exclusive ring per core in user space.
> >>>>>>
> >>>>>> [BTW in Windows IOCP is always one input+output queue for all(active) threads].
> >>>>>>
> >>>>>> Also we could pop out multiple completion events from a single CQ at
> >>>>>> once to spread the handling to cores-bound threads .
> >>>>>>
> >>>>>> I thought about one uring per core at first, but now I'am not sure -
> >>>>>> maybe the kernel devs have something to add to the discussion?
> >>>>>>
> >>>>>> P.S. uring is the main reason I'am switching from windows to linux dev
> >>>>>> for client-sever app so I want to extract the max performance possible
> >>>>>> out of this new exciting uring stuff. :)
> >>>>>>
> >>>>>> Thanks, Dmitry
> >>>>>
> >>>
>
> --
> Pavel Begunkov