Re: Suggestion: chain SQEs - single CQE for N chained SQEs

[Pavel Begunkov <asml.silence@gmail.com>]

On 4/18/2020 3:49 PM, H. de Vries wrote:
> Hi,
> 
> Following up on the discussion from here: https://twitter.com/i/status/1234135064323280897 and https://twitter.com/hielkedv/status/1250445647565729793
> 
> Using io_uring in event loops with IORING_FEAT_FAST_POLL can give a performance boost compared to epoll (https://twitter.com/hielkedv/status/1234135064323280897). However we need some way to manage 'in-flight' buffers, and IOSQE_BUFFER_SELECT is a solution for this. 
> 
> After a buffer has been used, it can be re-registered using IOSQE_BUFFER_SELECT by giving it a buffer ID (BID). We can also initially register a range of buffers, with e.g. BIDs 0-1000 . When buffer registration for this range is completed, this will result in a single CQE. 
> 
> However, because (network) events complete quite random, we cannot re-register a range of buffers. Maybe BIDs 3, 7, 39 and 420 are ready to be reused, but the rest of the buffers is still in-flight. So in each iteration of the event loop we need to re-register the buffer, which will result in one additional CQE for each event. The amount of CQEs to be handled in the event loop now becomes 2 times as much. If you're dealing with 200k requests per second, this can result in quite some performance loss.
> 
> If it would be possible to register multiple buffers by e.g. chaining multiple SQEs that would result in a single CQE, we could save many event loop iterations and increase performance of the event loop.

I've played with the idea before [1], it always returns only one CQE per
link, (for the last request on success, or for a failed one otherwise).
Looks like what you're suggesting. Is that so? As for me, it's just
simpler to deal with links on the user side.

It's actually in my TODO for 5.8, but depends on some changes for
sequences/drains/timeouts, that hopefully we'll push soon. We just need
to be careful to e.g. not lose CQEs with BIDs for IOSQE_BUFFER_SELECT
requests.

[1]
https://lore.kernel.org/io-uring/1a9a6022-7175-8ed3-4668-e4de3a2b9ff7@gmail.com/

-- 
Pavel Begunkov