From: Artyom Pavlov <[email protected]>
To: Jens Axboe <[email protected]>, [email protected]
Subject: Re: Sending CQE to a different ring
Date: Thu, 10 Mar 2022 05:11:28 +0300 [thread overview]
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
10.03.2022 04:36, Jens Axboe wrote:
> On 3/9/22 4:49 PM, Artyom Pavlov wrote:
>> Greetings!
>>
>> A common approach for multi-threaded servers is to have a number of
>> threads equal to a number of cores and launch a separate ring in each
>> one. AFAIK currently if we want to send an event to a different ring,
>> we have to write-lock this ring, create SQE, and update the index
>> ring. Alternatively, we could use some kind of user-space message
>> passing.
>>
>> Such approaches are somewhat inefficient and I think it can be solved
>> elegantly by updating the io_uring_sqe type to allow accepting fd of a
>> ring to which CQE must be sent by kernel. It can be done by
>> introducing an IOSQE_ flag and using one of currently unused padding
>> u64s.
>>
>> Such feature could be useful for load balancing and message passing
>> between threads which would ride on top of io-uring, i.e. you could
>> send NOP with user_data pointing to a message payload.
>
> So what you want is a NOP with 'fd' set to the fd of another ring, and
> that nop posts a CQE on that other ring? I don't think we'd need IOSQE
> flags for that, we just need a NOP that supports that. I see a few ways
> of going about that:
>
> 1) Add a new 'NOP' that takes an fd, and validates that that fd is an
> io_uring instance. It can then grab the completion lock on that ring
> and post an empty CQE.
>
> 2) We add a FEAT flag saying NOP supports taking an 'fd' argument, where
> 'fd' is another ring. Posting CQE same as above.
>
> 3) We add a specific opcode for this. Basically the same as #2, but
> maybe with a more descriptive name than NOP.
>
> Might make sense to pair that with a CQE flag or something like that, as
> there's no specific user_data that could be used as it doesn't match an
> existing SQE that has been issued. IORING_CQE_F_WAKEUP for example.
> Would be applicable to all the above cases.
>
> I kind of like #3 the best. Add a IORING_OP_RING_WAKEUP command, require
> that sqe->fd point to a ring (could even be the ring itself, doesn't
> matter). And add IORING_CQE_F_WAKEUP as a specific flag for that.
>
No, ideally I would like to be able to send any type of SQE to a
different ring. For example, if I see that the current ring is
overloaded, I can create exactly the same SQEs as during usual
operation, but with a changed recipient ring.
Your approach with a new "sendable" NOP will allow to emulate it in
user-space, but it will involve unnecessary ring round-trip and will be
a bit less pleasant in user code, e.g. we would need to encode a
separate state "the task is being sent to a different ring" instead of
simply telling io-uring "read data and report CQE on this ring" without
any intermediate states.
next prev parent reply other threads:[~2022-03-10 2:11 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-03-09 23:49 Sending CQE to a different ring Artyom Pavlov
2022-03-10 1:36 ` Jens Axboe
2022-03-10 1:55 ` Jens Axboe
2022-03-10 2:33 ` Jens Axboe
2022-03-10 9:15 ` Chris Panayis
2022-03-10 13:53 ` Pavel Begunkov
2022-03-10 15:38 ` Jens Axboe
2022-03-10 2:11 ` Artyom Pavlov [this message]
2022-03-10 3:00 ` Jens Axboe
2022-03-10 3:48 ` Artyom Pavlov
2022-03-10 4:03 ` Jens Axboe
2022-03-10 4:14 ` Jens Axboe
2022-03-10 14:00 ` Artyom Pavlov
2022-03-10 15:36 ` Artyom Pavlov
2022-03-10 15:43 ` Jens Axboe
2022-03-10 15:46 ` Jens Axboe
2022-03-10 15:52 ` Artyom Pavlov
2022-03-10 15:57 ` Jens Axboe
2022-03-10 16:07 ` Artyom Pavlov
2022-03-10 16:12 ` Jens Axboe
2022-03-10 16:22 ` Artyom Pavlov
2022-03-10 16:25 ` Jens Axboe
2022-03-10 16:28 ` Artyom Pavlov
2022-03-10 16:30 ` Jens Axboe
2022-03-10 13:34 ` Pavel Begunkov
2022-03-10 13:43 ` Jens Axboe
2022-03-10 13:51 ` Pavel Begunkov
2022-03-10 3:06 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox