Re: Sending CQE to a different ring

[Pavel Begunkov <asml.silence@gmail.com>]

On 3/10/22 13:43, Jens Axboe wrote:
> On 3/10/22 6:34 AM, Pavel Begunkov wrote:
>> On 3/10/22 03:00, Jens Axboe wrote:
>>> On 3/9/22 7:11 PM, Artyom Pavlov wrote:
>>>> 10.03.2022 04:36, Jens Axboe wrote:
>>>>> On 3/9/22 4:49 PM, Artyom Pavlov wrote:
[...]
>>> OK, so what you're asking is to be able to submit an sqe to ring1, but
>>> have the completion show up in ring2? With the idea being that the rings
>>> are setup so that you're basing this on which thread should ultimately
>>> process the request when it completes, which is why you want it to
>>> target another ring?
>>>
>>> It'd certainly be doable, but it's a bit of a strange beast. My main
>>> concern with that would be:
>>>
>>> 1) It's a fast path code addition to every request, we'd need to check
>>>      some new field (sqe->completion_ring_fd) and then also grab a
>>>      reference to that file for use at completion time.
>>>
>>> 2) Completions are protected by the completion lock, and it isn't
>>>      trivial to nest these. What happens if ring1 submits an sqe with
>>>      ring2 as the cqe target, and ring2 submits an sqe with ring1 as the
>>>      cqe target? We can't safely nest these, as we could easily introduce
>>>      deadlocks that way.
>>>
>>> My knee jerk reaction is that it'd be both simpler and cheaper to
>>> implement this in userspace... Unless there's an elegant solution to it,
>>> which I don't immediately see.
>>
>> Per request fd will be ugly and slow unfortunately. As people asked about
>> a similar thing before, the only thing I can suggest is to add a way
>> to pass another SQ. The execution will be slower, but at least can be
>> made zero overhead for the normal path.
> 
> The MSG_RING command seems like a good fit for me, and it'll both cater
> to the "I just need to wakeup this ring and I don't want to use signals"
> crowd, and passing actual (limited) information like what is needed in
> this case.

Agree if that's what is needed.

To clarify, another approach I suggested is to be able to submit from a
different userspace provided SQ, so there is no locking around the main
SQ. And this doesn't cross a ring boundary, one will still need to specify
only one target ring fd


-- 
Pavel Begunkov