Re: [PATCH 2/2] io_uring: add support for passing fixed file descriptors

[Hao Xu <hao.xu@linux.dev>]

On 6/18/22 21:16, Jens Axboe wrote:
> On 6/18/22 7:09 AM, Hao Xu wrote:
>> On 6/18/22 20:50, Jens Axboe wrote:
>>> On 6/18/22 6:47 AM, Hao Xu wrote:
>>>> On 6/18/22 19:34, Jens Axboe wrote:
>>>>> On 6/18/22 5:02 AM, Hao Xu wrote:
>>>>>> On 6/17/22 21:45, Jens Axboe wrote:
>>>>>>> With IORING_OP_MSG_RING, one ring can send a message to another ring.
>>>>>>> Extend that support to also allow sending a fixed file descriptor to
>>>>>>> that ring, enabling one ring to pass a registered descriptor to another
>>>>>>> one.
>>>>>>>
>>>>>>> Arguments are extended to pass in:
>>>>>>>
>>>>>>> sqe->addr3    fixed file slot in source ring
>>>>>>> sqe->file_index    fixed file slot in destination ring
>>>>>>>
>>>>>>> IORING_OP_MSG_RING is extended to take a command argument in sqe->addr.
>>>>>>> If set to zero (or IORING_MSG_DATA), it sends just a message like before.
>>>>>>> If set to IORING_MSG_SEND_FD, a fixed file descriptor is sent according
>>>>>>> to the above arguments.
>>>>>>>
>>>>>>> Undecided:
>>>>>>>        - Should we post a cqe with the send, or require that the sender
>>>>>>>          just link a separate IORING_OP_MSG_RING? This makes error
>>>>>>>          handling easier, as we cannot easily retract the installed
>>>>>>>          file descriptor if the target CQ ring is full. Right now we do
>>>>>>>          fill a CQE. If the request completes with -EOVERFLOW, then the
>>>>>>>          sender must re-send a CQE if the target must get notified.
>>>>>>
>>>>>> Hi Jens,
>>>>>> Since we are have open/accept direct feature, this may be useful. But I
>>>>>> just can't think of a real case that people use two rings and need to do
>>>>>> operations to same fd.
>>>>>
>>>>> The two cases that people bring up as missing for direct descriptors
>>>>> that you can currently do with a real fd is:
>>>>>
>>>>> 1) Server needs to be shutdown or restarted, pass file descriptors to
>>>>>       another onei
>>>>>
>>>>> 2) Backend is split, and one accepts connections, while others then get
>>>>>       the fd passed and handle the actual connection.
>>>>>
>>>>> Both of those are classic SCM_RIGHTS use cases, and it's not possible to
>>>>> support them with direct descriptors today.
>>>>
>>>> I see, thanks for detail explanation.
>>>
>>> I should put that in the commit message in fact. Will do so.
>>>
>>>>>> Assume there are real cases, then filling a cqe is necessary since users
>>>>>> need to first make sure the desired fd is registered before doing
>>>>>> something to it.
>>>>>
>>>>> Right, my quesion here was really whether it should be bundled with the
>>>>> IORING_MSG_SEND_FD operation, or whether the issuer of that should also
>>>>> be responsible for then posting a "normal" IORING_OP_MSG_SEND to the
>>>>> target ring to notify it if the fact that an fd has been sent to it.
>>>>>
>>>>> If the operation is split like the latter, then it makes the error
>>>>> handling a bit easier as we eliminate one failing part of the existing
>>>>> MSG_SEND_FD.
>>>>>
>>>>> You could then also pass a number of descriptors and then post a single
>>>>> OP_MSG_SEND with some data that tells you which descriptors were passed.
>>
>> [1]
>>
>>>>>
>>>>> For the basic use case of just passing a single descriptor, what the
>>>>> code currently does is probably the sanest approach - send the fd, post
>>>>> a cqe.
>>
>> I think it's fine to keep it like this, since we can achieve [1] by a
>> GROUP_DELIVER flag and set cqe_skip flag for send msg request when it
>> turns out [1] is indeed necessary.
> 
> The expected use case is probably CQE_SKIP for using this, as the sender
> doesn't care about being notified about a successful send. But for the
> target CQE, we'd then need to either have CQE_SKIP implying that we
> should skip CQE delivery there too, or we'd need to add an

Yea, that's what I meant, CQE_SKIP for the target_ctx cqe.

> IORING_OP_MSG_RING flag for that. I think the latter is the cleaner
> approach, and it would indeed then allow both use cases. If you're
> sending a bunch of fds and would prefer to notify with a single
> OP_MSG_RING when they are done, then you'd set that OP_MSG_RING flag
> that says "don't post a CQE to the target".
> 
> Hence my proposal would be to keep the CQE delivery by default as it
> stands in the patch, and add a flag for controlling whether or not
> OP_MSG_RING with MSG_SEND posts a CQE to the target ring or not.
> 
> Agree?
> 

Sounds good.