Re: [PATCH 2/2] io_uring: add support for passing fixed file descriptors

[Jens Axboe <axboe@kernel.dk>]

On 6/18/22 7:09 AM, Hao Xu wrote:
> On 6/18/22 20:50, Jens Axboe wrote:
>> On 6/18/22 6:47 AM, Hao Xu wrote:
>>> On 6/18/22 19:34, Jens Axboe wrote:
>>>> On 6/18/22 5:02 AM, Hao Xu wrote:
>>>>> On 6/17/22 21:45, Jens Axboe wrote:
>>>>>> With IORING_OP_MSG_RING, one ring can send a message to another ring.
>>>>>> Extend that support to also allow sending a fixed file descriptor to
>>>>>> that ring, enabling one ring to pass a registered descriptor to another
>>>>>> one.
>>>>>>
>>>>>> Arguments are extended to pass in:
>>>>>>
>>>>>> sqe->addr3    fixed file slot in source ring
>>>>>> sqe->file_index    fixed file slot in destination ring
>>>>>>
>>>>>> IORING_OP_MSG_RING is extended to take a command argument in sqe->addr.
>>>>>> If set to zero (or IORING_MSG_DATA), it sends just a message like before.
>>>>>> If set to IORING_MSG_SEND_FD, a fixed file descriptor is sent according
>>>>>> to the above arguments.
>>>>>>
>>>>>> Undecided:
>>>>>>       - Should we post a cqe with the send, or require that the sender
>>>>>>         just link a separate IORING_OP_MSG_RING? This makes error
>>>>>>         handling easier, as we cannot easily retract the installed
>>>>>>         file descriptor if the target CQ ring is full. Right now we do
>>>>>>         fill a CQE. If the request completes with -EOVERFLOW, then the
>>>>>>         sender must re-send a CQE if the target must get notified.
>>>>>
>>>>> Hi Jens,
>>>>> Since we are have open/accept direct feature, this may be useful. But I
>>>>> just can't think of a real case that people use two rings and need to do
>>>>> operations to same fd.
>>>>
>>>> The two cases that people bring up as missing for direct descriptors
>>>> that you can currently do with a real fd is:
>>>>
>>>> 1) Server needs to be shutdown or restarted, pass file descriptors to
>>>>      another onei
>>>>
>>>> 2) Backend is split, and one accepts connections, while others then get
>>>>      the fd passed and handle the actual connection.
>>>>
>>>> Both of those are classic SCM_RIGHTS use cases, and it's not possible to
>>>> support them with direct descriptors today.
>>>
>>> I see, thanks for detail explanation.
>>
>> I should put that in the commit message in fact. Will do so.
>>
>>>>> Assume there are real cases, then filling a cqe is necessary since users
>>>>> need to first make sure the desired fd is registered before doing
>>>>> something to it.
>>>>
>>>> Right, my quesion here was really whether it should be bundled with the
>>>> IORING_MSG_SEND_FD operation, or whether the issuer of that should also
>>>> be responsible for then posting a "normal" IORING_OP_MSG_SEND to the
>>>> target ring to notify it if the fact that an fd has been sent to it.
>>>>
>>>> If the operation is split like the latter, then it makes the error
>>>> handling a bit easier as we eliminate one failing part of the existing
>>>> MSG_SEND_FD.
>>>>
>>>> You could then also pass a number of descriptors and then post a single
>>>> OP_MSG_SEND with some data that tells you which descriptors were passed.
> 
> [1]
> 
>>>>
>>>> For the basic use case of just passing a single descriptor, what the
>>>> code currently does is probably the sanest approach - send the fd, post
>>>> a cqe.
> 
> I think it's fine to keep it like this, since we can achieve [1] by a
> GROUP_DELIVER flag and set cqe_skip flag for send msg request when it
> turns out [1] is indeed necessary.

The expected use case is probably CQE_SKIP for using this, as the sender
doesn't care about being notified about a successful send. But for the
target CQE, we'd then need to either have CQE_SKIP implying that we
should skip CQE delivery there too, or we'd need to add an
IORING_OP_MSG_RING flag for that. I think the latter is the cleaner
approach, and it would indeed then allow both use cases. If you're
sending a bunch of fds and would prefer to notify with a single
OP_MSG_RING when they are done, then you'd set that OP_MSG_RING flag
that says "don't post a CQE to the target".

Hence my proposal would be to keep the CQE delivery by default as it
stands in the patch, and add a flag for controlling whether or not
OP_MSG_RING with MSG_SEND posts a CQE to the target ring or not.

Agree?

-- 
Jens Axboe