Re: [PATCH] io_uring: add io_uring_enter(2) fixed file support

[Jens Axboe <axboe@kernel.dk>]

On 3/8/22 1:38 AM, Xiaoguang Wang wrote:
> hi,
> 
>>>> I'll take a look at liburing and see what we need to do there. I think
>>>> the sanest thing to do here is say that using a registered ring fd means
>>>> you cannot share the ring, ever. And then just have a
>>>> ring->enter_ring_fd which is normally just set to ring_fd when the ring
>>>> is setup, and if you register the ring fd, then we set it to whatever
>>>> the registered value is. Everything calling io_uring_enter() then just
>>>> needs to be modified to use ->enter_ring_fd instead of ->ring_fd.
>>> ok, look forward to use this api.
>> Can you take a look at the registered-ring branch for liburing:
>>
>> https://git.kernel.dk/cgit/liburing/log/?h=registered-ring
>>
>> which has the basic plumbing for it. Comments (or patches) welcome!
> Sorry for late reply, spend time to read your patch today. Basically it looks ok,
> there is one minor issue in "Add preliminary support for using a registered ring fd":
> @@ -417,6 +425,10 @@ struct io_uring_sqe *io_uring_get_sqe(struct io_uring *ring)
> 
>  int __io_uring_sqring_wait(struct io_uring *ring)
>  {
> -    return  ____sys_io_uring_enter(ring->ring_fd, 0, 0,
> -                       IORING_ENTER_SQ_WAIT, NULL);
> +    int flags = IORING_ENTER_SQ_WAIT;
> +
> +    if (ring->int_flags & INT_FLAG_REG_RING)
> +        flags |= IORING_ENTER_REGISTERED_RING;
> +
> +    return  ____sys_io_uring_enter(ring->ring_fd, 0, 0, flags, NULL);
>  }
> 
> Here it should be enter_ring_fd.

Ah good catch, I've fixed that up.

>> Few things I don't really love:
>>
>> 1) You need to call io_uring_register_ring_fd() after setting up the
>>     ring. We could provide init helpers for that, which just do queue
>>     init and then register ring. Maybe that'd make it more likely to get
>>     picked up by applications.
> Agree, that'd be better in some cases, but consider that currently the
> capacity of ring fd cache is just 16, I'd suggest to let users make
> their own decisions, in case some ring fds could not allocate one
> empty slot, but some ring fds don't need them at all, for example,
> ring fd which enable sqpoll may no need this feature.

Agree, that's the route I ended up taking too.

>> 2) For the setup where you do share the ring between a submitter and
>>     reaper, we need to ensure that the registered ring fd is the same
>>     between both of them. We need a helper for that. It's basically the
>>     same as io_uring_register_ring_fd(), but we need the specific offset.
>>     And if that fails with -EBUSY, we should just turn off
>>     INT_FLAG_RING_REG for the ring and you don't get the registered fd
>>     for either of them. At least it can be handled transparantly.
> Storing enter_ring_fd in struct io_uring seems not good, struct
> io_uring is a shared struct, as what you say, different threads that
> share one ring fd may have differed offset in ring fd cache. I have
> two suggestions:
> 1) Threads keep their offset in ring fd cache alone, and pass it to
> io_uring_submit, which may look ugly :)
> 2) define enter_ring_fd in struct io_ring to be a thread_local type,
> then your patches don't need to do any modifications.

Maybe I'm missing something here, but I don't see how your number 2 is
possible?

I think for now I'll just keep it as-is. Yes, then you can't use a
registered ring fd if you share the ring betwen threads, but so be it.

-- 
Jens Axboe