Re: [RFC 1/1] io_uring: improve register file feature's usability

[Pavel Begunkov <asml.silence@gmail.com>]

On 10/13/21 04:32, Xiaoguang Wang wrote:
> hi,
>> On 10/12/21 14:11, Xiaoguang Wang wrote:
>>>> On 10/12/21 09:48, Xiaoguang Wang wrote:
>>>>> The idea behind register file feature is good and straightforward, but
>>>>> there is a very big issue that it's hard to use for user apps. User apps
>>>>> need to bind slot info to file descriptor. For example, user app wants
>>>>> to register a file, then it first needs to find a free slot in register
>>>>> file infrastructure, that means user app needs to maintain slot info in
>>>>> userspace, which is a obvious burden for userspace developers.
>>>>
>>>> Slot allocation is specifically entirely given away to the userspace,
>>>> the userspace has more info and can use it more efficiently, e.g.
>>>> if there is only a small managed set of registered files they can
>>>> always have O(1) slot "lookup", and a couple of more use cases.
>>>
>>> Can you explain more what is slot "lookup", thanks. For me, it seems that
>>
>> I referred to nothing particular, just a way userspace finds a new index,
>> can be round robin or "index==fd".
>>
>>> use fd as slot is the simplest and most efficient way, user does not need to> mange slot info at all in userspace.
>>
>> As mentioned, it should be slightly more efficient to have a small table,
>> cache misses. Also, it's allocated with kvcalloc() so if it can't be
>> allocate physically contig memory it will set up virtual memory.
>>
>> So, if the userspace has some other way of indexing files, small tables
>> are preferred. For instance if it operates with 1-2 files, or stores files
>> in an array and the index in the array may serve the purpose, or any other
>> way. Also, additional memory for those who care.
> 
> Yeah, I agree with you that for small tables, current implementation seems good,
> 
> If user app just registers a small number of files, it may handle it well, but imagine
> 
> how netty, nginx or other network apps which will open thousands of socket files,
> 
> manage these socket files' slot info will be a obvious burden to developer, these
> 
> apps may need to develop a private component to record used or free slot. Especially
> 
> in a high concurrency scenario, frequent sockes opened or closed, this private component
> 
> may need locks to protect, that means this private component will introduce overhead too.
> 
> For a fd, vfs layer has already ensure its unique.
> 
>>
>>>> If userspace wants to mimic a fdtable into io_uring's registered table,
>>>> it's possible to do as is and without extra fdtable tracking
>>>>
>>>> fd = open();
>>>> io_uring_update_slot(off=fd, fd=fd);
>>>
>>> No, currently it's hard to do above work, unless we register a big number of files initially.
>>
>> If they intend to use a big number of files that's the way to go. They
>> can unregister/register if needed, usual grow factor=2  should make
>> it workable.
> 
> I'm not sure un-register/register are appropriate，say a app registers 1000 files, then
> 
> it needs to un-register 1000 files firstly, there are doubts whether can do this un-registration
> 
> work, if some of these files are used by other threads, which submit sqes with FIXED_FILE
> 
> flags continually, so the first un-registration work needs to synchronize with threads which
> 
> are submitting requests. And later app needs to prepare a new files array, saving current 1000
> 
> files and new files info to this new array, for me, it can works, but not efficient and somewhat
> 
> hard to use :)

Sounds reasonable. What I oppose is wiring it solely based on fd. On the
other hand, it sounds what you need is a "grow table" feature.

We can also think about adding new format, instead of array of fds, add
passing an array of pairs {offset, fd}.

-- 
Pavel Begunkov