Re: [PATCH 4/4] io_uring: mark opcodes that always need io-wq punt

[Pavel Begunkov <asml.silence@gmail.com>]

On 4/25/23 15:42, Ming Lei wrote:
> On Tue, Apr 25, 2023 at 07:31:10AM -0600, Jens Axboe wrote:
>> On 4/24/23 8:50?PM, Ming Lei wrote:
>>> On Mon, Apr 24, 2023 at 08:18:02PM -0600, Jens Axboe wrote:
>>>> On 4/24/23 8:13?PM, Ming Lei wrote:
>>>>> On Mon, Apr 24, 2023 at 08:08:09PM -0600, Jens Axboe wrote:
>>>>>> On 4/24/23 6:57?PM, Ming Lei wrote:
>>>>>>> On Mon, Apr 24, 2023 at 09:24:33AM -0600, Jens Axboe wrote:
>>>>>>>> On 4/24/23 1:30?AM, Ming Lei wrote:
>>>>>>>>> On Thu, Apr 20, 2023 at 12:31:35PM -0600, Jens Axboe wrote:
>>>>>>>>>> Add an opdef bit for them, and set it for the opcodes where we always
>>>>>>>>>> need io-wq punt. With that done, exclude them from the file_can_poll()
>>>>>>>>>> check in terms of whether or not we need to punt them if any of the
>>>>>>>>>> NO_OFFLOAD flags are set.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Jens Axboe <axboe@kernel.dk>
>>>>>>>>>> ---
>>>>>>>>>>   io_uring/io_uring.c |  2 +-
>>>>>>>>>>   io_uring/opdef.c    | 22 ++++++++++++++++++++--
>>>>>>>>>>   io_uring/opdef.h    |  2 ++
>>>>>>>>>>   3 files changed, 23 insertions(+), 3 deletions(-)
>>>>>>>>>>
>>>>>>>>>> diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
>>>>>>>>>> index fee3e461e149..420cfd35ebc6 100644
>>>>>>>>>> --- a/io_uring/io_uring.c
>>>>>>>>>> +++ b/io_uring/io_uring.c
>>>>>>>>>> @@ -1948,7 +1948,7 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)
>>>>>>>>>>   		return -EBADF;
>>>>>>>>>>   
>>>>>>>>>>   	if (issue_flags & IO_URING_F_NO_OFFLOAD &&
>>>>>>>>>> -	    (!req->file || !file_can_poll(req->file)))
>>>>>>>>>> +	    (!req->file || !file_can_poll(req->file) || def->always_iowq))
>>>>>>>>>>   		issue_flags &= ~IO_URING_F_NONBLOCK;
>>>>>>>>>
>>>>>>>>> I guess the check should be !def->always_iowq?
>>>>>>>>
>>>>>>>> How so? Nobody that takes pollable files should/is setting
>>>>>>>> ->always_iowq. If we can poll the file, we should not force inline
>>>>>>>> submission. Basically the ones setting ->always_iowq always do -EAGAIN
>>>>>>>> returns if nonblock == true.
>>>>>>>
>>>>>>> I meant IO_URING_F_NONBLOCK is cleared here for  ->always_iowq, and
>>>>>>> these OPs won't return -EAGAIN, then run in the current task context
>>>>>>> directly.
>>>>>>
>>>>>> Right, of IO_URING_F_NO_OFFLOAD is set, which is entirely the point of
>>>>>> it :-)
>>>>>
>>>>> But ->always_iowq isn't actually _always_ since fallocate/fsync/... are
>>>>> not punted to iowq in case of IO_URING_F_NO_OFFLOAD, looks the naming of
>>>>> ->always_iowq is a bit confusing?
>>>>
>>>> Yeah naming isn't that great, I can see how that's bit confusing. I'll
>>>> be happy to take suggestions on what would make it clearer.
>>>
>>> Except for the naming, I am also wondering why these ->always_iowq OPs
>>> aren't punted to iowq in case of IO_URING_F_NO_OFFLOAD, given it
>>> shouldn't improve performance by doing so because these OPs are supposed
>>> to be slow and always slept, not like others(buffered writes, ...),
>>> can you provide one hint about not offloading these OPs? Or is it just that
>>> NO_OFFLOAD needs to not offload every OPs?
>>
>> The whole point of NO_OFFLOAD is that items that would normally be
>> passed to io-wq are just run inline. This provides a way to reap the
>> benefits of batched submissions and syscall reductions. Some opcodes
>> will just never be async, and io-wq offloads are not very fast. Some of
> 
> Yeah, seems io-wq is much slower than inline issue, maybe it needs
> to be looked into, and it is easy to run into io-wq for IOSQE_IO_LINK.

There were attempts like this one from Hao (CC'ed)

https://lore.kernel.org/io-uring/20220627133541.15223-5-hao.xu@linux.dev/t/

Not sure why it got stalled, but maybe Hao would be willing
to pick it up again.

-- 
Pavel Begunkov