Re: [PATCH 4/4] io_uring: mark opcodes that always need io-wq punt

[Ming Lei <ming.lei@redhat.com>]

On Tue, Apr 25, 2023 at 07:31:10AM -0600, Jens Axboe wrote:
> On 4/24/23 8:50?PM, Ming Lei wrote:
> > On Mon, Apr 24, 2023 at 08:18:02PM -0600, Jens Axboe wrote:
> >> On 4/24/23 8:13?PM, Ming Lei wrote:
> >>> On Mon, Apr 24, 2023 at 08:08:09PM -0600, Jens Axboe wrote:
> >>>> On 4/24/23 6:57?PM, Ming Lei wrote:
> >>>>> On Mon, Apr 24, 2023 at 09:24:33AM -0600, Jens Axboe wrote:
> >>>>>> On 4/24/23 1:30?AM, Ming Lei wrote:
> >>>>>>> On Thu, Apr 20, 2023 at 12:31:35PM -0600, Jens Axboe wrote:
> >>>>>>>> Add an opdef bit for them, and set it for the opcodes where we always
> >>>>>>>> need io-wq punt. With that done, exclude them from the file_can_poll()
> >>>>>>>> check in terms of whether or not we need to punt them if any of the
> >>>>>>>> NO_OFFLOAD flags are set.
> >>>>>>>>
> >>>>>>>> Signed-off-by: Jens Axboe <axboe@kernel.dk>
> >>>>>>>> ---
> >>>>>>>>  io_uring/io_uring.c |  2 +-
> >>>>>>>>  io_uring/opdef.c    | 22 ++++++++++++++++++++--
> >>>>>>>>  io_uring/opdef.h    |  2 ++
> >>>>>>>>  3 files changed, 23 insertions(+), 3 deletions(-)
> >>>>>>>>
> >>>>>>>> diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
> >>>>>>>> index fee3e461e149..420cfd35ebc6 100644
> >>>>>>>> --- a/io_uring/io_uring.c
> >>>>>>>> +++ b/io_uring/io_uring.c
> >>>>>>>> @@ -1948,7 +1948,7 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)
> >>>>>>>>  		return -EBADF;
> >>>>>>>>  
> >>>>>>>>  	if (issue_flags & IO_URING_F_NO_OFFLOAD &&
> >>>>>>>> -	    (!req->file || !file_can_poll(req->file)))
> >>>>>>>> +	    (!req->file || !file_can_poll(req->file) || def->always_iowq))
> >>>>>>>>  		issue_flags &= ~IO_URING_F_NONBLOCK;
> >>>>>>>
> >>>>>>> I guess the check should be !def->always_iowq?
> >>>>>>
> >>>>>> How so? Nobody that takes pollable files should/is setting
> >>>>>> ->always_iowq. If we can poll the file, we should not force inline
> >>>>>> submission. Basically the ones setting ->always_iowq always do -EAGAIN
> >>>>>> returns if nonblock == true.
> >>>>>
> >>>>> I meant IO_URING_F_NONBLOCK is cleared here for  ->always_iowq, and
> >>>>> these OPs won't return -EAGAIN, then run in the current task context
> >>>>> directly.
> >>>>
> >>>> Right, of IO_URING_F_NO_OFFLOAD is set, which is entirely the point of
> >>>> it :-)
> >>>
> >>> But ->always_iowq isn't actually _always_ since fallocate/fsync/... are
> >>> not punted to iowq in case of IO_URING_F_NO_OFFLOAD, looks the naming of
> >>> ->always_iowq is a bit confusing?
> >>
> >> Yeah naming isn't that great, I can see how that's bit confusing. I'll
> >> be happy to take suggestions on what would make it clearer.
> > 
> > Except for the naming, I am also wondering why these ->always_iowq OPs
> > aren't punted to iowq in case of IO_URING_F_NO_OFFLOAD, given it
> > shouldn't improve performance by doing so because these OPs are supposed
> > to be slow and always slept, not like others(buffered writes, ...),
> > can you provide one hint about not offloading these OPs? Or is it just that
> > NO_OFFLOAD needs to not offload every OPs?
> 
> The whole point of NO_OFFLOAD is that items that would normally be
> passed to io-wq are just run inline. This provides a way to reap the
> benefits of batched submissions and syscall reductions. Some opcodes
> will just never be async, and io-wq offloads are not very fast. Some of

Yeah, seems io-wq is much slower than inline issue, maybe it needs
to be looked into, and it is easy to run into io-wq for IOSQE_IO_LINK.

> them can eventually be migrated to async support, if the underlying
> mechanics support it.
> 
> You'll note that none of the ->always_iowq opcodes are pollable. If

True, then looks the ->always_iowq flag doesn't make a difference here
because your patch clears IO_URING_F_NONBLOCK for !file_can_poll(req->file).

Also almost all these ->always_iowq OPs are slow and blocked, if they are
issued inline, the submission pipeline will be blocked.

> NO_OFFLOAD is setup, it's pointless NOT to issue them with NONBLOCK
> cleared, as you'd just get -EAGAIN and then need to call them again with
> NONBLOCK cleared from the same context.

My point is that these OPs are slow and slept, so inline issue won't
improve performance actually for them, and punting to io-wq couldn't
be worse too. On the other side, inline issue may hurt perf because
submission pipeline is blocked when issuing these OPs.


Thanks, 
Ming