* IORING_OP_READ and O_NONBLOCK behaviour @ 2020-09-02 10:09 Norman Maurer 2020-09-02 14:45 ` Jens Axboe 0 siblings, 1 reply; 7+ messages in thread From: Norman Maurer @ 2020-09-02 10:09 UTC (permalink / raw) To: io-uring; +Cc: Josef Hi there, We are currently working on integrating io_uring into netty and found some “suprising” behaviour which seems like a bug to me. When a socket is marked as non blocking (accepted with O_NONBLOCK flag set) and there is no data to be read IORING_OP_READ should complete directly with EAGAIN or EWOULDBLOCK. This is not the case and it basically blocks forever until there is some data to read. Is this expected ? This seems to be somehow related to a bug that was fixed for IO_URING_ACCEPT with non blocking sockets: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=e697deed834de15d2322d0619d51893022c90ea2 Thanks Norman ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: IORING_OP_READ and O_NONBLOCK behaviour 2020-09-02 10:09 IORING_OP_READ and O_NONBLOCK behaviour Norman Maurer @ 2020-09-02 14:45 ` Jens Axboe 2020-09-02 15:26 ` Pavel Begunkov 0 siblings, 1 reply; 7+ messages in thread From: Jens Axboe @ 2020-09-02 14:45 UTC (permalink / raw) To: Norman Maurer, io-uring; +Cc: Josef On 9/2/20 4:09 AM, Norman Maurer wrote: > Hi there, > > We are currently working on integrating io_uring into netty and found > some “suprising” behaviour which seems like a bug to me. > > When a socket is marked as non blocking (accepted with O_NONBLOCK flag > set) and there is no data to be read IORING_OP_READ should complete > directly with EAGAIN or EWOULDBLOCK. This is not the case and it > basically blocks forever until there is some data to read. Is this > expected ? > > This seems to be somehow related to a bug that was fixed for > IO_URING_ACCEPT with non blocking sockets: > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=e697deed834de15d2322d0619d51893022c90ea2 I agree with you that this is a bug, in general it's useful (and expected) that we'd return -EAGAIN for that case. I'll take a look. -- Jens Axboe ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: IORING_OP_READ and O_NONBLOCK behaviour 2020-09-02 14:45 ` Jens Axboe @ 2020-09-02 15:26 ` Pavel Begunkov 2020-09-02 15:35 ` Jens Axboe 0 siblings, 1 reply; 7+ messages in thread From: Pavel Begunkov @ 2020-09-02 15:26 UTC (permalink / raw) To: Jens Axboe, Norman Maurer, io-uring; +Cc: Josef On 02/09/2020 17:45, Jens Axboe wrote: > On 9/2/20 4:09 AM, Norman Maurer wrote: >> Hi there, >> >> We are currently working on integrating io_uring into netty and found >> some “suprising” behaviour which seems like a bug to me. >> >> When a socket is marked as non blocking (accepted with O_NONBLOCK flag >> set) and there is no data to be read IORING_OP_READ should complete >> directly with EAGAIN or EWOULDBLOCK. This is not the case and it >> basically blocks forever until there is some data to read. Is this >> expected ? >> >> This seems to be somehow related to a bug that was fixed for >> IO_URING_ACCEPT with non blocking sockets: >> >> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=e697deed834de15d2322d0619d51893022c90ea2 > > I agree with you that this is a bug, in general it's useful (and > expected) that we'd return -EAGAIN for that case. I'll take a look. > That's I mentioned that doing retries for nonblock requests in io_wq_submit_work() doesn't look consistent. I think killing it off may help. -- Pavel Begunkov ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: IORING_OP_READ and O_NONBLOCK behaviour 2020-09-02 15:26 ` Pavel Begunkov @ 2020-09-02 15:35 ` Jens Axboe 2020-09-02 16:00 ` Pavel Begunkov 0 siblings, 1 reply; 7+ messages in thread From: Jens Axboe @ 2020-09-02 15:35 UTC (permalink / raw) To: Pavel Begunkov, Norman Maurer, io-uring; +Cc: Josef On 9/2/20 9:26 AM, Pavel Begunkov wrote: > On 02/09/2020 17:45, Jens Axboe wrote: >> On 9/2/20 4:09 AM, Norman Maurer wrote: >>> Hi there, >>> >>> We are currently working on integrating io_uring into netty and found >>> some “suprising” behaviour which seems like a bug to me. >>> >>> When a socket is marked as non blocking (accepted with O_NONBLOCK flag >>> set) and there is no data to be read IORING_OP_READ should complete >>> directly with EAGAIN or EWOULDBLOCK. This is not the case and it >>> basically blocks forever until there is some data to read. Is this >>> expected ? >>> >>> This seems to be somehow related to a bug that was fixed for >>> IO_URING_ACCEPT with non blocking sockets: >>> >>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=e697deed834de15d2322d0619d51893022c90ea2 >> >> I agree with you that this is a bug, in general it's useful (and >> expected) that we'd return -EAGAIN for that case. I'll take a look. >> > > That's I mentioned that doing retries for nonblock requests in > io_wq_submit_work() doesn't look consistent. I think killing it > off may help. Right, we should not retry those _in general_, the exception is regular files or block devices to handle IOPOLL retry where we do need it. The below is what I came up with for this one. Might not hurt to make this more explicit for 5.10. commit c78e0f02c3861b5b176b2f79552677b3604deb76 Author: Jens Axboe <[email protected]> Date: Wed Sep 2 09:30:31 2020 -0600 io_uring: no read-retry on -EAGAIN error and O_NONBLOCK marked file Actually two things that need fixing up here: - The io_rw_reissue() -EAGAIN retry is explicit to block devices and regular files, so don't ever attempt to do that on other types of files. - If we hit -EAGAIN on a nonblock marked file, don't arm poll handler for it. It should just complete with -EAGAIN. Cc: [email protected] Reported-by: Norman Maurer <[email protected]> Signed-off-by: Jens Axboe <[email protected]> diff --git a/fs/io_uring.c b/fs/io_uring.c index b1ccd7072d93..dc27cd5b8ad6 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -2300,8 +2300,11 @@ static bool io_resubmit_prep(struct io_kiocb *req, int error) static bool io_rw_reissue(struct io_kiocb *req, long res) { #ifdef CONFIG_BLOCK + umode_t mode = file_inode(req->file)->i_mode; int ret; + if (!S_ISBLK(mode) && !S_ISREG(mode)) + return false; if ((res != -EAGAIN && res != -EOPNOTSUPP) || io_wq_current_is_worker()) return false; @@ -3146,6 +3149,9 @@ static int io_read(struct io_kiocb *req, bool force_nonblock, /* IOPOLL retry should happen for io-wq threads */ if (!force_nonblock && !(req->ctx->flags & IORING_SETUP_IOPOLL)) goto done; + /* no retry on NONBLOCK marked file */ + if (kiocb->ki_flags & IOCB_NOWAIT) + goto done; /* some cases will consume bytes even on error returns */ iov_iter_revert(iter, iov_count - iov_iter_count(iter)); ret = io_setup_async_rw(req, iovec, inline_vecs, iter, false); -- Jens Axboe ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: IORING_OP_READ and O_NONBLOCK behaviour 2020-09-02 15:35 ` Jens Axboe @ 2020-09-02 16:00 ` Pavel Begunkov 2020-09-02 16:02 ` Pavel Begunkov 2020-09-02 16:05 ` Jens Axboe 0 siblings, 2 replies; 7+ messages in thread From: Pavel Begunkov @ 2020-09-02 16:00 UTC (permalink / raw) To: Jens Axboe, Norman Maurer, io-uring; +Cc: Josef On 02/09/2020 18:35, Jens Axboe wrote: > On 9/2/20 9:26 AM, Pavel Begunkov wrote: >> On 02/09/2020 17:45, Jens Axboe wrote: >>> On 9/2/20 4:09 AM, Norman Maurer wrote: >>>> Hi there, >>>> >>>> We are currently working on integrating io_uring into netty and found >>>> some “suprising” behaviour which seems like a bug to me. >>>> >>>> When a socket is marked as non blocking (accepted with O_NONBLOCK flag >>>> set) and there is no data to be read IORING_OP_READ should complete >>>> directly with EAGAIN or EWOULDBLOCK. This is not the case and it >>>> basically blocks forever until there is some data to read. Is this >>>> expected ? >>>> >>>> This seems to be somehow related to a bug that was fixed for >>>> IO_URING_ACCEPT with non blocking sockets: >>>> >>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=e697deed834de15d2322d0619d51893022c90ea2 >>> >>> I agree with you that this is a bug, in general it's useful (and >>> expected) that we'd return -EAGAIN for that case. I'll take a look. >>> >> >> That's I mentioned that doing retries for nonblock requests in >> io_wq_submit_work() doesn't look consistent. I think killing it >> off may help. > > Right, we should not retry those _in general_, the exception is regular > files or block devices to handle IOPOLL retry where we do need it. The > below is what I came up with for this one. Might not hurt to make this > more explicit for 5.10. Hmm, I didn't checked it, but if we > > > commit c78e0f02c3861b5b176b2f79552677b3604deb76 > Author: Jens Axboe <[email protected]> > Date: Wed Sep 2 09:30:31 2020 -0600 > > io_uring: no read-retry on -EAGAIN error and O_NONBLOCK marked file > > Actually two things that need fixing up here: > > - The io_rw_reissue() -EAGAIN retry is explicit to block devices and > regular files, so don't ever attempt to do that on other types of > files. > > - If we hit -EAGAIN on a nonblock marked file, don't arm poll handler for > it. It should just complete with -EAGAIN. > > Cc: [email protected] > Reported-by: Norman Maurer <[email protected]> > Signed-off-by: Jens Axboe <[email protected]> > > diff --git a/fs/io_uring.c b/fs/io_uring.c > index b1ccd7072d93..dc27cd5b8ad6 100644 > --- a/fs/io_uring.c > +++ b/fs/io_uring.c > @@ -2300,8 +2300,11 @@ static bool io_resubmit_prep(struct io_kiocb *req, int error) > static bool io_rw_reissue(struct io_kiocb *req, long res) > { > #ifdef CONFIG_BLOCK > + umode_t mode = file_inode(req->file)->i_mode; > int ret; > > + if (!S_ISBLK(mode) && !S_ISREG(mode)) > + return false; > if ((res != -EAGAIN && res != -EOPNOTSUPP) || io_wq_current_is_worker()) > return false; > > @@ -3146,6 +3149,9 @@ static int io_read(struct io_kiocb *req, bool force_nonblock, > /* IOPOLL retry should happen for io-wq threads */ > if (!force_nonblock && !(req->ctx->flags & IORING_SETUP_IOPOLL)) > goto done; > + /* no retry on NONBLOCK marked file */ > + if (kiocb->ki_flags & IOCB_NOWAIT) > + goto done; We clearing and setting IOCB_NOWAIT depending on @force_nonblock, so it may not work. E.g. with IOSQE_IO_ASYNC io_read() will clear it at the beginning. Maybe REQ_F_NOWAIT? > /* some cases will consume bytes even on error returns */ > iov_iter_revert(iter, iov_count - iov_iter_count(iter)); > ret = io_setup_async_rw(req, iovec, inline_vecs, iter, false); > -- Pavel Begunkov ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: IORING_OP_READ and O_NONBLOCK behaviour 2020-09-02 16:00 ` Pavel Begunkov @ 2020-09-02 16:02 ` Pavel Begunkov 2020-09-02 16:05 ` Jens Axboe 1 sibling, 0 replies; 7+ messages in thread From: Pavel Begunkov @ 2020-09-02 16:02 UTC (permalink / raw) To: Jens Axboe, Norman Maurer, io-uring; +Cc: Josef On 02/09/2020 19:00, Pavel Begunkov wrote: > On 02/09/2020 18:35, Jens Axboe wrote: >> On 9/2/20 9:26 AM, Pavel Begunkov wrote: >>> On 02/09/2020 17:45, Jens Axboe wrote: >>>> On 9/2/20 4:09 AM, Norman Maurer wrote: >>>>> Hi there, >>>>> >>>>> We are currently working on integrating io_uring into netty and found >>>>> some “suprising” behaviour which seems like a bug to me. >>>>> >>>>> When a socket is marked as non blocking (accepted with O_NONBLOCK flag >>>>> set) and there is no data to be read IORING_OP_READ should complete >>>>> directly with EAGAIN or EWOULDBLOCK. This is not the case and it >>>>> basically blocks forever until there is some data to read. Is this >>>>> expected ? >>>>> >>>>> This seems to be somehow related to a bug that was fixed for >>>>> IO_URING_ACCEPT with non blocking sockets: >>>>> >>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=e697deed834de15d2322d0619d51893022c90ea2 >>>> >>>> I agree with you that this is a bug, in general it's useful (and >>>> expected) that we'd return -EAGAIN for that case. I'll take a look. >>>> >>> >>> That's I mentioned that doing retries for nonblock requests in >>> io_wq_submit_work() doesn't look consistent. I think killing it >>> off may help. >> >> Right, we should not retry those _in general_, the exception is regular >> files or block devices to handle IOPOLL retry where we do need it. The >> below is what I came up with for this one. Might not hurt to make this >> more explicit for 5.10. > > Hmm, I didn't checked it, but if we Oops, garbage text. > >> >> >> commit c78e0f02c3861b5b176b2f79552677b3604deb76 >> Author: Jens Axboe <[email protected]> >> Date: Wed Sep 2 09:30:31 2020 -0600 >> >> io_uring: no read-retry on -EAGAIN error and O_NONBLOCK marked file >> >> Actually two things that need fixing up here: >> >> - The io_rw_reissue() -EAGAIN retry is explicit to block devices and >> regular files, so don't ever attempt to do that on other types of >> files. >> >> - If we hit -EAGAIN on a nonblock marked file, don't arm poll handler for >> it. It should just complete with -EAGAIN. >> >> Cc: [email protected] >> Reported-by: Norman Maurer <[email protected]> >> Signed-off-by: Jens Axboe <[email protected]> >> >> diff --git a/fs/io_uring.c b/fs/io_uring.c >> index b1ccd7072d93..dc27cd5b8ad6 100644 >> --- a/fs/io_uring.c >> +++ b/fs/io_uring.c >> @@ -2300,8 +2300,11 @@ static bool io_resubmit_prep(struct io_kiocb *req, int error) >> static bool io_rw_reissue(struct io_kiocb *req, long res) >> { >> #ifdef CONFIG_BLOCK >> + umode_t mode = file_inode(req->file)->i_mode; >> int ret; >> >> + if (!S_ISBLK(mode) && !S_ISREG(mode)) >> + return false; >> if ((res != -EAGAIN && res != -EOPNOTSUPP) || io_wq_current_is_worker()) >> return false; >> >> @@ -3146,6 +3149,9 @@ static int io_read(struct io_kiocb *req, bool force_nonblock, >> /* IOPOLL retry should happen for io-wq threads */ >> if (!force_nonblock && !(req->ctx->flags & IORING_SETUP_IOPOLL)) >> goto done; >> + /* no retry on NONBLOCK marked file */ >> + if (kiocb->ki_flags & IOCB_NOWAIT) >> + goto done; > > We clearing and setting IOCB_NOWAIT depending on @force_nonblock, so it may not > work. E.g. with IOSQE_IO_ASYNC io_read() will clear it at the beginning. > Maybe REQ_F_NOWAIT? > >> /* some cases will consume bytes even on error returns */ >> iov_iter_revert(iter, iov_count - iov_iter_count(iter)); >> ret = io_setup_async_rw(req, iovec, inline_vecs, iter, false); >> > -- Pavel Begunkov ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: IORING_OP_READ and O_NONBLOCK behaviour 2020-09-02 16:00 ` Pavel Begunkov 2020-09-02 16:02 ` Pavel Begunkov @ 2020-09-02 16:05 ` Jens Axboe 1 sibling, 0 replies; 7+ messages in thread From: Jens Axboe @ 2020-09-02 16:05 UTC (permalink / raw) To: Pavel Begunkov, Norman Maurer, io-uring; +Cc: Josef On 9/2/20 10:00 AM, Pavel Begunkov wrote: > On 02/09/2020 18:35, Jens Axboe wrote: >> On 9/2/20 9:26 AM, Pavel Begunkov wrote: >>> On 02/09/2020 17:45, Jens Axboe wrote: >>>> On 9/2/20 4:09 AM, Norman Maurer wrote: >>>>> Hi there, >>>>> >>>>> We are currently working on integrating io_uring into netty and found >>>>> some “suprising” behaviour which seems like a bug to me. >>>>> >>>>> When a socket is marked as non blocking (accepted with O_NONBLOCK flag >>>>> set) and there is no data to be read IORING_OP_READ should complete >>>>> directly with EAGAIN or EWOULDBLOCK. This is not the case and it >>>>> basically blocks forever until there is some data to read. Is this >>>>> expected ? >>>>> >>>>> This seems to be somehow related to a bug that was fixed for >>>>> IO_URING_ACCEPT with non blocking sockets: >>>>> >>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=e697deed834de15d2322d0619d51893022c90ea2 >>>> >>>> I agree with you that this is a bug, in general it's useful (and >>>> expected) that we'd return -EAGAIN for that case. I'll take a look. >>>> >>> >>> That's I mentioned that doing retries for nonblock requests in >>> io_wq_submit_work() doesn't look consistent. I think killing it >>> off may help. >> >> Right, we should not retry those _in general_, the exception is regular >> files or block devices to handle IOPOLL retry where we do need it. The >> below is what I came up with for this one. Might not hurt to make this >> more explicit for 5.10. > > Hmm, I didn't checked it, but if we > >> >> >> commit c78e0f02c3861b5b176b2f79552677b3604deb76 >> Author: Jens Axboe <[email protected]> >> Date: Wed Sep 2 09:30:31 2020 -0600 >> >> io_uring: no read-retry on -EAGAIN error and O_NONBLOCK marked file >> >> Actually two things that need fixing up here: >> >> - The io_rw_reissue() -EAGAIN retry is explicit to block devices and >> regular files, so don't ever attempt to do that on other types of >> files. >> >> - If we hit -EAGAIN on a nonblock marked file, don't arm poll handler for >> it. It should just complete with -EAGAIN. >> >> Cc: [email protected] >> Reported-by: Norman Maurer <[email protected]> >> Signed-off-by: Jens Axboe <[email protected]> >> >> diff --git a/fs/io_uring.c b/fs/io_uring.c >> index b1ccd7072d93..dc27cd5b8ad6 100644 >> --- a/fs/io_uring.c >> +++ b/fs/io_uring.c >> @@ -2300,8 +2300,11 @@ static bool io_resubmit_prep(struct io_kiocb *req, int error) >> static bool io_rw_reissue(struct io_kiocb *req, long res) >> { >> #ifdef CONFIG_BLOCK >> + umode_t mode = file_inode(req->file)->i_mode; >> int ret; >> >> + if (!S_ISBLK(mode) && !S_ISREG(mode)) >> + return false; >> if ((res != -EAGAIN && res != -EOPNOTSUPP) || io_wq_current_is_worker()) >> return false; >> >> @@ -3146,6 +3149,9 @@ static int io_read(struct io_kiocb *req, bool force_nonblock, >> /* IOPOLL retry should happen for io-wq threads */ >> if (!force_nonblock && !(req->ctx->flags & IORING_SETUP_IOPOLL)) >> goto done; >> + /* no retry on NONBLOCK marked file */ >> + if (kiocb->ki_flags & IOCB_NOWAIT) >> + goto done; > > We clearing and setting IOCB_NOWAIT depending on @force_nonblock, so it may not > work. E.g. with IOSQE_IO_ASYNC io_read() will clear it at the beginning. > Maybe REQ_F_NOWAIT? Yeah, the posted version did get it right: https://lore.kernel.org/io-uring/[email protected]/T/#u -- Jens Axboe ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2020-09-02 16:05 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2020-09-02 10:09 IORING_OP_READ and O_NONBLOCK behaviour Norman Maurer 2020-09-02 14:45 ` Jens Axboe 2020-09-02 15:26 ` Pavel Begunkov 2020-09-02 15:35 ` Jens Axboe 2020-09-02 16:00 ` Pavel Begunkov 2020-09-02 16:02 ` Pavel Begunkov 2020-09-02 16:05 ` Jens Axboe
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox