* IORING_OP_READ and O_NONBLOCK behaviour
@ 2020-09-02 10:09 Norman Maurer
2020-09-02 14:45 ` Jens Axboe
0 siblings, 1 reply; 7+ messages in thread
From: Norman Maurer @ 2020-09-02 10:09 UTC (permalink / raw)
To: io-uring; +Cc: Josef
Hi there,
We are currently working on integrating io_uring into netty and found some “suprising” behaviour which seems like a bug to me.
When a socket is marked as non blocking (accepted with O_NONBLOCK flag set) and there is no data to be read IORING_OP_READ should complete directly with EAGAIN or EWOULDBLOCK. This is not the case and it basically blocks forever until there is some data to read. Is this expected ?
This seems to be somehow related to a bug that was fixed for IO_URING_ACCEPT with non blocking sockets:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=e697deed834de15d2322d0619d51893022c90ea2
Thanks
Norman
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: IORING_OP_READ and O_NONBLOCK behaviour
2020-09-02 10:09 IORING_OP_READ and O_NONBLOCK behaviour Norman Maurer
@ 2020-09-02 14:45 ` Jens Axboe
2020-09-02 15:26 ` Pavel Begunkov
0 siblings, 1 reply; 7+ messages in thread
From: Jens Axboe @ 2020-09-02 14:45 UTC (permalink / raw)
To: Norman Maurer, io-uring; +Cc: Josef
On 9/2/20 4:09 AM, Norman Maurer wrote:
> Hi there,
>
> We are currently working on integrating io_uring into netty and found
> some “suprising” behaviour which seems like a bug to me.
>
> When a socket is marked as non blocking (accepted with O_NONBLOCK flag
> set) and there is no data to be read IORING_OP_READ should complete
> directly with EAGAIN or EWOULDBLOCK. This is not the case and it
> basically blocks forever until there is some data to read. Is this
> expected ?
>
> This seems to be somehow related to a bug that was fixed for
> IO_URING_ACCEPT with non blocking sockets:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=e697deed834de15d2322d0619d51893022c90ea2
I agree with you that this is a bug, in general it's useful (and
expected) that we'd return -EAGAIN for that case. I'll take a look.
--
Jens Axboe
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: IORING_OP_READ and O_NONBLOCK behaviour
2020-09-02 14:45 ` Jens Axboe
@ 2020-09-02 15:26 ` Pavel Begunkov
2020-09-02 15:35 ` Jens Axboe
0 siblings, 1 reply; 7+ messages in thread
From: Pavel Begunkov @ 2020-09-02 15:26 UTC (permalink / raw)
To: Jens Axboe, Norman Maurer, io-uring; +Cc: Josef
On 02/09/2020 17:45, Jens Axboe wrote:
> On 9/2/20 4:09 AM, Norman Maurer wrote:
>> Hi there,
>>
>> We are currently working on integrating io_uring into netty and found
>> some “suprising” behaviour which seems like a bug to me.
>>
>> When a socket is marked as non blocking (accepted with O_NONBLOCK flag
>> set) and there is no data to be read IORING_OP_READ should complete
>> directly with EAGAIN or EWOULDBLOCK. This is not the case and it
>> basically blocks forever until there is some data to read. Is this
>> expected ?
>>
>> This seems to be somehow related to a bug that was fixed for
>> IO_URING_ACCEPT with non blocking sockets:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=e697deed834de15d2322d0619d51893022c90ea2
>
> I agree with you that this is a bug, in general it's useful (and
> expected) that we'd return -EAGAIN for that case. I'll take a look.
>
That's I mentioned that doing retries for nonblock requests in
io_wq_submit_work() doesn't look consistent. I think killing it
off may help.
--
Pavel Begunkov
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: IORING_OP_READ and O_NONBLOCK behaviour
2020-09-02 15:26 ` Pavel Begunkov
@ 2020-09-02 15:35 ` Jens Axboe
2020-09-02 16:00 ` Pavel Begunkov
0 siblings, 1 reply; 7+ messages in thread
From: Jens Axboe @ 2020-09-02 15:35 UTC (permalink / raw)
To: Pavel Begunkov, Norman Maurer, io-uring; +Cc: Josef
On 9/2/20 9:26 AM, Pavel Begunkov wrote:
> On 02/09/2020 17:45, Jens Axboe wrote:
>> On 9/2/20 4:09 AM, Norman Maurer wrote:
>>> Hi there,
>>>
>>> We are currently working on integrating io_uring into netty and found
>>> some “suprising” behaviour which seems like a bug to me.
>>>
>>> When a socket is marked as non blocking (accepted with O_NONBLOCK flag
>>> set) and there is no data to be read IORING_OP_READ should complete
>>> directly with EAGAIN or EWOULDBLOCK. This is not the case and it
>>> basically blocks forever until there is some data to read. Is this
>>> expected ?
>>>
>>> This seems to be somehow related to a bug that was fixed for
>>> IO_URING_ACCEPT with non blocking sockets:
>>>
>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=e697deed834de15d2322d0619d51893022c90ea2
>>
>> I agree with you that this is a bug, in general it's useful (and
>> expected) that we'd return -EAGAIN for that case. I'll take a look.
>>
>
> That's I mentioned that doing retries for nonblock requests in
> io_wq_submit_work() doesn't look consistent. I think killing it
> off may help.
Right, we should not retry those _in general_, the exception is regular
files or block devices to handle IOPOLL retry where we do need it. The
below is what I came up with for this one. Might not hurt to make this
more explicit for 5.10.
commit c78e0f02c3861b5b176b2f79552677b3604deb76
Author: Jens Axboe <[email protected]>
Date: Wed Sep 2 09:30:31 2020 -0600
io_uring: no read-retry on -EAGAIN error and O_NONBLOCK marked file
Actually two things that need fixing up here:
- The io_rw_reissue() -EAGAIN retry is explicit to block devices and
regular files, so don't ever attempt to do that on other types of
files.
- If we hit -EAGAIN on a nonblock marked file, don't arm poll handler for
it. It should just complete with -EAGAIN.
Cc: [email protected]
Reported-by: Norman Maurer <[email protected]>
Signed-off-by: Jens Axboe <[email protected]>
diff --git a/fs/io_uring.c b/fs/io_uring.c
index b1ccd7072d93..dc27cd5b8ad6 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -2300,8 +2300,11 @@ static bool io_resubmit_prep(struct io_kiocb *req, int error)
static bool io_rw_reissue(struct io_kiocb *req, long res)
{
#ifdef CONFIG_BLOCK
+ umode_t mode = file_inode(req->file)->i_mode;
int ret;
+ if (!S_ISBLK(mode) && !S_ISREG(mode))
+ return false;
if ((res != -EAGAIN && res != -EOPNOTSUPP) || io_wq_current_is_worker())
return false;
@@ -3146,6 +3149,9 @@ static int io_read(struct io_kiocb *req, bool force_nonblock,
/* IOPOLL retry should happen for io-wq threads */
if (!force_nonblock && !(req->ctx->flags & IORING_SETUP_IOPOLL))
goto done;
+ /* no retry on NONBLOCK marked file */
+ if (kiocb->ki_flags & IOCB_NOWAIT)
+ goto done;
/* some cases will consume bytes even on error returns */
iov_iter_revert(iter, iov_count - iov_iter_count(iter));
ret = io_setup_async_rw(req, iovec, inline_vecs, iter, false);
--
Jens Axboe
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: IORING_OP_READ and O_NONBLOCK behaviour
2020-09-02 15:35 ` Jens Axboe
@ 2020-09-02 16:00 ` Pavel Begunkov
2020-09-02 16:02 ` Pavel Begunkov
2020-09-02 16:05 ` Jens Axboe
0 siblings, 2 replies; 7+ messages in thread
From: Pavel Begunkov @ 2020-09-02 16:00 UTC (permalink / raw)
To: Jens Axboe, Norman Maurer, io-uring; +Cc: Josef
On 02/09/2020 18:35, Jens Axboe wrote:
> On 9/2/20 9:26 AM, Pavel Begunkov wrote:
>> On 02/09/2020 17:45, Jens Axboe wrote:
>>> On 9/2/20 4:09 AM, Norman Maurer wrote:
>>>> Hi there,
>>>>
>>>> We are currently working on integrating io_uring into netty and found
>>>> some “suprising” behaviour which seems like a bug to me.
>>>>
>>>> When a socket is marked as non blocking (accepted with O_NONBLOCK flag
>>>> set) and there is no data to be read IORING_OP_READ should complete
>>>> directly with EAGAIN or EWOULDBLOCK. This is not the case and it
>>>> basically blocks forever until there is some data to read. Is this
>>>> expected ?
>>>>
>>>> This seems to be somehow related to a bug that was fixed for
>>>> IO_URING_ACCEPT with non blocking sockets:
>>>>
>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=e697deed834de15d2322d0619d51893022c90ea2
>>>
>>> I agree with you that this is a bug, in general it's useful (and
>>> expected) that we'd return -EAGAIN for that case. I'll take a look.
>>>
>>
>> That's I mentioned that doing retries for nonblock requests in
>> io_wq_submit_work() doesn't look consistent. I think killing it
>> off may help.
>
> Right, we should not retry those _in general_, the exception is regular
> files or block devices to handle IOPOLL retry where we do need it. The
> below is what I came up with for this one. Might not hurt to make this
> more explicit for 5.10.
Hmm, I didn't checked it, but if we
>
>
> commit c78e0f02c3861b5b176b2f79552677b3604deb76
> Author: Jens Axboe <[email protected]>
> Date: Wed Sep 2 09:30:31 2020 -0600
>
> io_uring: no read-retry on -EAGAIN error and O_NONBLOCK marked file
>
> Actually two things that need fixing up here:
>
> - The io_rw_reissue() -EAGAIN retry is explicit to block devices and
> regular files, so don't ever attempt to do that on other types of
> files.
>
> - If we hit -EAGAIN on a nonblock marked file, don't arm poll handler for
> it. It should just complete with -EAGAIN.
>
> Cc: [email protected]
> Reported-by: Norman Maurer <[email protected]>
> Signed-off-by: Jens Axboe <[email protected]>
>
> diff --git a/fs/io_uring.c b/fs/io_uring.c
> index b1ccd7072d93..dc27cd5b8ad6 100644
> --- a/fs/io_uring.c
> +++ b/fs/io_uring.c
> @@ -2300,8 +2300,11 @@ static bool io_resubmit_prep(struct io_kiocb *req, int error)
> static bool io_rw_reissue(struct io_kiocb *req, long res)
> {
> #ifdef CONFIG_BLOCK
> + umode_t mode = file_inode(req->file)->i_mode;
> int ret;
>
> + if (!S_ISBLK(mode) && !S_ISREG(mode))
> + return false;
> if ((res != -EAGAIN && res != -EOPNOTSUPP) || io_wq_current_is_worker())
> return false;
>
> @@ -3146,6 +3149,9 @@ static int io_read(struct io_kiocb *req, bool force_nonblock,
> /* IOPOLL retry should happen for io-wq threads */
> if (!force_nonblock && !(req->ctx->flags & IORING_SETUP_IOPOLL))
> goto done;
> + /* no retry on NONBLOCK marked file */
> + if (kiocb->ki_flags & IOCB_NOWAIT)
> + goto done;
We clearing and setting IOCB_NOWAIT depending on @force_nonblock, so it may not
work. E.g. with IOSQE_IO_ASYNC io_read() will clear it at the beginning.
Maybe REQ_F_NOWAIT?
> /* some cases will consume bytes even on error returns */
> iov_iter_revert(iter, iov_count - iov_iter_count(iter));
> ret = io_setup_async_rw(req, iovec, inline_vecs, iter, false);
>
--
Pavel Begunkov
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: IORING_OP_READ and O_NONBLOCK behaviour
2020-09-02 16:00 ` Pavel Begunkov
@ 2020-09-02 16:02 ` Pavel Begunkov
2020-09-02 16:05 ` Jens Axboe
1 sibling, 0 replies; 7+ messages in thread
From: Pavel Begunkov @ 2020-09-02 16:02 UTC (permalink / raw)
To: Jens Axboe, Norman Maurer, io-uring; +Cc: Josef
On 02/09/2020 19:00, Pavel Begunkov wrote:
> On 02/09/2020 18:35, Jens Axboe wrote:
>> On 9/2/20 9:26 AM, Pavel Begunkov wrote:
>>> On 02/09/2020 17:45, Jens Axboe wrote:
>>>> On 9/2/20 4:09 AM, Norman Maurer wrote:
>>>>> Hi there,
>>>>>
>>>>> We are currently working on integrating io_uring into netty and found
>>>>> some “suprising” behaviour which seems like a bug to me.
>>>>>
>>>>> When a socket is marked as non blocking (accepted with O_NONBLOCK flag
>>>>> set) and there is no data to be read IORING_OP_READ should complete
>>>>> directly with EAGAIN or EWOULDBLOCK. This is not the case and it
>>>>> basically blocks forever until there is some data to read. Is this
>>>>> expected ?
>>>>>
>>>>> This seems to be somehow related to a bug that was fixed for
>>>>> IO_URING_ACCEPT with non blocking sockets:
>>>>>
>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=e697deed834de15d2322d0619d51893022c90ea2
>>>>
>>>> I agree with you that this is a bug, in general it's useful (and
>>>> expected) that we'd return -EAGAIN for that case. I'll take a look.
>>>>
>>>
>>> That's I mentioned that doing retries for nonblock requests in
>>> io_wq_submit_work() doesn't look consistent. I think killing it
>>> off may help.
>>
>> Right, we should not retry those _in general_, the exception is regular
>> files or block devices to handle IOPOLL retry where we do need it. The
>> below is what I came up with for this one. Might not hurt to make this
>> more explicit for 5.10.
>
> Hmm, I didn't checked it, but if we
Oops, garbage text.
>
>>
>>
>> commit c78e0f02c3861b5b176b2f79552677b3604deb76
>> Author: Jens Axboe <[email protected]>
>> Date: Wed Sep 2 09:30:31 2020 -0600
>>
>> io_uring: no read-retry on -EAGAIN error and O_NONBLOCK marked file
>>
>> Actually two things that need fixing up here:
>>
>> - The io_rw_reissue() -EAGAIN retry is explicit to block devices and
>> regular files, so don't ever attempt to do that on other types of
>> files.
>>
>> - If we hit -EAGAIN on a nonblock marked file, don't arm poll handler for
>> it. It should just complete with -EAGAIN.
>>
>> Cc: [email protected]
>> Reported-by: Norman Maurer <[email protected]>
>> Signed-off-by: Jens Axboe <[email protected]>
>>
>> diff --git a/fs/io_uring.c b/fs/io_uring.c
>> index b1ccd7072d93..dc27cd5b8ad6 100644
>> --- a/fs/io_uring.c
>> +++ b/fs/io_uring.c
>> @@ -2300,8 +2300,11 @@ static bool io_resubmit_prep(struct io_kiocb *req, int error)
>> static bool io_rw_reissue(struct io_kiocb *req, long res)
>> {
>> #ifdef CONFIG_BLOCK
>> + umode_t mode = file_inode(req->file)->i_mode;
>> int ret;
>>
>> + if (!S_ISBLK(mode) && !S_ISREG(mode))
>> + return false;
>> if ((res != -EAGAIN && res != -EOPNOTSUPP) || io_wq_current_is_worker())
>> return false;
>>
>> @@ -3146,6 +3149,9 @@ static int io_read(struct io_kiocb *req, bool force_nonblock,
>> /* IOPOLL retry should happen for io-wq threads */
>> if (!force_nonblock && !(req->ctx->flags & IORING_SETUP_IOPOLL))
>> goto done;
>> + /* no retry on NONBLOCK marked file */
>> + if (kiocb->ki_flags & IOCB_NOWAIT)
>> + goto done;
>
> We clearing and setting IOCB_NOWAIT depending on @force_nonblock, so it may not
> work. E.g. with IOSQE_IO_ASYNC io_read() will clear it at the beginning.
> Maybe REQ_F_NOWAIT?
>
>> /* some cases will consume bytes even on error returns */
>> iov_iter_revert(iter, iov_count - iov_iter_count(iter));
>> ret = io_setup_async_rw(req, iovec, inline_vecs, iter, false);
>>
>
--
Pavel Begunkov
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: IORING_OP_READ and O_NONBLOCK behaviour
2020-09-02 16:00 ` Pavel Begunkov
2020-09-02 16:02 ` Pavel Begunkov
@ 2020-09-02 16:05 ` Jens Axboe
1 sibling, 0 replies; 7+ messages in thread
From: Jens Axboe @ 2020-09-02 16:05 UTC (permalink / raw)
To: Pavel Begunkov, Norman Maurer, io-uring; +Cc: Josef
On 9/2/20 10:00 AM, Pavel Begunkov wrote:
> On 02/09/2020 18:35, Jens Axboe wrote:
>> On 9/2/20 9:26 AM, Pavel Begunkov wrote:
>>> On 02/09/2020 17:45, Jens Axboe wrote:
>>>> On 9/2/20 4:09 AM, Norman Maurer wrote:
>>>>> Hi there,
>>>>>
>>>>> We are currently working on integrating io_uring into netty and found
>>>>> some “suprising” behaviour which seems like a bug to me.
>>>>>
>>>>> When a socket is marked as non blocking (accepted with O_NONBLOCK flag
>>>>> set) and there is no data to be read IORING_OP_READ should complete
>>>>> directly with EAGAIN or EWOULDBLOCK. This is not the case and it
>>>>> basically blocks forever until there is some data to read. Is this
>>>>> expected ?
>>>>>
>>>>> This seems to be somehow related to a bug that was fixed for
>>>>> IO_URING_ACCEPT with non blocking sockets:
>>>>>
>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=e697deed834de15d2322d0619d51893022c90ea2
>>>>
>>>> I agree with you that this is a bug, in general it's useful (and
>>>> expected) that we'd return -EAGAIN for that case. I'll take a look.
>>>>
>>>
>>> That's I mentioned that doing retries for nonblock requests in
>>> io_wq_submit_work() doesn't look consistent. I think killing it
>>> off may help.
>>
>> Right, we should not retry those _in general_, the exception is regular
>> files or block devices to handle IOPOLL retry where we do need it. The
>> below is what I came up with for this one. Might not hurt to make this
>> more explicit for 5.10.
>
> Hmm, I didn't checked it, but if we
>
>>
>>
>> commit c78e0f02c3861b5b176b2f79552677b3604deb76
>> Author: Jens Axboe <[email protected]>
>> Date: Wed Sep 2 09:30:31 2020 -0600
>>
>> io_uring: no read-retry on -EAGAIN error and O_NONBLOCK marked file
>>
>> Actually two things that need fixing up here:
>>
>> - The io_rw_reissue() -EAGAIN retry is explicit to block devices and
>> regular files, so don't ever attempt to do that on other types of
>> files.
>>
>> - If we hit -EAGAIN on a nonblock marked file, don't arm poll handler for
>> it. It should just complete with -EAGAIN.
>>
>> Cc: [email protected]
>> Reported-by: Norman Maurer <[email protected]>
>> Signed-off-by: Jens Axboe <[email protected]>
>>
>> diff --git a/fs/io_uring.c b/fs/io_uring.c
>> index b1ccd7072d93..dc27cd5b8ad6 100644
>> --- a/fs/io_uring.c
>> +++ b/fs/io_uring.c
>> @@ -2300,8 +2300,11 @@ static bool io_resubmit_prep(struct io_kiocb *req, int error)
>> static bool io_rw_reissue(struct io_kiocb *req, long res)
>> {
>> #ifdef CONFIG_BLOCK
>> + umode_t mode = file_inode(req->file)->i_mode;
>> int ret;
>>
>> + if (!S_ISBLK(mode) && !S_ISREG(mode))
>> + return false;
>> if ((res != -EAGAIN && res != -EOPNOTSUPP) || io_wq_current_is_worker())
>> return false;
>>
>> @@ -3146,6 +3149,9 @@ static int io_read(struct io_kiocb *req, bool force_nonblock,
>> /* IOPOLL retry should happen for io-wq threads */
>> if (!force_nonblock && !(req->ctx->flags & IORING_SETUP_IOPOLL))
>> goto done;
>> + /* no retry on NONBLOCK marked file */
>> + if (kiocb->ki_flags & IOCB_NOWAIT)
>> + goto done;
>
> We clearing and setting IOCB_NOWAIT depending on @force_nonblock, so it may not
> work. E.g. with IOSQE_IO_ASYNC io_read() will clear it at the beginning.
> Maybe REQ_F_NOWAIT?
Yeah, the posted version did get it right:
https://lore.kernel.org/io-uring/[email protected]/T/#u
--
Jens Axboe
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2020-09-02 16:05 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-09-02 10:09 IORING_OP_READ and O_NONBLOCK behaviour Norman Maurer
2020-09-02 14:45 ` Jens Axboe
2020-09-02 15:26 ` Pavel Begunkov
2020-09-02 15:35 ` Jens Axboe
2020-09-02 16:00 ` Pavel Begunkov
2020-09-02 16:02 ` Pavel Begunkov
2020-09-02 16:05 ` Jens Axboe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox