Re: IORING_OP_READ and O_NONBLOCK behaviour

[Jens Axboe <axboe@kernel.dk>]

On 9/2/20 10:00 AM, Pavel Begunkov wrote:
> On 02/09/2020 18:35, Jens Axboe wrote:
>> On 9/2/20 9:26 AM, Pavel Begunkov wrote:
>>> On 02/09/2020 17:45, Jens Axboe wrote:
>>>> On 9/2/20 4:09 AM, Norman Maurer wrote:
>>>>> Hi there,
>>>>>
>>>>> We are currently working on integrating io_uring into netty and found
>>>>> some “suprising” behaviour which seems like a bug to me.
>>>>>
>>>>> When a socket is marked as non blocking (accepted with O_NONBLOCK flag
>>>>> set) and there is no data to be read IORING_OP_READ should complete
>>>>> directly with EAGAIN or EWOULDBLOCK. This is not the case and it
>>>>> basically blocks forever until there is some data to read. Is this
>>>>> expected ?
>>>>>
>>>>> This seems to be somehow related to a bug that was fixed for
>>>>> IO_URING_ACCEPT with non blocking sockets:
>>>>>
>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v5.8&id=e697deed834de15d2322d0619d51893022c90ea2
>>>>
>>>> I agree with you that this is a bug, in general it's useful (and
>>>> expected) that we'd return -EAGAIN for that case. I'll take a look.
>>>>
>>>
>>> That's I mentioned that doing retries for nonblock requests in
>>> io_wq_submit_work() doesn't look consistent. I think killing it
>>> off may help.
>>
>> Right, we should not retry those _in general_, the exception is regular
>> files or block devices to handle IOPOLL retry where we do need it. The
>> below is what I came up with for this one. Might not hurt to make this
>> more explicit for 5.10.
> 
> Hmm, I didn't checked it, but if we 
> 
>>
>>
>> commit c78e0f02c3861b5b176b2f79552677b3604deb76
>> Author: Jens Axboe <axboe@kernel.dk>
>> Date:   Wed Sep 2 09:30:31 2020 -0600
>>
>>     io_uring: no read-retry on -EAGAIN error and O_NONBLOCK marked file
>>     
>>     Actually two things that need fixing up here:
>>     
>>     - The io_rw_reissue() -EAGAIN retry is explicit to block devices and
>>       regular files, so don't ever attempt to do that on other types of
>>       files.
>>     
>>     - If we hit -EAGAIN on a nonblock marked file, don't arm poll handler for
>>       it. It should just complete with -EAGAIN.
>>     
>>     Cc: stable@vger.kernel.org
>>     Reported-by: Norman Maurer <norman.maurer@googlemail.com>
>>     Signed-off-by: Jens Axboe <axboe@kernel.dk>
>>
>> diff --git a/fs/io_uring.c b/fs/io_uring.c
>> index b1ccd7072d93..dc27cd5b8ad6 100644
>> --- a/fs/io_uring.c
>> +++ b/fs/io_uring.c
>> @@ -2300,8 +2300,11 @@ static bool io_resubmit_prep(struct io_kiocb *req, int error)
>>  static bool io_rw_reissue(struct io_kiocb *req, long res)
>>  {
>>  #ifdef CONFIG_BLOCK
>> +	umode_t mode = file_inode(req->file)->i_mode;
>>  	int ret;
>>  
>> +	if (!S_ISBLK(mode) && !S_ISREG(mode))
>> +		return false;
>>  	if ((res != -EAGAIN && res != -EOPNOTSUPP) || io_wq_current_is_worker())
>>  		return false;
>>  
>> @@ -3146,6 +3149,9 @@ static int io_read(struct io_kiocb *req, bool force_nonblock,
>>  		/* IOPOLL retry should happen for io-wq threads */
>>  		if (!force_nonblock && !(req->ctx->flags & IORING_SETUP_IOPOLL))
>>  			goto done;
>> +		/* no retry on NONBLOCK marked file */
>> +		if (kiocb->ki_flags & IOCB_NOWAIT)
>> +			goto done;
> 
> We clearing and setting IOCB_NOWAIT depending on @force_nonblock, so it may not
> work. E.g. with IOSQE_IO_ASYNC io_read() will clear it at the beginning.
> Maybe REQ_F_NOWAIT?

Yeah, the posted version did get it right:

https://lore.kernel.org/io-uring/5d91a8ea-5748-803a-d2dc-ef21fe27e39e@kernel.dk/T/#u


-- 
Jens Axboe