[PATCH for-5.10] io_uring: remove req cancel in ->flush()

[PATCH for-5.10] io_uring: remove req cancel in ->flush()

[Pavel Begunkov]

Every close(io_uring) causes cancellation of all inflight requests
carrying ->files. That's not nice but was neccessary up until recently.
Now task->files removal is handled in the core code, so that part of
flush can be removed.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
 fs/io_uring.c | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/fs/io_uring.c b/fs/io_uring.c
index 95d2bb7069c6..6536e24eb44e 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -8748,16 +8748,12 @@ void __io_uring_task_cancel(void)
 
 static int io_uring_flush(struct file *file, void *data)
 {
-	struct io_ring_ctx *ctx = file->private_data;
+	bool exiting = !data;
 
-	/*
-	 * If the task is going away, cancel work it may have pending
-	 */
 	if (fatal_signal_pending(current) || (current->flags & PF_EXITING))
-		data = NULL;
+		exiting = true;
 
-	io_uring_cancel_task_requests(ctx, data);
-	io_uring_attempt_task_drop(file, !data);
+	io_uring_attempt_task_drop(file, exiting);
 	return 0;
 }
 
-- 
2.24.0

Re: [PATCH for-5.10] io_uring: remove req cancel in ->flush()

[Jens Axboe]

On 10/19/20 9:45 AM, Pavel Begunkov wrote:
> Every close(io_uring) causes cancellation of all inflight requests
> carrying ->files. That's not nice but was neccessary up until recently.
> Now task->files removal is handled in the core code, so that part of
> flush can be removed.

It does change the behavior, but I'd wager that's safe. One minor
comment:

> diff --git a/fs/io_uring.c b/fs/io_uring.c
> index 95d2bb7069c6..6536e24eb44e 100644
> --- a/fs/io_uring.c
> +++ b/fs/io_uring.c
> @@ -8748,16 +8748,12 @@ void __io_uring_task_cancel(void)
>  
>  static int io_uring_flush(struct file *file, void *data)
>  {
> -	struct io_ring_ctx *ctx = file->private_data;
> +	bool exiting = !data;
>  
> -	/*
> -	 * If the task is going away, cancel work it may have pending
> -	 */
>  	if (fatal_signal_pending(current) || (current->flags & PF_EXITING))
> -		data = NULL;
> +		exiting = true;
>  
> -	io_uring_cancel_task_requests(ctx, data);
> -	io_uring_attempt_task_drop(file, !data);
> +	io_uring_attempt_task_drop(file, exiting);
>  	return 0;
>  }

Why not just keep the !data for task_drop? Would make the diff take
away just the hunk we're interested in. Even adding a comment would be
better, imho.

-- 
Jens Axboe

Re: [PATCH for-5.10] io_uring: remove req cancel in ->flush()

[Pavel Begunkov]

On 19/10/2020 21:08, Jens Axboe wrote:
> On 10/19/20 9:45 AM, Pavel Begunkov wrote:
>> Every close(io_uring) causes cancellation of all inflight requests
>> carrying ->files. That's not nice but was neccessary up until recently.
>> Now task->files removal is handled in the core code, so that part of
>> flush can be removed.
> 
> It does change the behavior, but I'd wager that's safe. One minor
> comment:

Right, but I would think that users are not happy that every close
kills requests without apparent reasons.

> 
>> diff --git a/fs/io_uring.c b/fs/io_uring.c
>> index 95d2bb7069c6..6536e24eb44e 100644
>> --- a/fs/io_uring.c
>> +++ b/fs/io_uring.c
>> @@ -8748,16 +8748,12 @@ void __io_uring_task_cancel(void)
>>  
>>  static int io_uring_flush(struct file *file, void *data)
>>  {
>> -	struct io_ring_ctx *ctx = file->private_data;
>> +	bool exiting = !data;
>>  
>> -	/*
>> -	 * If the task is going away, cancel work it may have pending
>> -	 */
>>  	if (fatal_signal_pending(current) || (current->flags & PF_EXITING))
>> -		data = NULL;
>> +		exiting = true;
>>  
>> -	io_uring_cancel_task_requests(ctx, data);
>> -	io_uring_attempt_task_drop(file, !data);
>> +	io_uring_attempt_task_drop(file, exiting);
>>  	return 0;
>>  }
> 
> Why not just keep the !data for task_drop? Would make the diff take
> away just the hunk we're interested in. Even adding a comment would be
> better, imho.

That would look cleaner, but I just left what already was there. TBH,
I don't even entirely understand why exiting=!data. Looking up how
exit_files() works, it passes down non-NULL files to
put_files_struct() -> ... filp_close() -> f_op->flush().

I'm curious how does this filp_close(file, files=NULL) happens?

Moreover, if that's exit_files() which is interesting, then first
it calls io_uring_cancel_task_requests(), which should remove all
struct file from tctx->xa. I haven't tested it though.

-- 
Pavel Begunkov

Re: [PATCH for-5.10] io_uring: remove req cancel in ->flush()

[Jens Axboe]

On 10/19/20 5:40 PM, Pavel Begunkov wrote:
> On 19/10/2020 21:08, Jens Axboe wrote:
>> On 10/19/20 9:45 AM, Pavel Begunkov wrote:
>>> Every close(io_uring) causes cancellation of all inflight requests
>>> carrying ->files. That's not nice but was neccessary up until recently.
>>> Now task->files removal is handled in the core code, so that part of
>>> flush can be removed.
>>
>> It does change the behavior, but I'd wager that's safe. One minor
>> comment:
> 
> Right, but I would think that users are not happy that every close
> kills requests without apparent reasons.

To be fair, closing the ring fd is kind of unexpected and I would not
expect any real applications to do that. If they did, I would have
expected queries on why file table requests get canceled on close. Hence
I'm not too worried about anyone hitting any issues related to this
change, as the change is certainly one for the better.

>>> diff --git a/fs/io_uring.c b/fs/io_uring.c
>>> index 95d2bb7069c6..6536e24eb44e 100644
>>> --- a/fs/io_uring.c
>>> +++ b/fs/io_uring.c
>>> @@ -8748,16 +8748,12 @@ void __io_uring_task_cancel(void)
>>>  
>>>  static int io_uring_flush(struct file *file, void *data)
>>>  {
>>> -	struct io_ring_ctx *ctx = file->private_data;
>>> +	bool exiting = !data;
>>>  
>>> -	/*
>>> -	 * If the task is going away, cancel work it may have pending
>>> -	 */
>>>  	if (fatal_signal_pending(current) || (current->flags & PF_EXITING))
>>> -		data = NULL;
>>> +		exiting = true;
>>>  
>>> -	io_uring_cancel_task_requests(ctx, data);
>>> -	io_uring_attempt_task_drop(file, !data);
>>> +	io_uring_attempt_task_drop(file, exiting);
>>>  	return 0;
>>>  }
>>
>> Why not just keep the !data for task_drop? Would make the diff take
>> away just the hunk we're interested in. Even adding a comment would be
>> better, imho.
> 
> That would look cleaner, but I just left what already was there. TBH,
> I don't even entirely understand why exiting=!data. Looking up how
> exit_files() works, it passes down non-NULL files to
> put_files_struct() -> ... filp_close() -> f_op->flush().
> 
> I'm curious how does this filp_close(file, files=NULL) happens?

It doesn't, we just clear it internall to match all requests, not just
files backed ones.

> Moreover, if that's exit_files() which is interesting, then first
> it calls io_uring_cancel_task_requests(), which should remove all
> struct file from tctx->xa. I haven't tested it though.

Yep, further cleanups are certainly possible there.

I've queued this up, thanks.

-- 
Jens Axboe

Re: [PATCH for-5.10] io_uring: remove req cancel in ->flush()

[Pavel Begunkov]

On 20/10/2020 15:09, Jens Axboe wrote:
> On 10/19/20 5:40 PM, Pavel Begunkov wrote:
>> On 19/10/2020 21:08, Jens Axboe wrote:
>>> On 10/19/20 9:45 AM, Pavel Begunkov wrote:
>>>> Every close(io_uring) causes cancellation of all inflight requests
>>>> carrying ->files. That's not nice but was neccessary up until recently.
>>>> Now task->files removal is handled in the core code, so that part of
>>>> flush can be removed.
>>>
>>> Why not just keep the !data for task_drop? Would make the diff take
>>> away just the hunk we're interested in. Even adding a comment would be
>>> better, imho.
>>
>> That would look cleaner, but I just left what already was there. TBH,
>> I don't even entirely understand why exiting=!data. Looking up how
>> exit_files() works, it passes down non-NULL files to
>> put_files_struct() -> ... filp_close() -> f_op->flush().
>>
>> I'm curious how does this filp_close(file, files=NULL) happens?
> 
> It doesn't, we just clear it internall to match all requests, not just
> files backed ones.

Then my "bool exiting = !data;" at the start doesn't make sense since
passed in @data is always non-NULL.

> 
>> Moreover, if that's exit_files() which is interesting, then first
>> it calls io_uring_cancel_task_requests(), which should remove all
>> struct file from tctx->xa. I haven't tested it though.
> 
> Yep, further cleanups are certainly possible there.
> 
> I've queued this up, thanks.
> 

-- 
Pavel Begunkov

Re: [PATCH for-5.10] io_uring: remove req cancel in ->flush()

[Jens Axboe]

On 10/20/20 10:29 AM, Pavel Begunkov wrote:
> On 20/10/2020 15:09, Jens Axboe wrote:
>> On 10/19/20 5:40 PM, Pavel Begunkov wrote:
>>> On 19/10/2020 21:08, Jens Axboe wrote:
>>>> On 10/19/20 9:45 AM, Pavel Begunkov wrote:
>>>>> Every close(io_uring) causes cancellation of all inflight requests
>>>>> carrying ->files. That's not nice but was neccessary up until recently.
>>>>> Now task->files removal is handled in the core code, so that part of
>>>>> flush can be removed.
>>>>
>>>> Why not just keep the !data for task_drop? Would make the diff take
>>>> away just the hunk we're interested in. Even adding a comment would be
>>>> better, imho.
>>>
>>> That would look cleaner, but I just left what already was there. TBH,
>>> I don't even entirely understand why exiting=!data. Looking up how
>>> exit_files() works, it passes down non-NULL files to
>>> put_files_struct() -> ... filp_close() -> f_op->flush().
>>>
>>> I'm curious how does this filp_close(file, files=NULL) happens?
>>
>> It doesn't, we just clear it internall to match all requests, not just
>> files backed ones.
> 
> Then my "bool exiting = !data;" at the start doesn't make sense since
> passed in @data is always non-NULL.

Right, only if we retain this part:

if (fatal_signal_pending(current) || (current->flags & PF_EXITING))
	data = NULL;

in there. I suspect we'll want something ala the below instead.

diff --git a/fs/io_uring.c b/fs/io_uring.c
index 09e7a5f20060..4870db000f04 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -8695,14 +8695,18 @@ static void __io_uring_attempt_task_drop(struct file *file)
  * Drop task note for this file if we're the only ones that hold it after
  * pending fput()
  */
-static void io_uring_attempt_task_drop(struct file *file, bool exiting)
+static void io_uring_attempt_task_drop(struct file *file)
 {
+	bool exiting;
+
 	if (!current->io_uring)
 		return;
+
 	/*
 	 * fput() is pending, will be 2 if the only other ref is our potential
 	 * task file note. If the task is exiting, drop regardless of count.
 	 */
+	exiting = fatal_signal_pending(current) || (current->flags & PF_EXITING);
 	if (!exiting && atomic_long_read(&file->f_count) != 2)
 		return;
 
@@ -8764,16 +8768,7 @@ void __io_uring_task_cancel(void)
 
 static int io_uring_flush(struct file *file, void *data)
 {
-	struct io_ring_ctx *ctx = file->private_data;
-
-	/*
-	 * If the task is going away, cancel work it may have pending
-	 */
-	if (fatal_signal_pending(current) || (current->flags & PF_EXITING))
-		data = NULL;
-
-	io_uring_cancel_task_requests(ctx, data);
-	io_uring_attempt_task_drop(file, !data);
+	io_uring_attempt_task_drop(file);
 	return 0;
 }
 

-- 
Jens Axboe

Re: [PATCH for-5.10] io_uring: remove req cancel in ->flush()

[Xiaoguang Wang]

hi,

> Every close(io_uring) causes cancellation of all inflight requests
> carrying ->files. That's not nice but was neccessary up until recently.
> Now task->files removal is handled in the core code, so that part of
> flush can be removed.
I don't catch up with newest io_uring codes yet, but have one question about
the initial implementation "io_uring: io_uring: add support for async work
inheriting files": https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fcb323cc53e29d9cc696d606bb42736b32dd9825
There was such comments:
+static int io_grab_files(struct io_ring_ctx *ctx, struct io_kiocb *req)
+{
+       int ret = -EBADF;
+
+       rcu_read_lock();
+       spin_lock_irq(&ctx->inflight_lock);
+       /*
+        * We use the f_ops->flush() handler to ensure that we can flush
+        * out work accessing these files if the fd is closed. Check if
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
I wonder why we only need to flush reqs specifically when they access current->files, are there
any special reasons?

Regards,
Xiaoguang Wang

+        * the fd has changed since we started down this path, and disallow
+        * this operation if it has.
+        */
+       if (fcheck(req->submit.ring_fd) == req->submit.ring_file) {
+               list_add(&req->inflight_entry, &ctx->inflight_list);
+               req->flags |= REQ_F_INFLIGHT;
+               req->work.files = current->files;
+               ret = 0;
+       }
> 
> Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
> ---
>   fs/io_uring.c | 10 +++-------
>   1 file changed, 3 insertions(+), 7 deletions(-)
> 
> diff --git a/fs/io_uring.c b/fs/io_uring.c
> index 95d2bb7069c6..6536e24eb44e 100644
> --- a/fs/io_uring.c
> +++ b/fs/io_uring.c
> @@ -8748,16 +8748,12 @@ void __io_uring_task_cancel(void)
>   
>   static int io_uring_flush(struct file *file, void *data)
>   {
> -	struct io_ring_ctx *ctx = file->private_data;
> +	bool exiting = !data;
>   
> -	/*
> -	 * If the task is going away, cancel work it may have pending
> -	 */
>   	if (fatal_signal_pending(current) || (current->flags & PF_EXITING))
> -		data = NULL;
> +		exiting = true;
>   
> -	io_uring_cancel_task_requests(ctx, data);
> -	io_uring_attempt_task_drop(file, !data);
> +	io_uring_attempt_task_drop(file, exiting);
>   	return 0;
>   }
>   
> 

Re: [PATCH for-5.10] io_uring: remove req cancel in ->flush()

[Pavel Begunkov]

On 22/10/2020 07:42, Xiaoguang Wang wrote:
>> Every close(io_uring) causes cancellation of all inflight requests
>> carrying ->files. That's not nice but was neccessary up until recently.
>> Now task->files removal is handled in the core code, so that part of
>> flush can be removed.
> I don't catch up with newest io_uring codes yet, but have one question about
> the initial implementation "io_uring: io_uring: add support for async work
> inheriting files": https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fcb323cc53e29d9cc696d606bb42736b32dd9825
> There was such comments:
> +static int io_grab_files(struct io_ring_ctx *ctx, struct io_kiocb *req)
> +{
> +       int ret = -EBADF;
> +
> +       rcu_read_lock();
> +       spin_lock_irq(&ctx->inflight_lock);
> +       /*
> +        * We use the f_ops->flush() handler to ensure that we can flush
> +        * out work accessing these files if the fd is closed. Check if
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> I wonder why we only need to flush reqs specifically when they access current->files, are there
> any special reasons?
We could have taken a reference to ->files, so it doesn't get freed
while a request is using it, but that creates a circular dependency.

IIUC, because if there are ->files refs io_uring won't be closed on exit,
but io_uring would be holding ->files refs.

So, it was working without taking ->files references (i.e. weak refs)
on the basis that the files won't be destroyed until the task itself is
gone, and we can *kind of* intercept when task is exiting by ->flush()
and cancel all requests with ->files there.

-- 
Pavel Begunkov

Re: [PATCH for-5.10] io_uring: remove req cancel in ->flush()

[Xiaoguang Wang]

hi,

> On 22/10/2020 07:42, Xiaoguang Wang wrote:
>>> Every close(io_uring) causes cancellation of all inflight requests
>>> carrying ->files. That's not nice but was neccessary up until recently.
>>> Now task->files removal is handled in the core code, so that part of
>>> flush can be removed.
>> I don't catch up with newest io_uring codes yet, but have one question about
>> the initial implementation "io_uring: io_uring: add support for async work
>> inheriting files": https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fcb323cc53e29d9cc696d606bb42736b32dd9825
>> There was such comments:
>> +static int io_grab_files(struct io_ring_ctx *ctx, struct io_kiocb *req)
>> +{
>> +       int ret = -EBADF;
>> +
>> +       rcu_read_lock();
>> +       spin_lock_irq(&ctx->inflight_lock);
>> +       /*
>> +        * We use the f_ops->flush() handler to ensure that we can flush
>> +        * out work accessing these files if the fd is closed. Check if
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> I wonder why we only need to flush reqs specifically when they access current->files, are there
>> any special reasons?
> We could have taken a reference to ->files, so it doesn't get freed
> while a request is using it, but that creates a circular dependency.
> 
> IIUC, because if there are ->files refs io_uring won't be closed on exit,
> but io_uring would be holding ->files refs.
> 
> So, it was working without taking ->files references (i.e. weak refs)
> on the basis that the files won't be destroyed until the task itself is
> gone, and we can *kind of* intercept when task is exiting by ->flush()
> and cancel all requests with ->files there.
Now I see :) will help me to understand current codes better.
Really thanks for your explanations.

Regards,
Xiaoguang Wang
>