[PATCH 1/2] io_uring: wait potential ->release() on resurrect

[PATCH 1/2] io_uring: wait potential ->release() on resurrect

[Pavel Begunkov]

There is a short window where percpu_refs are already turned zero, but
we try to do resurrect(). Play nicer and wait for all users to leave RCU
section.

Cc: <stable@vger.kernel.org> # 5.5+
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
 fs/io_uring.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/fs/io_uring.c b/fs/io_uring.c
index f3af499b12a9..ce5fccf00367 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -7351,6 +7351,7 @@ static int io_rsrc_ref_quiesce(struct fixed_rsrc_data *data,
 			break;
 
 		percpu_ref_resurrect(&data->refs);
+		synchronize_rcu();
 		io_sqe_rsrc_set_node(ctx, data, backup_node);
 		reinit_completion(&data->done);
 		mutex_unlock(&ctx->uring_lock);
@@ -10089,6 +10090,7 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,
 
 		if (ret) {
 			percpu_ref_resurrect(&ctx->refs);
+			synchronize_rcu();
 			goto out_quiesce;
 		}
 	}
-- 
2.24.0

Re: [PATCH 1/2] io_uring: wait potential ->release() on resurrect

[Jens Axboe]

On 2/19/21 6:39 PM, Pavel Begunkov wrote:
> There is a short window where percpu_refs are already turned zero, but
> we try to do resurrect(). Play nicer and wait for all users to leave RCU
> section.

We need to do something better than synchronize_rcu() here, that can
take a long time on a loaded box. I'll try and think about this one.

-- 
Jens Axboe

Re: [PATCH 1/2] io_uring: wait potential ->release() on resurrect

[Pavel Begunkov]

On 20/02/2021 03:40, Jens Axboe wrote:
> On 2/19/21 6:39 PM, Pavel Begunkov wrote:
>> There is a short window where percpu_refs are already turned zero, but
>> we try to do resurrect(). Play nicer and wait for all users to leave RCU
>> section.
> 
> We need to do something better than synchronize_rcu() here, that can
> take a long time on a loaded box. I'll try and think about this one.

It only happens when it can't be drained and there are task_works or
signals. I have another patch, doing it via tryget, but it's uglier and
I'd rather prefer synchronize_rcu for stable.

Want me to send it tomorrow (on top or not)?

-- 
Pavel Begunkov

Re: [PATCH 1/2] io_uring: wait potential ->release() on resurrect

[Jens Axboe]

On 2/19/21 8:47 PM, Pavel Begunkov wrote:
> On 20/02/2021 03:40, Jens Axboe wrote:
>> On 2/19/21 6:39 PM, Pavel Begunkov wrote:
>>> There is a short window where percpu_refs are already turned zero, but
>>> we try to do resurrect(). Play nicer and wait for all users to leave RCU
>>> section.
>>
>> We need to do something better than synchronize_rcu() here, that can
>> take a long time on a loaded box. I'll try and think about this one.
> 
> It only happens when it can't be drained and there are task_works or
> signals. I have another patch, doing it via tryget, but it's uglier and
> I'd rather prefer synchronize_rcu for stable.

Right, but the task_work coming in may not be unlikely. So it's not
strictly an error path.

-- 
Jens Axboe

[PATCH 0/2] resurrect

[Pavel Begunkov]

Combined approach with tryget, fallbacking to trying completion and
doing synchronize_rcu, where the last one is very unlikely to happen
and limited by one such per register syscall.

I expect 1/2 to cleanly apply for stable, 2/2 needs to be backported
by hand.

Pavel Begunkov (2):
  io_uring: wait potential ->release() on resurrect
  io_uring: wait for ->release() on rsrc resurrect

 fs/io_uring.c | 26 ++++++++++++++++++--------
 1 file changed, 18 insertions(+), 8 deletions(-)

-- 
2.24.0

[PATCH 1/2] io_uring: wait potential ->release() on resurrect

[Pavel Begunkov]

There is a short window where percpu_refs are already turned zero, but
we try to do resurrect(). Play nicer and wait for ->release() to happen
in this case and proceed as everything is ok. One little downside is
that we can ignore signal_pending() on a rare occasion, but someone
else should check for it later if needed.

Cc: <stable@vger.kernel.org> # 5.5+
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
 fs/io_uring.c | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/fs/io_uring.c b/fs/io_uring.c
index f2fdebaf28fe..6ea4633e5ed5 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -1104,6 +1104,21 @@ static inline void io_set_resource_node(struct io_kiocb *req)
 	}
 }
 
+static bool io_refs_resurrect(struct percpu_ref *ref, struct completion *compl)
+{
+	if (!percpu_ref_tryget(ref)) {
+		/* already at zero, wait for ->release() */
+		if (!try_wait_for_completion(compl))
+			synchronize_rcu();
+		return false;
+	}
+
+	percpu_ref_resurrect(ref);
+	reinit_completion(compl);
+	percpu_ref_put(ref);
+	return true;
+}
+
 static bool io_match_task(struct io_kiocb *head,
 			  struct task_struct *task,
 			  struct files_struct *files)
@@ -10094,10 +10109,8 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,
 
 		mutex_lock(&ctx->uring_lock);
 
-		if (ret) {
-			percpu_ref_resurrect(&ctx->refs);
-			goto out_quiesce;
-		}
+		if (ret && io_refs_resurrect(&ctx->refs, &ctx->ref_comp))
+			return ret;
 	}
 
 	if (ctx->restricted) {
@@ -10189,7 +10202,6 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,
 	if (io_register_op_must_quiesce(opcode)) {
 		/* bring the ctx back to life */
 		percpu_ref_reinit(&ctx->refs);
-out_quiesce:
 		reinit_completion(&ctx->ref_comp);
 	}
 	return ret;
-- 
2.24.0

[PATCH 2/2] io_uring: wait for ->release() on rsrc resurrect

[Pavel Begunkov]

As with ctx refs, on resurrect wait for potentially concurrently running
->release().

Cc: <stable@vger.kernel.org> # 5.10+
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
 fs/io_uring.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/fs/io_uring.c b/fs/io_uring.c
index 6ea4633e5ed5..c08d32523f79 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -7368,13 +7368,11 @@ static int io_rsrc_ref_quiesce(struct fixed_rsrc_data *data,
 		flush_delayed_work(&ctx->rsrc_put_work);
 
 		ret = wait_for_completion_interruptible(&data->done);
-		if (!ret)
+		if (!ret || !io_refs_resurrect(&data->refs, &data->done))
 			break;
 
-		percpu_ref_resurrect(&data->refs);
 		io_sqe_rsrc_set_node(ctx, data, backup_node);
 		backup_node = NULL;
-		reinit_completion(&data->done);
 		mutex_unlock(&ctx->uring_lock);
 		ret = io_run_task_work_sig();
 		mutex_lock(&ctx->uring_lock);
-- 
2.24.0

Re: [PATCH 0/2] resurrect

[Pavel Begunkov]

On 20/02/2021 17:21, Pavel Begunkov wrote:
> Combined approach with tryget, fallbacking to trying completion and
> doing synchronize_rcu, where the last one is very unlikely to happen
> and limited by one such per register syscall.
> 
> I expect 1/2 to cleanly apply for stable, 2/2 needs to be backported
> by hand.

nevermind this, need to rebase and resend

> 
> Pavel Begunkov (2):
>   io_uring: wait potential ->release() on resurrect
>   io_uring: wait for ->release() on rsrc resurrect
> 
>  fs/io_uring.c | 26 ++++++++++++++++++--------
>  1 file changed, 18 insertions(+), 8 deletions(-)
> 

-- 
Pavel Begunkov