[PATCH 1/2] io_uring: optimise COMP_LOCK-less flush_completion

[Pavel Begunkov <asml.silence@gmail.com>]

After removing REQ_F_COMP_LOCKED io_submit_flush_completions() was left
deferring completions via task_work_add() in the hot path, that might
end up in a performance regression.

io_put_req() takes ->completion_lock only when at least one of
REQ_F_FAIL_LINK or REQ_F_LINK_TIMEOUT is set. There is also
REQ_F_WORK_INITIALIZED, freeing which while holding the lock have to be
avoided because it may deadlock with work.fs->lock. And if none of them
is set we can put under ->completion_lock and save an extra unlock/lock.

That actually works even better because it also works for most linked
requests unlike it was before.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
 fs/io_uring.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/fs/io_uring.c b/fs/io_uring.c
index cb2640f6fdb2..f61af4d487fd 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -1601,12 +1601,19 @@ static void io_submit_flush_completions(struct io_comp_state *cs)
 		req = list_first_entry(&cs->list, struct io_kiocb, compl.list);
 		list_del(&req->compl.list);
 		__io_cqring_fill_event(req, req->result, req->compl.cflags);
-		if (!(req->flags & REQ_F_LINK_HEAD)) {
-			io_put_req_deferred(req, 1);
-		} else {
+
+		/*
+		 * io_free_req() doesn't care about completion_lock unless one
+		 * of these flags is set. REQ_F_WORK_INITIALIZED is in the list
+		 * because of a potential deadlock with req->work.fs->lock
+		 */
+		if (req->flags & (REQ_F_FAIL_LINK|REQ_F_LINK_TIMEOUT
+				 |REQ_F_WORK_INITIALIZED)) {
 			spin_unlock_irq(&ctx->completion_lock);
 			io_put_req(req);
 			spin_lock_irq(&ctx->completion_lock);
+		} else {
+			io_put_req(req);
 		}
 	}
 	io_commit_cqring(ctx);
-- 
2.24.0