public inbox for [email protected]
 help / color / mirror / Atom feed
* [PATCH for-next 00/10] 5.20 cleanups
@ 2022-06-15 16:33 Pavel Begunkov
  2022-06-15 16:33 ` [PATCH for-next 01/10] io_uring: make reg buf init consistent Pavel Begunkov
                   ` (10 more replies)
  0 siblings, 11 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
  To: io-uring; +Cc: Jens Axboe, asml.silence

Simple cleanups split off from a larger series.

Pavel Begunkov (10):
  io_uring: make reg buf init consistent
  io_uring: move defer_list to slow data
  io_uring: better caching for ctx timeout fields
  io_uring: refactor ctx slow data placement
  io_uring: move small helpers to headers
  io_uring: explain io_wq_work::cancel_seq placement
  io_uring: inline ->registered_rings
  io_uring: never defer-complete multi-apoll
  io_uring: remove check_cq checking from hot paths
  io_uring: don't set REQ_F_COMPLETE_INLINE in tw

 io_uring/io-wq.h          |  1 +
 io_uring/io_uring.c       | 55 +++++++++----------------
 io_uring/io_uring.h       | 22 ++++++++++
 io_uring/io_uring_types.h | 87 ++++++++++++++++++++-------------------
 io_uring/rsrc.c           |  9 ++--
 io_uring/tctx.c           |  9 ----
 io_uring/tctx.h           |  3 +-
 7 files changed, 93 insertions(+), 93 deletions(-)

-- 
2.36.1



^ permalink raw reply	[flat|nested] 12+ messages in thread

* [PATCH for-next 01/10] io_uring: make reg buf init consistent
  2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
  2022-06-15 16:33 ` [PATCH for-next 02/10] io_uring: move defer_list to slow data Pavel Begunkov
                   ` (9 subsequent siblings)
  10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
  To: io-uring; +Cc: Jens Axboe, asml.silence

The default (i.e. empty) state of register buffer is dummy_ubuf, so set
it to dummy on init instead of NULL.

Signed-off-by: Pavel Begunkov <[email protected]>
---
 io_uring/rsrc.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index fef46972c327..fd1323482030 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -567,7 +567,7 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,
 				io_buffer_unmap(ctx, &imu);
 				break;
 			}
-			ctx->user_bufs[i] = NULL;
+			ctx->user_bufs[i] = ctx->dummy_ubuf;
 			needs_switch = true;
 		}
 
@@ -1200,14 +1200,11 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov,
 	size_t size;
 	int ret, nr_pages, i;
 
-	if (!iov->iov_base) {
-		*pimu = ctx->dummy_ubuf;
+	*pimu = ctx->dummy_ubuf;
+	if (!iov->iov_base)
 		return 0;
-	}
 
-	*pimu = NULL;
 	ret = -ENOMEM;
-
 	pages = io_pin_pages((unsigned long) iov->iov_base, iov->iov_len,
 				&nr_pages);
 	if (IS_ERR(pages)) {
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH for-next 02/10] io_uring: move defer_list to slow data
  2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
  2022-06-15 16:33 ` [PATCH for-next 01/10] io_uring: make reg buf init consistent Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
  2022-06-15 16:33 ` [PATCH for-next 03/10] io_uring: better caching for ctx timeout fields Pavel Begunkov
                   ` (8 subsequent siblings)
  10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
  To: io-uring; +Cc: Jens Axboe, asml.silence

draining is slow path, move defer_list to the end where slow data lives
inside the context.

Signed-off-by: Pavel Begunkov <[email protected]>
---
 io_uring/io_uring_types.h | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h
index 147e1e597530..bff73107f0f3 100644
--- a/io_uring/io_uring_types.h
+++ b/io_uring/io_uring_types.h
@@ -160,7 +160,6 @@ struct io_ring_ctx {
 		struct io_uring_sqe	*sq_sqes;
 		unsigned		cached_sq_head;
 		unsigned		sq_entries;
-		struct list_head	defer_list;
 
 		/*
 		 * Fixed resources fast path, should be accessed only under
@@ -272,8 +271,12 @@ struct io_ring_ctx {
 		struct work_struct		exit_work;
 		struct list_head		tctx_list;
 		struct completion		ref_comp;
+
+		/* io-wq management, e.g. thread count */
 		u32				iowq_limits[2];
 		bool				iowq_limits_set;
+
+		struct list_head		defer_list;
 	};
 };
 
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH for-next 03/10] io_uring: better caching for ctx timeout fields
  2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
  2022-06-15 16:33 ` [PATCH for-next 01/10] io_uring: make reg buf init consistent Pavel Begunkov
  2022-06-15 16:33 ` [PATCH for-next 02/10] io_uring: move defer_list to slow data Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
  2022-06-15 16:33 ` [PATCH for-next 04/10] io_uring: refactor ctx slow data placement Pavel Begunkov
                   ` (7 subsequent siblings)
  10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
  To: io-uring; +Cc: Jens Axboe, asml.silence

Following timeout fields access patterns, move all of them into a
separate cache line inside ctx, so they don't intervene with normal
completion caching, especially since timeout removals and completion
are separated and the later is done via tw.

It also sheds some bytes from io_ring_ctx, 1216B -> 1152B

Signed-off-by: Pavel Begunkov <[email protected]>
---
 io_uring/io_uring_types.h | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h
index bff73107f0f3..e050c3c4a7ac 100644
--- a/io_uring/io_uring_types.h
+++ b/io_uring/io_uring_types.h
@@ -179,8 +179,6 @@ struct io_ring_ctx {
 		struct xarray		io_bl_xa;
 		struct list_head	io_buffers_cache;
 
-		struct list_head	timeout_list;
-		struct list_head	ltimeout_list;
 		struct list_head	cq_overflow_list;
 		struct list_head	apoll_cache;
 		struct xarray		personalities;
@@ -213,15 +211,11 @@ struct io_ring_ctx {
 		struct io_ev_fd	__rcu	*io_ev_fd;
 		struct wait_queue_head	cq_wait;
 		unsigned		cq_extra;
-		atomic_t		cq_timeouts;
-		unsigned		cq_last_tm_flush;
 	} ____cacheline_aligned_in_smp;
 
 	struct {
 		spinlock_t		completion_lock;
 
-		spinlock_t		timeout_lock;
-
 		/*
 		 * ->iopoll_list is protected by the ctx->uring_lock for
 		 * io_uring instances that don't use IORING_SETUP_SQPOLL.
@@ -253,6 +247,15 @@ struct io_ring_ctx {
 		struct list_head	io_buffers_pages;
 	};
 
+	/* timeouts */
+	struct {
+		spinlock_t		timeout_lock;
+		atomic_t		cq_timeouts;
+		struct list_head	timeout_list;
+		struct list_head	ltimeout_list;
+		unsigned		cq_last_tm_flush;
+	} ____cacheline_aligned_in_smp;
+
 	/* Keep this last, we don't need it for the fast path */
 	struct {
 		#if defined(CONFIG_UNIX)
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH for-next 04/10] io_uring: refactor ctx slow data placement
  2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
                   ` (2 preceding siblings ...)
  2022-06-15 16:33 ` [PATCH for-next 03/10] io_uring: better caching for ctx timeout fields Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
  2022-06-15 16:33 ` [PATCH for-next 05/10] io_uring: move small helpers to headers Pavel Begunkov
                   ` (6 subsequent siblings)
  10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
  To: io-uring; +Cc: Jens Axboe, asml.silence

Shove all slow path data at the end of ctx and get rid of extra
indention.

Signed-off-by: Pavel Begunkov <[email protected]>
---
 io_uring/io_uring_types.h | 81 +++++++++++++++++++--------------------
 1 file changed, 39 insertions(+), 42 deletions(-)

diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h
index e050c3c4a7ac..94f76256721a 100644
--- a/io_uring/io_uring_types.h
+++ b/io_uring/io_uring_types.h
@@ -183,7 +183,6 @@ struct io_ring_ctx {
 		struct list_head	apoll_cache;
 		struct xarray		personalities;
 		u32			pers_next;
-		unsigned		sq_thread_idle;
 	} ____cacheline_aligned_in_smp;
 
 	/* IRQ completion list, under ->completion_lock */
@@ -230,23 +229,6 @@ struct io_ring_ctx {
 		struct list_head	io_buffers_comp;
 	} ____cacheline_aligned_in_smp;
 
-	struct io_restriction		restrictions;
-
-	/* slow path rsrc auxilary data, used by update/register */
-	struct {
-		struct io_rsrc_node		*rsrc_backup_node;
-		struct io_mapped_ubuf		*dummy_ubuf;
-		struct io_rsrc_data		*file_data;
-		struct io_rsrc_data		*buf_data;
-
-		struct delayed_work		rsrc_put_work;
-		struct llist_head		rsrc_put_llist;
-		struct list_head		rsrc_ref_list;
-		spinlock_t			rsrc_ref_lock;
-
-		struct list_head	io_buffers_pages;
-	};
-
 	/* timeouts */
 	struct {
 		spinlock_t		timeout_lock;
@@ -257,30 +239,45 @@ struct io_ring_ctx {
 	} ____cacheline_aligned_in_smp;
 
 	/* Keep this last, we don't need it for the fast path */
-	struct {
-		#if defined(CONFIG_UNIX)
-			struct socket		*ring_sock;
-		#endif
-		/* hashed buffered write serialization */
-		struct io_wq_hash		*hash_map;
-
-		/* Only used for accounting purposes */
-		struct user_struct		*user;
-		struct mm_struct		*mm_account;
-
-		/* ctx exit and cancelation */
-		struct llist_head		fallback_llist;
-		struct delayed_work		fallback_work;
-		struct work_struct		exit_work;
-		struct list_head		tctx_list;
-		struct completion		ref_comp;
-
-		/* io-wq management, e.g. thread count */
-		u32				iowq_limits[2];
-		bool				iowq_limits_set;
-
-		struct list_head		defer_list;
-	};
+
+	struct io_restriction		restrictions;
+
+	/* slow path rsrc auxilary data, used by update/register */
+	struct io_rsrc_node		*rsrc_backup_node;
+	struct io_mapped_ubuf		*dummy_ubuf;
+	struct io_rsrc_data		*file_data;
+	struct io_rsrc_data		*buf_data;
+
+	struct delayed_work		rsrc_put_work;
+	struct llist_head		rsrc_put_llist;
+	struct list_head		rsrc_ref_list;
+	spinlock_t			rsrc_ref_lock;
+
+	struct list_head		io_buffers_pages;
+
+	#if defined(CONFIG_UNIX)
+		struct socket		*ring_sock;
+	#endif
+	/* hashed buffered write serialization */
+	struct io_wq_hash		*hash_map;
+
+	/* Only used for accounting purposes */
+	struct user_struct		*user;
+	struct mm_struct		*mm_account;
+
+	/* ctx exit and cancelation */
+	struct llist_head		fallback_llist;
+	struct delayed_work		fallback_work;
+	struct work_struct		exit_work;
+	struct list_head		tctx_list;
+	struct completion		ref_comp;
+
+	/* io-wq management, e.g. thread count */
+	u32				iowq_limits[2];
+	bool				iowq_limits_set;
+
+	struct list_head		defer_list;
+	unsigned			sq_thread_idle;
 };
 
 enum {
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH for-next 05/10] io_uring: move small helpers to headers
  2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
                   ` (3 preceding siblings ...)
  2022-06-15 16:33 ` [PATCH for-next 04/10] io_uring: refactor ctx slow data placement Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
  2022-06-15 16:33 ` [PATCH for-next 06/10] io_uring: explain io_wq_work::cancel_seq placement Pavel Begunkov
                   ` (5 subsequent siblings)
  10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
  To: io-uring; +Cc: Jens Axboe, asml.silence

There is a bunch of inline helpers that will be useful not only to the
core of io_uring, move them to headers.

Signed-off-by: Pavel Begunkov <[email protected]>
---
 io_uring/io_uring.c | 17 -----------------
 io_uring/io_uring.h | 22 ++++++++++++++++++++++
 2 files changed, 22 insertions(+), 17 deletions(-)

diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 78aee292d109..38b53011e0e9 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -615,14 +615,6 @@ struct sock *io_uring_get_socket(struct file *file)
 }
 EXPORT_SYMBOL(io_uring_get_socket);
 
-static inline void io_tw_lock(struct io_ring_ctx *ctx, bool *locked)
-{
-	if (!*locked) {
-		mutex_lock(&ctx->uring_lock);
-		*locked = true;
-	}
-}
-
 static inline void io_submit_flush_completions(struct io_ring_ctx *ctx)
 {
 	if (!wq_list_empty(&ctx->submit_state.compl_reqs))
@@ -878,15 +870,6 @@ static void io_prep_async_link(struct io_kiocb *req)
 	}
 }
 
-static inline void io_req_add_compl_list(struct io_kiocb *req)
-{
-	struct io_submit_state *state = &req->ctx->submit_state;
-
-	if (!(req->flags & REQ_F_CQE_SKIP))
-		state->flush_cqes = true;
-	wq_list_add_tail(&req->comp_list, &state->compl_reqs);
-}
-
 void io_queue_iowq(struct io_kiocb *req, bool *dont_use)
 {
 	struct io_kiocb *link = io_prep_linked_timeout(req);
diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h
index ee718f7d54d3..e30e639c2822 100644
--- a/io_uring/io_uring.h
+++ b/io_uring/io_uring.h
@@ -217,6 +217,28 @@ static inline bool io_run_task_work(void)
 	return false;
 }
 
+static inline void io_req_complete_state(struct io_kiocb *req)
+{
+	req->flags |= REQ_F_COMPLETE_INLINE;
+}
+
+static inline void io_tw_lock(struct io_ring_ctx *ctx, bool *locked)
+{
+	if (!*locked) {
+		mutex_lock(&ctx->uring_lock);
+		*locked = true;
+	}
+}
+
+static inline void io_req_add_compl_list(struct io_kiocb *req)
+{
+	struct io_submit_state *state = &req->ctx->submit_state;
+
+	if (!(req->flags & REQ_F_CQE_SKIP))
+		state->flush_cqes = true;
+	wq_list_add_tail(&req->comp_list, &state->compl_reqs);
+}
+
 int io_run_task_work_sig(void);
 void io_req_complete_failed(struct io_kiocb *req, s32 res);
 void __io_req_complete(struct io_kiocb *req, unsigned issue_flags);
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH for-next 06/10] io_uring: explain io_wq_work::cancel_seq placement
  2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
                   ` (4 preceding siblings ...)
  2022-06-15 16:33 ` [PATCH for-next 05/10] io_uring: move small helpers to headers Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
  2022-06-15 16:33 ` [PATCH for-next 07/10] io_uring: inline ->registered_rings Pavel Begunkov
                   ` (4 subsequent siblings)
  10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
  To: io-uring; +Cc: Jens Axboe, asml.silence

Add a comment on why we keep ->cancel_seq in struct io_wq_work instead
of struct io_kiocb despite it needed only by io_uring but not io-wq.

Signed-off-by: Pavel Begunkov <[email protected]>
---
 io_uring/io-wq.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h
index ba6eee76d028..3f54ee2a8eeb 100644
--- a/io_uring/io-wq.h
+++ b/io_uring/io-wq.h
@@ -155,6 +155,7 @@ struct io_wq_work_node *wq_stack_extract(struct io_wq_work_node *stack)
 struct io_wq_work {
 	struct io_wq_work_node list;
 	unsigned flags;
+	/* place it here instead of io_kiocb as it fills padding and saves 4B */
 	int cancel_seq;
 };
 
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH for-next 07/10] io_uring: inline ->registered_rings
  2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
                   ` (5 preceding siblings ...)
  2022-06-15 16:33 ` [PATCH for-next 06/10] io_uring: explain io_wq_work::cancel_seq placement Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
  2022-06-15 16:33 ` [PATCH for-next 08/10] io_uring: never defer-complete multi-apoll Pavel Begunkov
                   ` (3 subsequent siblings)
  10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
  To: io-uring; +Cc: Jens Axboe, asml.silence

There can be only 16 registered rings, no need to allocate an array for
them separately but store it in tctx.

Signed-off-by: Pavel Begunkov <[email protected]>
---
 io_uring/io_uring.c | 1 -
 io_uring/tctx.c     | 9 ---------
 io_uring/tctx.h     | 3 ++-
 3 files changed, 2 insertions(+), 11 deletions(-)

diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 38b53011e0e9..dec288b5f5cd 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2744,7 +2744,6 @@ void __io_uring_free(struct task_struct *tsk)
 	WARN_ON_ONCE(tctx->io_wq);
 	WARN_ON_ONCE(tctx->cached_refs);
 
-	kfree(tctx->registered_rings);
 	percpu_counter_destroy(&tctx->inflight);
 	kfree(tctx);
 	tsk->io_uring = NULL;
diff --git a/io_uring/tctx.c b/io_uring/tctx.c
index f3262eef55d4..6adf659687f8 100644
--- a/io_uring/tctx.c
+++ b/io_uring/tctx.c
@@ -55,16 +55,8 @@ __cold int io_uring_alloc_task_context(struct task_struct *task,
 	if (unlikely(!tctx))
 		return -ENOMEM;
 
-	tctx->registered_rings = kcalloc(IO_RINGFD_REG_MAX,
-					 sizeof(struct file *), GFP_KERNEL);
-	if (unlikely(!tctx->registered_rings)) {
-		kfree(tctx);
-		return -ENOMEM;
-	}
-
 	ret = percpu_counter_init(&tctx->inflight, 0, GFP_KERNEL);
 	if (unlikely(ret)) {
-		kfree(tctx->registered_rings);
 		kfree(tctx);
 		return ret;
 	}
@@ -73,7 +65,6 @@ __cold int io_uring_alloc_task_context(struct task_struct *task,
 	if (IS_ERR(tctx->io_wq)) {
 		ret = PTR_ERR(tctx->io_wq);
 		percpu_counter_destroy(&tctx->inflight);
-		kfree(tctx->registered_rings);
 		kfree(tctx);
 		return ret;
 	}
diff --git a/io_uring/tctx.h b/io_uring/tctx.h
index f4964e40d07e..7684713e950f 100644
--- a/io_uring/tctx.h
+++ b/io_uring/tctx.h
@@ -20,8 +20,9 @@ struct io_uring_task {
 	struct io_wq_work_list	task_list;
 	struct io_wq_work_list	prio_task_list;
 	struct callback_head	task_work;
-	struct file		**registered_rings;
 	bool			task_running;
+
+	struct file		*registered_rings[IO_RINGFD_REG_MAX];
 };
 
 struct io_tctx_node {
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH for-next 08/10] io_uring: never defer-complete multi-apoll
  2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
                   ` (6 preceding siblings ...)
  2022-06-15 16:33 ` [PATCH for-next 07/10] io_uring: inline ->registered_rings Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
  2022-06-15 16:33 ` [PATCH for-next 09/10] io_uring: remove check_cq checking from hot paths Pavel Begunkov
                   ` (2 subsequent siblings)
  10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
  To: io-uring; +Cc: Jens Axboe, asml.silence

Luckily, nnobody completes multi-apoll requests outside the polling
functions, but don't set IO_URING_F_COMPLETE_DEFER in any case as
there is nobody who is catching REQ_F_COMPLETE_INLINE, and so will leak
requests if used.

Signed-off-by: Pavel Begunkov <[email protected]>
---
 io_uring/io_uring.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index dec288b5f5cd..68ce8666bd32 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2054,7 +2054,7 @@ int io_poll_issue(struct io_kiocb *req, bool *locked)
 	io_tw_lock(req->ctx, locked);
 	if (unlikely(req->task->flags & PF_EXITING))
 		return -EFAULT;
-	return io_issue_sqe(req, IO_URING_F_NONBLOCK|IO_URING_F_COMPLETE_DEFER);
+	return io_issue_sqe(req, IO_URING_F_NONBLOCK);
 }
 
 struct io_wq_work *io_wq_free_work(struct io_wq_work *work)
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH for-next 09/10] io_uring: remove check_cq checking from hot paths
  2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
                   ` (7 preceding siblings ...)
  2022-06-15 16:33 ` [PATCH for-next 08/10] io_uring: never defer-complete multi-apoll Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
  2022-06-15 16:33 ` [PATCH for-next 10/10] io_uring: don't set REQ_F_COMPLETE_INLINE in tw Pavel Begunkov
  2022-06-15 21:30 ` [PATCH for-next 00/10] 5.20 cleanups Jens Axboe
  10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
  To: io-uring; +Cc: Jens Axboe, asml.silence

All ctx->check_cq events are slow path, don't test every single flag one
by one in the hot path, but add a common guarding if.

Signed-off-by: Pavel Begunkov <[email protected]>
---
 io_uring/io_uring.c | 34 +++++++++++++++++++---------------
 1 file changed, 19 insertions(+), 15 deletions(-)

diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 68ce8666bd32..f47de2906549 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1714,24 +1714,25 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min)
 	int ret = 0;
 	unsigned long check_cq;
 
+	check_cq = READ_ONCE(ctx->check_cq);
+	if (unlikely(check_cq)) {
+		if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT))
+			__io_cqring_overflow_flush(ctx, false);
+		/*
+		 * Similarly do not spin if we have not informed the user of any
+		 * dropped CQE.
+		 */
+		if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT))
+			return -EBADR;
+	}
 	/*
 	 * Don't enter poll loop if we already have events pending.
 	 * If we do, we can potentially be spinning for commands that
 	 * already triggered a CQE (eg in error).
 	 */
-	check_cq = READ_ONCE(ctx->check_cq);
-	if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT))
-		__io_cqring_overflow_flush(ctx, false);
 	if (io_cqring_events(ctx))
 		return 0;
 
-	/*
-	 * Similarly do not spin if we have not informed the user of any
-	 * dropped CQE.
-	 */
-	if (unlikely(check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)))
-		return -EBADR;
-
 	do {
 		/*
 		 * If a submit got punted to a workqueue, we can have the
@@ -2657,12 +2658,15 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx,
 	ret = io_run_task_work_sig();
 	if (ret || io_should_wake(iowq))
 		return ret;
+
 	check_cq = READ_ONCE(ctx->check_cq);
-	/* let the caller flush overflows, retry */
-	if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT))
-		return 1;
-	if (unlikely(check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)))
-		return -EBADR;
+	if (unlikely(check_cq)) {
+		/* let the caller flush overflows, retry */
+		if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT))
+			return 1;
+		if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT))
+			return -EBADR;
+	}
 	if (!schedule_hrtimeout(&timeout, HRTIMER_MODE_ABS))
 		return -ETIME;
 	return 1;
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* [PATCH for-next 10/10] io_uring: don't set REQ_F_COMPLETE_INLINE in tw
  2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
                   ` (8 preceding siblings ...)
  2022-06-15 16:33 ` [PATCH for-next 09/10] io_uring: remove check_cq checking from hot paths Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
  2022-06-15 21:30 ` [PATCH for-next 00/10] 5.20 cleanups Jens Axboe
  10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
  To: io-uring; +Cc: Jens Axboe, asml.silence

io_req_task_complete() enqueues requests for state completion itself, no
need for REQ_F_COMPLETE_INLINE, which is only serve the purpose of not
bloating the kernel.

Signed-off-by: Pavel Begunkov <[email protected]>
---
 io_uring/io_uring.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index f47de2906549..ce3302a62112 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1769,7 +1769,6 @@ inline void io_req_task_complete(struct io_kiocb *req, bool *locked)
 {
 	if (*locked) {
 		req->cqe.flags |= io_put_kbuf(req, 0);
-		req->flags |= REQ_F_COMPLETE_INLINE;
 		io_req_add_compl_list(req);
 	} else {
 		req->cqe.flags |= io_put_kbuf(req, IO_URING_F_UNLOCKED);
-- 
2.36.1



^ permalink raw reply related	[flat|nested] 12+ messages in thread

* Re: [PATCH for-next 00/10] 5.20 cleanups
  2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
                   ` (9 preceding siblings ...)
  2022-06-15 16:33 ` [PATCH for-next 10/10] io_uring: don't set REQ_F_COMPLETE_INLINE in tw Pavel Begunkov
@ 2022-06-15 21:30 ` Jens Axboe
  10 siblings, 0 replies; 12+ messages in thread
From: Jens Axboe @ 2022-06-15 21:30 UTC (permalink / raw)
  To: asml.silence, io-uring

On Wed, 15 Jun 2022 17:33:46 +0100, Pavel Begunkov wrote:
> Simple cleanups split off from a larger series.
> 
> Pavel Begunkov (10):
>   io_uring: make reg buf init consistent
>   io_uring: move defer_list to slow data
>   io_uring: better caching for ctx timeout fields
>   io_uring: refactor ctx slow data placement
>   io_uring: move small helpers to headers
>   io_uring: explain io_wq_work::cancel_seq placement
>   io_uring: inline ->registered_rings
>   io_uring: never defer-complete multi-apoll
>   io_uring: remove check_cq checking from hot paths
>   io_uring: don't set REQ_F_COMPLETE_INLINE in tw
> 
> [...]

Applied, thanks!

[01/10] io_uring: make reg buf init consistent
        commit: 8c81b9a8afeb9bf9a77ed7b8ae18fdcdd5e8738c
[02/10] io_uring: move defer_list to slow data
        commit: 2946124bd54c6bde7d8223764f9e29ee5e9c2872
[03/10] io_uring: better caching for ctx timeout fields
        commit: fee5d8c21d58b32c0e7f4dbddfa79ea2badfe080
[04/10] io_uring: refactor ctx slow data placement
        commit: 5545259f66477791ead5305d080e7315ab93e1d2
[05/10] io_uring: move small helpers to headers
        commit: 8f056215cea9a0b8a86d980c71da5587291f11c8
[06/10] io_uring: explain io_wq_work::cancel_seq placement
        commit: 588383e3417729d24c804d43d9f08f3b1756c5cf
[07/10] io_uring: inline ->registered_rings
        commit: 9d0222c4d9d1de014fea4ef151e6743b8eb30e8a
[08/10] io_uring: never defer-complete multi-apoll
        commit: 5bbc2038f4d8f3de273c74779882ecb9a959a46d
[09/10] io_uring: remove check_cq checking from hot paths
        commit: bc132bba5459cb501737c5793d1a273354dbf8db
[10/10] io_uring: don't set REQ_F_COMPLETE_INLINE in tw
        commit: 01955c135f1753e60587fc28679fc1fab8ebc4d4

Best regards,
-- 
Jens Axboe




^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2022-06-15 21:30 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 01/10] io_uring: make reg buf init consistent Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 02/10] io_uring: move defer_list to slow data Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 03/10] io_uring: better caching for ctx timeout fields Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 04/10] io_uring: refactor ctx slow data placement Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 05/10] io_uring: move small helpers to headers Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 06/10] io_uring: explain io_wq_work::cancel_seq placement Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 07/10] io_uring: inline ->registered_rings Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 08/10] io_uring: never defer-complete multi-apoll Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 09/10] io_uring: remove check_cq checking from hot paths Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 10/10] io_uring: don't set REQ_F_COMPLETE_INLINE in tw Pavel Begunkov
2022-06-15 21:30 ` [PATCH for-next 00/10] 5.20 cleanups Jens Axboe

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox