public inbox for [email protected]
 help / color / mirror / Atom feed
* [PATCH for-next 1/1] io_uring: optinise io_uring_task layout
@ 2022-06-20 14:27 Pavel Begunkov
  2022-06-20 15:03 ` Jens Axboe
  0 siblings, 1 reply; 2+ messages in thread
From: Pavel Begunkov @ 2022-06-20 14:27 UTC (permalink / raw)
  To: io-uring; +Cc: Jens Axboe, asml.silence

task_work bits of io_uring_task are split into two cache lines causing
extra cache bouncing, place them into a separate cache line. Also move
the most used submission path fields closer together, so there are hot.

Signed-off-by: Pavel Begunkov <[email protected]>
---
 io_uring/tctx.h | 34 ++++++++++++++++++----------------
 1 file changed, 18 insertions(+), 16 deletions(-)

diff --git a/io_uring/tctx.h b/io_uring/tctx.h
index dde82ce4d8e2..dead0ed00429 100644
--- a/io_uring/tctx.h
+++ b/io_uring/tctx.h
@@ -7,22 +7,24 @@
 
 struct io_uring_task {
 	/* submission side */
-	int			cached_refs;
-	struct xarray		xa;
-	struct wait_queue_head	wait;
-	const struct io_ring_ctx *last;
-	struct io_wq		*io_wq;
-	struct percpu_counter	inflight;
-	atomic_t		inflight_tracked;
-	atomic_t		in_idle;
-
-	spinlock_t		task_lock;
-	struct io_wq_work_list	task_list;
-	struct io_wq_work_list	prio_task_list;
-	struct callback_head	task_work;
-	bool			task_running;
-
-	struct file		*registered_rings[IO_RINGFD_REG_MAX];
+	int				cached_refs;
+	const struct io_ring_ctx 	*last;
+	struct io_wq			*io_wq;
+	struct file			*registered_rings[IO_RINGFD_REG_MAX];
+
+	struct xarray			xa;
+	struct wait_queue_head		wait;
+	atomic_t			in_idle;
+	atomic_t			inflight_tracked;
+	struct percpu_counter		inflight;
+
+	struct { /* task_work */
+		spinlock_t		task_lock;
+		bool			task_running;
+		struct io_wq_work_list	task_list;
+		struct io_wq_work_list	prio_task_list;
+		struct callback_head	task_work;
+	} ____cacheline_aligned_in_smp;
 };
 
 struct io_tctx_node {
-- 
2.36.1


^ permalink raw reply related	[flat|nested] 2+ messages in thread

* Re: [PATCH for-next 1/1] io_uring: optinise io_uring_task layout
  2022-06-20 14:27 [PATCH for-next 1/1] io_uring: optinise io_uring_task layout Pavel Begunkov
@ 2022-06-20 15:03 ` Jens Axboe
  0 siblings, 0 replies; 2+ messages in thread
From: Jens Axboe @ 2022-06-20 15:03 UTC (permalink / raw)
  To: Pavel Begunkov, io-uring

On 6/20/22 8:27 AM, Pavel Begunkov wrote:
> task_work bits of io_uring_task are split into two cache lines causing
> extra cache bouncing, place them into a separate cache line. Also move
> the most used submission path fields closer together, so there are hot.

Thanks, applied. The current layout is really stupid...

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2022-06-20 15:15 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-06-20 14:27 [PATCH for-next 1/1] io_uring: optinise io_uring_task layout Pavel Begunkov
2022-06-20 15:03 ` Jens Axboe

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox