* [PATCH for-next 1/1] io_uring: optinise io_uring_task layout
@ 2022-06-20 14:27 Pavel Begunkov
2022-06-20 15:03 ` Jens Axboe
0 siblings, 1 reply; 2+ messages in thread
From: Pavel Begunkov @ 2022-06-20 14:27 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
task_work bits of io_uring_task are split into two cache lines causing
extra cache bouncing, place them into a separate cache line. Also move
the most used submission path fields closer together, so there are hot.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/tctx.h | 34 ++++++++++++++++++----------------
1 file changed, 18 insertions(+), 16 deletions(-)
diff --git a/io_uring/tctx.h b/io_uring/tctx.h
index dde82ce4d8e2..dead0ed00429 100644
--- a/io_uring/tctx.h
+++ b/io_uring/tctx.h
@@ -7,22 +7,24 @@
struct io_uring_task {
/* submission side */
- int cached_refs;
- struct xarray xa;
- struct wait_queue_head wait;
- const struct io_ring_ctx *last;
- struct io_wq *io_wq;
- struct percpu_counter inflight;
- atomic_t inflight_tracked;
- atomic_t in_idle;
-
- spinlock_t task_lock;
- struct io_wq_work_list task_list;
- struct io_wq_work_list prio_task_list;
- struct callback_head task_work;
- bool task_running;
-
- struct file *registered_rings[IO_RINGFD_REG_MAX];
+ int cached_refs;
+ const struct io_ring_ctx *last;
+ struct io_wq *io_wq;
+ struct file *registered_rings[IO_RINGFD_REG_MAX];
+
+ struct xarray xa;
+ struct wait_queue_head wait;
+ atomic_t in_idle;
+ atomic_t inflight_tracked;
+ struct percpu_counter inflight;
+
+ struct { /* task_work */
+ spinlock_t task_lock;
+ bool task_running;
+ struct io_wq_work_list task_list;
+ struct io_wq_work_list prio_task_list;
+ struct callback_head task_work;
+ } ____cacheline_aligned_in_smp;
};
struct io_tctx_node {
--
2.36.1
^ permalink raw reply related [flat|nested] 2+ messages in thread
* Re: [PATCH for-next 1/1] io_uring: optinise io_uring_task layout
2022-06-20 14:27 [PATCH for-next 1/1] io_uring: optinise io_uring_task layout Pavel Begunkov
@ 2022-06-20 15:03 ` Jens Axboe
0 siblings, 0 replies; 2+ messages in thread
From: Jens Axboe @ 2022-06-20 15:03 UTC (permalink / raw)
To: Pavel Begunkov, io-uring
On 6/20/22 8:27 AM, Pavel Begunkov wrote:
> task_work bits of io_uring_task are split into two cache lines causing
> extra cache bouncing, place them into a separate cache line. Also move
> the most used submission path fields closer together, so there are hot.
Thanks, applied. The current layout is really stupid...
--
Jens Axboe
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2022-06-20 15:15 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-06-20 14:27 [PATCH for-next 1/1] io_uring: optinise io_uring_task layout Pavel Begunkov
2022-06-20 15:03 ` Jens Axboe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox