* [PATCH for-next 01/10] io_uring: make reg buf init consistent
2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 02/10] io_uring: move defer_list to slow data Pavel Begunkov
` (9 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
The default (i.e. empty) state of register buffer is dummy_ubuf, so set
it to dummy on init instead of NULL.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/rsrc.c | 9 +++------
1 file changed, 3 insertions(+), 6 deletions(-)
diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index fef46972c327..fd1323482030 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -567,7 +567,7 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,
io_buffer_unmap(ctx, &imu);
break;
}
- ctx->user_bufs[i] = NULL;
+ ctx->user_bufs[i] = ctx->dummy_ubuf;
needs_switch = true;
}
@@ -1200,14 +1200,11 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov,
size_t size;
int ret, nr_pages, i;
- if (!iov->iov_base) {
- *pimu = ctx->dummy_ubuf;
+ *pimu = ctx->dummy_ubuf;
+ if (!iov->iov_base)
return 0;
- }
- *pimu = NULL;
ret = -ENOMEM;
-
pages = io_pin_pages((unsigned long) iov->iov_base, iov->iov_len,
&nr_pages);
if (IS_ERR(pages)) {
--
2.36.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH for-next 02/10] io_uring: move defer_list to slow data
2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 01/10] io_uring: make reg buf init consistent Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 03/10] io_uring: better caching for ctx timeout fields Pavel Begunkov
` (8 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
draining is slow path, move defer_list to the end where slow data lives
inside the context.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/io_uring_types.h | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h
index 147e1e597530..bff73107f0f3 100644
--- a/io_uring/io_uring_types.h
+++ b/io_uring/io_uring_types.h
@@ -160,7 +160,6 @@ struct io_ring_ctx {
struct io_uring_sqe *sq_sqes;
unsigned cached_sq_head;
unsigned sq_entries;
- struct list_head defer_list;
/*
* Fixed resources fast path, should be accessed only under
@@ -272,8 +271,12 @@ struct io_ring_ctx {
struct work_struct exit_work;
struct list_head tctx_list;
struct completion ref_comp;
+
+ /* io-wq management, e.g. thread count */
u32 iowq_limits[2];
bool iowq_limits_set;
+
+ struct list_head defer_list;
};
};
--
2.36.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH for-next 03/10] io_uring: better caching for ctx timeout fields
2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 01/10] io_uring: make reg buf init consistent Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 02/10] io_uring: move defer_list to slow data Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 04/10] io_uring: refactor ctx slow data placement Pavel Begunkov
` (7 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
Following timeout fields access patterns, move all of them into a
separate cache line inside ctx, so they don't intervene with normal
completion caching, especially since timeout removals and completion
are separated and the later is done via tw.
It also sheds some bytes from io_ring_ctx, 1216B -> 1152B
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/io_uring_types.h | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h
index bff73107f0f3..e050c3c4a7ac 100644
--- a/io_uring/io_uring_types.h
+++ b/io_uring/io_uring_types.h
@@ -179,8 +179,6 @@ struct io_ring_ctx {
struct xarray io_bl_xa;
struct list_head io_buffers_cache;
- struct list_head timeout_list;
- struct list_head ltimeout_list;
struct list_head cq_overflow_list;
struct list_head apoll_cache;
struct xarray personalities;
@@ -213,15 +211,11 @@ struct io_ring_ctx {
struct io_ev_fd __rcu *io_ev_fd;
struct wait_queue_head cq_wait;
unsigned cq_extra;
- atomic_t cq_timeouts;
- unsigned cq_last_tm_flush;
} ____cacheline_aligned_in_smp;
struct {
spinlock_t completion_lock;
- spinlock_t timeout_lock;
-
/*
* ->iopoll_list is protected by the ctx->uring_lock for
* io_uring instances that don't use IORING_SETUP_SQPOLL.
@@ -253,6 +247,15 @@ struct io_ring_ctx {
struct list_head io_buffers_pages;
};
+ /* timeouts */
+ struct {
+ spinlock_t timeout_lock;
+ atomic_t cq_timeouts;
+ struct list_head timeout_list;
+ struct list_head ltimeout_list;
+ unsigned cq_last_tm_flush;
+ } ____cacheline_aligned_in_smp;
+
/* Keep this last, we don't need it for the fast path */
struct {
#if defined(CONFIG_UNIX)
--
2.36.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH for-next 04/10] io_uring: refactor ctx slow data placement
2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
` (2 preceding siblings ...)
2022-06-15 16:33 ` [PATCH for-next 03/10] io_uring: better caching for ctx timeout fields Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 05/10] io_uring: move small helpers to headers Pavel Begunkov
` (6 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
Shove all slow path data at the end of ctx and get rid of extra
indention.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/io_uring_types.h | 81 +++++++++++++++++++--------------------
1 file changed, 39 insertions(+), 42 deletions(-)
diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h
index e050c3c4a7ac..94f76256721a 100644
--- a/io_uring/io_uring_types.h
+++ b/io_uring/io_uring_types.h
@@ -183,7 +183,6 @@ struct io_ring_ctx {
struct list_head apoll_cache;
struct xarray personalities;
u32 pers_next;
- unsigned sq_thread_idle;
} ____cacheline_aligned_in_smp;
/* IRQ completion list, under ->completion_lock */
@@ -230,23 +229,6 @@ struct io_ring_ctx {
struct list_head io_buffers_comp;
} ____cacheline_aligned_in_smp;
- struct io_restriction restrictions;
-
- /* slow path rsrc auxilary data, used by update/register */
- struct {
- struct io_rsrc_node *rsrc_backup_node;
- struct io_mapped_ubuf *dummy_ubuf;
- struct io_rsrc_data *file_data;
- struct io_rsrc_data *buf_data;
-
- struct delayed_work rsrc_put_work;
- struct llist_head rsrc_put_llist;
- struct list_head rsrc_ref_list;
- spinlock_t rsrc_ref_lock;
-
- struct list_head io_buffers_pages;
- };
-
/* timeouts */
struct {
spinlock_t timeout_lock;
@@ -257,30 +239,45 @@ struct io_ring_ctx {
} ____cacheline_aligned_in_smp;
/* Keep this last, we don't need it for the fast path */
- struct {
- #if defined(CONFIG_UNIX)
- struct socket *ring_sock;
- #endif
- /* hashed buffered write serialization */
- struct io_wq_hash *hash_map;
-
- /* Only used for accounting purposes */
- struct user_struct *user;
- struct mm_struct *mm_account;
-
- /* ctx exit and cancelation */
- struct llist_head fallback_llist;
- struct delayed_work fallback_work;
- struct work_struct exit_work;
- struct list_head tctx_list;
- struct completion ref_comp;
-
- /* io-wq management, e.g. thread count */
- u32 iowq_limits[2];
- bool iowq_limits_set;
-
- struct list_head defer_list;
- };
+
+ struct io_restriction restrictions;
+
+ /* slow path rsrc auxilary data, used by update/register */
+ struct io_rsrc_node *rsrc_backup_node;
+ struct io_mapped_ubuf *dummy_ubuf;
+ struct io_rsrc_data *file_data;
+ struct io_rsrc_data *buf_data;
+
+ struct delayed_work rsrc_put_work;
+ struct llist_head rsrc_put_llist;
+ struct list_head rsrc_ref_list;
+ spinlock_t rsrc_ref_lock;
+
+ struct list_head io_buffers_pages;
+
+ #if defined(CONFIG_UNIX)
+ struct socket *ring_sock;
+ #endif
+ /* hashed buffered write serialization */
+ struct io_wq_hash *hash_map;
+
+ /* Only used for accounting purposes */
+ struct user_struct *user;
+ struct mm_struct *mm_account;
+
+ /* ctx exit and cancelation */
+ struct llist_head fallback_llist;
+ struct delayed_work fallback_work;
+ struct work_struct exit_work;
+ struct list_head tctx_list;
+ struct completion ref_comp;
+
+ /* io-wq management, e.g. thread count */
+ u32 iowq_limits[2];
+ bool iowq_limits_set;
+
+ struct list_head defer_list;
+ unsigned sq_thread_idle;
};
enum {
--
2.36.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH for-next 05/10] io_uring: move small helpers to headers
2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
` (3 preceding siblings ...)
2022-06-15 16:33 ` [PATCH for-next 04/10] io_uring: refactor ctx slow data placement Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 06/10] io_uring: explain io_wq_work::cancel_seq placement Pavel Begunkov
` (5 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
There is a bunch of inline helpers that will be useful not only to the
core of io_uring, move them to headers.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/io_uring.c | 17 -----------------
io_uring/io_uring.h | 22 ++++++++++++++++++++++
2 files changed, 22 insertions(+), 17 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 78aee292d109..38b53011e0e9 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -615,14 +615,6 @@ struct sock *io_uring_get_socket(struct file *file)
}
EXPORT_SYMBOL(io_uring_get_socket);
-static inline void io_tw_lock(struct io_ring_ctx *ctx, bool *locked)
-{
- if (!*locked) {
- mutex_lock(&ctx->uring_lock);
- *locked = true;
- }
-}
-
static inline void io_submit_flush_completions(struct io_ring_ctx *ctx)
{
if (!wq_list_empty(&ctx->submit_state.compl_reqs))
@@ -878,15 +870,6 @@ static void io_prep_async_link(struct io_kiocb *req)
}
}
-static inline void io_req_add_compl_list(struct io_kiocb *req)
-{
- struct io_submit_state *state = &req->ctx->submit_state;
-
- if (!(req->flags & REQ_F_CQE_SKIP))
- state->flush_cqes = true;
- wq_list_add_tail(&req->comp_list, &state->compl_reqs);
-}
-
void io_queue_iowq(struct io_kiocb *req, bool *dont_use)
{
struct io_kiocb *link = io_prep_linked_timeout(req);
diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h
index ee718f7d54d3..e30e639c2822 100644
--- a/io_uring/io_uring.h
+++ b/io_uring/io_uring.h
@@ -217,6 +217,28 @@ static inline bool io_run_task_work(void)
return false;
}
+static inline void io_req_complete_state(struct io_kiocb *req)
+{
+ req->flags |= REQ_F_COMPLETE_INLINE;
+}
+
+static inline void io_tw_lock(struct io_ring_ctx *ctx, bool *locked)
+{
+ if (!*locked) {
+ mutex_lock(&ctx->uring_lock);
+ *locked = true;
+ }
+}
+
+static inline void io_req_add_compl_list(struct io_kiocb *req)
+{
+ struct io_submit_state *state = &req->ctx->submit_state;
+
+ if (!(req->flags & REQ_F_CQE_SKIP))
+ state->flush_cqes = true;
+ wq_list_add_tail(&req->comp_list, &state->compl_reqs);
+}
+
int io_run_task_work_sig(void);
void io_req_complete_failed(struct io_kiocb *req, s32 res);
void __io_req_complete(struct io_kiocb *req, unsigned issue_flags);
--
2.36.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH for-next 06/10] io_uring: explain io_wq_work::cancel_seq placement
2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
` (4 preceding siblings ...)
2022-06-15 16:33 ` [PATCH for-next 05/10] io_uring: move small helpers to headers Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 07/10] io_uring: inline ->registered_rings Pavel Begunkov
` (4 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
Add a comment on why we keep ->cancel_seq in struct io_wq_work instead
of struct io_kiocb despite it needed only by io_uring but not io-wq.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/io-wq.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h
index ba6eee76d028..3f54ee2a8eeb 100644
--- a/io_uring/io-wq.h
+++ b/io_uring/io-wq.h
@@ -155,6 +155,7 @@ struct io_wq_work_node *wq_stack_extract(struct io_wq_work_node *stack)
struct io_wq_work {
struct io_wq_work_node list;
unsigned flags;
+ /* place it here instead of io_kiocb as it fills padding and saves 4B */
int cancel_seq;
};
--
2.36.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH for-next 07/10] io_uring: inline ->registered_rings
2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
` (5 preceding siblings ...)
2022-06-15 16:33 ` [PATCH for-next 06/10] io_uring: explain io_wq_work::cancel_seq placement Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 08/10] io_uring: never defer-complete multi-apoll Pavel Begunkov
` (3 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
There can be only 16 registered rings, no need to allocate an array for
them separately but store it in tctx.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/io_uring.c | 1 -
io_uring/tctx.c | 9 ---------
io_uring/tctx.h | 3 ++-
3 files changed, 2 insertions(+), 11 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 38b53011e0e9..dec288b5f5cd 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2744,7 +2744,6 @@ void __io_uring_free(struct task_struct *tsk)
WARN_ON_ONCE(tctx->io_wq);
WARN_ON_ONCE(tctx->cached_refs);
- kfree(tctx->registered_rings);
percpu_counter_destroy(&tctx->inflight);
kfree(tctx);
tsk->io_uring = NULL;
diff --git a/io_uring/tctx.c b/io_uring/tctx.c
index f3262eef55d4..6adf659687f8 100644
--- a/io_uring/tctx.c
+++ b/io_uring/tctx.c
@@ -55,16 +55,8 @@ __cold int io_uring_alloc_task_context(struct task_struct *task,
if (unlikely(!tctx))
return -ENOMEM;
- tctx->registered_rings = kcalloc(IO_RINGFD_REG_MAX,
- sizeof(struct file *), GFP_KERNEL);
- if (unlikely(!tctx->registered_rings)) {
- kfree(tctx);
- return -ENOMEM;
- }
-
ret = percpu_counter_init(&tctx->inflight, 0, GFP_KERNEL);
if (unlikely(ret)) {
- kfree(tctx->registered_rings);
kfree(tctx);
return ret;
}
@@ -73,7 +65,6 @@ __cold int io_uring_alloc_task_context(struct task_struct *task,
if (IS_ERR(tctx->io_wq)) {
ret = PTR_ERR(tctx->io_wq);
percpu_counter_destroy(&tctx->inflight);
- kfree(tctx->registered_rings);
kfree(tctx);
return ret;
}
diff --git a/io_uring/tctx.h b/io_uring/tctx.h
index f4964e40d07e..7684713e950f 100644
--- a/io_uring/tctx.h
+++ b/io_uring/tctx.h
@@ -20,8 +20,9 @@ struct io_uring_task {
struct io_wq_work_list task_list;
struct io_wq_work_list prio_task_list;
struct callback_head task_work;
- struct file **registered_rings;
bool task_running;
+
+ struct file *registered_rings[IO_RINGFD_REG_MAX];
};
struct io_tctx_node {
--
2.36.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH for-next 08/10] io_uring: never defer-complete multi-apoll
2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
` (6 preceding siblings ...)
2022-06-15 16:33 ` [PATCH for-next 07/10] io_uring: inline ->registered_rings Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 09/10] io_uring: remove check_cq checking from hot paths Pavel Begunkov
` (2 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
Luckily, nnobody completes multi-apoll requests outside the polling
functions, but don't set IO_URING_F_COMPLETE_DEFER in any case as
there is nobody who is catching REQ_F_COMPLETE_INLINE, and so will leak
requests if used.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/io_uring.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index dec288b5f5cd..68ce8666bd32 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -2054,7 +2054,7 @@ int io_poll_issue(struct io_kiocb *req, bool *locked)
io_tw_lock(req->ctx, locked);
if (unlikely(req->task->flags & PF_EXITING))
return -EFAULT;
- return io_issue_sqe(req, IO_URING_F_NONBLOCK|IO_URING_F_COMPLETE_DEFER);
+ return io_issue_sqe(req, IO_URING_F_NONBLOCK);
}
struct io_wq_work *io_wq_free_work(struct io_wq_work *work)
--
2.36.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH for-next 09/10] io_uring: remove check_cq checking from hot paths
2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
` (7 preceding siblings ...)
2022-06-15 16:33 ` [PATCH for-next 08/10] io_uring: never defer-complete multi-apoll Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
2022-06-15 16:33 ` [PATCH for-next 10/10] io_uring: don't set REQ_F_COMPLETE_INLINE in tw Pavel Begunkov
2022-06-15 21:30 ` [PATCH for-next 00/10] 5.20 cleanups Jens Axboe
10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
All ctx->check_cq events are slow path, don't test every single flag one
by one in the hot path, but add a common guarding if.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/io_uring.c | 34 +++++++++++++++++++---------------
1 file changed, 19 insertions(+), 15 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 68ce8666bd32..f47de2906549 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1714,24 +1714,25 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min)
int ret = 0;
unsigned long check_cq;
+ check_cq = READ_ONCE(ctx->check_cq);
+ if (unlikely(check_cq)) {
+ if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT))
+ __io_cqring_overflow_flush(ctx, false);
+ /*
+ * Similarly do not spin if we have not informed the user of any
+ * dropped CQE.
+ */
+ if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT))
+ return -EBADR;
+ }
/*
* Don't enter poll loop if we already have events pending.
* If we do, we can potentially be spinning for commands that
* already triggered a CQE (eg in error).
*/
- check_cq = READ_ONCE(ctx->check_cq);
- if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT))
- __io_cqring_overflow_flush(ctx, false);
if (io_cqring_events(ctx))
return 0;
- /*
- * Similarly do not spin if we have not informed the user of any
- * dropped CQE.
- */
- if (unlikely(check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)))
- return -EBADR;
-
do {
/*
* If a submit got punted to a workqueue, we can have the
@@ -2657,12 +2658,15 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx,
ret = io_run_task_work_sig();
if (ret || io_should_wake(iowq))
return ret;
+
check_cq = READ_ONCE(ctx->check_cq);
- /* let the caller flush overflows, retry */
- if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT))
- return 1;
- if (unlikely(check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)))
- return -EBADR;
+ if (unlikely(check_cq)) {
+ /* let the caller flush overflows, retry */
+ if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT))
+ return 1;
+ if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT))
+ return -EBADR;
+ }
if (!schedule_hrtimeout(&timeout, HRTIMER_MODE_ABS))
return -ETIME;
return 1;
--
2.36.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH for-next 10/10] io_uring: don't set REQ_F_COMPLETE_INLINE in tw
2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
` (8 preceding siblings ...)
2022-06-15 16:33 ` [PATCH for-next 09/10] io_uring: remove check_cq checking from hot paths Pavel Begunkov
@ 2022-06-15 16:33 ` Pavel Begunkov
2022-06-15 21:30 ` [PATCH for-next 00/10] 5.20 cleanups Jens Axboe
10 siblings, 0 replies; 12+ messages in thread
From: Pavel Begunkov @ 2022-06-15 16:33 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
io_req_task_complete() enqueues requests for state completion itself, no
need for REQ_F_COMPLETE_INLINE, which is only serve the purpose of not
bloating the kernel.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/io_uring.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index f47de2906549..ce3302a62112 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -1769,7 +1769,6 @@ inline void io_req_task_complete(struct io_kiocb *req, bool *locked)
{
if (*locked) {
req->cqe.flags |= io_put_kbuf(req, 0);
- req->flags |= REQ_F_COMPLETE_INLINE;
io_req_add_compl_list(req);
} else {
req->cqe.flags |= io_put_kbuf(req, IO_URING_F_UNLOCKED);
--
2.36.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH for-next 00/10] 5.20 cleanups
2022-06-15 16:33 [PATCH for-next 00/10] 5.20 cleanups Pavel Begunkov
` (9 preceding siblings ...)
2022-06-15 16:33 ` [PATCH for-next 10/10] io_uring: don't set REQ_F_COMPLETE_INLINE in tw Pavel Begunkov
@ 2022-06-15 21:30 ` Jens Axboe
10 siblings, 0 replies; 12+ messages in thread
From: Jens Axboe @ 2022-06-15 21:30 UTC (permalink / raw)
To: asml.silence, io-uring
On Wed, 15 Jun 2022 17:33:46 +0100, Pavel Begunkov wrote:
> Simple cleanups split off from a larger series.
>
> Pavel Begunkov (10):
> io_uring: make reg buf init consistent
> io_uring: move defer_list to slow data
> io_uring: better caching for ctx timeout fields
> io_uring: refactor ctx slow data placement
> io_uring: move small helpers to headers
> io_uring: explain io_wq_work::cancel_seq placement
> io_uring: inline ->registered_rings
> io_uring: never defer-complete multi-apoll
> io_uring: remove check_cq checking from hot paths
> io_uring: don't set REQ_F_COMPLETE_INLINE in tw
>
> [...]
Applied, thanks!
[01/10] io_uring: make reg buf init consistent
commit: 8c81b9a8afeb9bf9a77ed7b8ae18fdcdd5e8738c
[02/10] io_uring: move defer_list to slow data
commit: 2946124bd54c6bde7d8223764f9e29ee5e9c2872
[03/10] io_uring: better caching for ctx timeout fields
commit: fee5d8c21d58b32c0e7f4dbddfa79ea2badfe080
[04/10] io_uring: refactor ctx slow data placement
commit: 5545259f66477791ead5305d080e7315ab93e1d2
[05/10] io_uring: move small helpers to headers
commit: 8f056215cea9a0b8a86d980c71da5587291f11c8
[06/10] io_uring: explain io_wq_work::cancel_seq placement
commit: 588383e3417729d24c804d43d9f08f3b1756c5cf
[07/10] io_uring: inline ->registered_rings
commit: 9d0222c4d9d1de014fea4ef151e6743b8eb30e8a
[08/10] io_uring: never defer-complete multi-apoll
commit: 5bbc2038f4d8f3de273c74779882ecb9a959a46d
[09/10] io_uring: remove check_cq checking from hot paths
commit: bc132bba5459cb501737c5793d1a273354dbf8db
[10/10] io_uring: don't set REQ_F_COMPLETE_INLINE in tw
commit: 01955c135f1753e60587fc28679fc1fab8ebc4d4
Best regards,
--
Jens Axboe
^ permalink raw reply [flat|nested] 12+ messages in thread