From: Pavel Begunkov <[email protected]>
To: [email protected], [email protected],
[email protected]
Cc: "David S . Miller" <[email protected]>,
Jakub Kicinski <[email protected]>,
Jonathan Lemon <[email protected]>,
Willem de Bruijn <[email protected]>,
Jens Axboe <[email protected]>, David Ahern <[email protected]>,
[email protected], Pavel Begunkov <[email protected]>
Subject: [PATCH net-next v3 15/25] io_uring: complete notifiers in tw
Date: Tue, 5 Jul 2022 16:01:15 +0100 [thread overview]
Message-ID: <591b24351034d95bc4f39a3d1cbbb7132109218d.1656318994.git.asml.silence@gmail.com> (raw)
In-Reply-To: <[email protected]>
We need a task context to post CQEs but using wq is too expensive.
Try to complete notifiers using task_work and fall back to wq if fails.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/notif.c | 22 +++++++++++++++++++---
io_uring/notif.h | 3 +++
2 files changed, 22 insertions(+), 3 deletions(-)
diff --git a/io_uring/notif.c b/io_uring/notif.c
index ffbd5ce03c36..f795e820de56 100644
--- a/io_uring/notif.c
+++ b/io_uring/notif.c
@@ -13,6 +13,11 @@ static void __io_notif_complete_tw(struct callback_head *cb)
struct io_notif *notif = container_of(cb, struct io_notif, task_work);
struct io_ring_ctx *ctx = notif->ctx;
+ if (likely(notif->task)) {
+ io_put_task(notif->task, 1);
+ notif->task = NULL;
+ }
+
io_cq_lock(ctx);
io_fill_cqe_aux(ctx, notif->tag, 0, notif->seq);
@@ -43,6 +48,14 @@ static void io_uring_tx_zerocopy_callback(struct sk_buff *skb,
if (!refcount_dec_and_test(&uarg->refcnt))
return;
+
+ if (likely(notif->task)) {
+ init_task_work(¬if->task_work, __io_notif_complete_tw);
+ if (likely(!task_work_add(notif->task, ¬if->task_work,
+ TWA_SIGNAL)))
+ return;
+ }
+
INIT_WORK(¬if->commit_work, io_notif_complete_wq);
queue_work(system_unbound_wq, ¬if->commit_work);
}
@@ -134,12 +147,15 @@ __cold int io_notif_unregister(struct io_ring_ctx *ctx)
for (i = 0; i < ctx->nr_notif_slots; i++) {
struct io_notif_slot *slot = &ctx->notif_slots[i];
- if (slot->notif)
- io_notif_slot_flush(slot);
+ if (!slot->notif)
+ continue;
+ if (WARN_ON_ONCE(slot->notif->task))
+ slot->notif->task = NULL;
+ io_notif_slot_flush(slot);
}
kvfree(ctx->notif_slots);
ctx->notif_slots = NULL;
ctx->nr_notif_slots = 0;
return 0;
-}
\ No newline at end of file
+}
diff --git a/io_uring/notif.h b/io_uring/notif.h
index b23c9c0515bb..23ca7620fff9 100644
--- a/io_uring/notif.h
+++ b/io_uring/notif.h
@@ -11,6 +11,9 @@ struct io_notif {
struct ubuf_info uarg;
struct io_ring_ctx *ctx;
+ /* complete via tw if ->task is non-NULL, fallback to wq otherwise */
+ struct task_struct *task;
+
/* cqe->user_data, io_notif_slot::tag if not overridden */
u64 tag;
/* see struct io_notif_slot::seq */
--
2.36.1
next prev parent reply other threads:[~2022-07-05 15:03 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-07-05 15:01 [PATCH net-next v3 00/25] io_uring zerocopy send Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 01/25] ipv4: avoid partial copy for zc Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 02/25] ipv6: " Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 03/25] skbuff: add SKBFL_DONT_ORPHAN flag Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 04/25] skbuff: carry external ubuf_info in msghdr Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 05/25] net: bvec specific path in zerocopy_sg_from_iter Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 06/25] net: optimise bvec-based zc page referencing Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 07/25] net: don't track pfmemalloc for managed frags Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 08/25] skbuff: don't mix ubuf_info of different types Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 09/25] ipv4/udp: support zc with managed data Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 10/25] ipv6/udp: " Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 11/25] tcp: " Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 12/25] io_uring: add zc notification infrastructure Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 13/25] io_uring: export task put Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 14/25] io_uring: cache struct io_notif Pavel Begunkov
2022-07-05 15:01 ` Pavel Begunkov [this message]
2022-07-05 15:01 ` [PATCH net-next v3 16/25] io_uring: add notification slot registration Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 17/25] io_uring: wire send zc request type Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 18/25] io_uring: account locked pages for non-fixed zc Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 19/25] io_uring: allow to pass addr into sendzc Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 20/25] io_uring: add rsrc referencing for notifiers Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 21/25] io_uring: sendzc with fixed buffers Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 22/25] io_uring: flush notifiers after sendzc Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 23/25] io_uring: rename IORING_OP_FILES_UPDATE Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 24/25] io_uring: add zc notification flush requests Pavel Begunkov
2022-07-05 15:01 ` [PATCH net-next v3 25/25] selftests/io_uring: test zerocopy send Pavel Begunkov
2022-07-05 15:04 ` [PATCH net-next v3 00/25] io_uring " Pavel Begunkov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=591b24351034d95bc4f39a3d1cbbb7132109218d.1656318994.git.asml.silence@gmail.com \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox