From: Dylan Yudaken <[email protected]>
To: Jens Axboe <[email protected]>, Pavel Begunkov <[email protected]>
Cc: <[email protected]>, <[email protected]>,
Dylan Yudaken <[email protected]>
Subject: [PATCH for-next 00/12] io_uring: retarget rsrc nodes periodically
Date: Mon, 31 Oct 2022 06:41:14 -0700 [thread overview]
Message-ID: <[email protected]> (raw)
There is a problem with long running io_uring requests and rsrc node
cleanup, where a single long running request can block cleanup of all
subsequent nodes. For example a network server may have both long running
accepts and also use registered files for connections. When sockets are
closed and returned (either through close_direct, or through
register_files_update) the underlying file will not be freed until that
original accept completes. For the case of multishot this may be the
lifetime of the application, which will cause file numbers to grow
unbounded - resulting in either OOMs or ENFILE errors.
To fix this I propose retargeting the rsrc nodes from ongoing requests to
the current main request of the io_uring. This only needs to happen for
long running requests types, and specifically those that happen as a
result of some external event. For this reason I exclude write/send style
ops for the time being as even though these can cause this issue in
reality it would be unexpected to have a write block for hours. This
support can obviously be added later if needed.
In order to retarget nodes all the outstanding requests (in both poll
tables and io-wq) need to be iterated and the request needs to be checked
to make sure the retargeting is valid. For example for FIXED_FILE requests
this involves ensuring the file is still referenced in the current node.
This O(N) operation seems to take ~1ms locally for 30k outstanding
requests. Note it locks the io_uring while it happens and so no other work
can occur. In order to amortize this cost slightly, I propose running this
operation at most every 60 seconds. It is hard coded currently, but would
be happy to take suggestions if this should be customizable (and how to do
such a thing).
Without customizable retargeting period, it's a bit difficult to submit
tests for this. I have a test but it obviously takes a many minutes to run
which is not going to be acceptable for liburing.
Patches 1-5 are the basic io_uring infrastructure
Patch 6 is a helper function used in the per op customisations
Patch 7 splits out the zerocopy specific field in io_sr_msg
Patches 8-12 are opcode implementations for retargeting
Dylan Yudaken (12):
io_uring: infrastructure for retargeting rsrc nodes
io_uring: io-wq helper to iterate all work
io_uring: support retargeting rsrc on requests in the io-wq
io_uring: reschedule retargeting at shutdown of ring
io_uring: add tracing for io_uring_rsrc_retarget
io_uring: add fixed file peeking function
io_uring: split send_zc specific struct out of io_sr_msg
io_uring: recv/recvmsg retarget_rsrc support
io_uring: accept retarget_rsrc support
io_uring: read retarget_rsrc support
io_uring: read_fixed retarget_rsrc support
io_uring: poll_add retarget_rsrc support
include/linux/io_uring_types.h | 2 +
include/trace/events/io_uring.h | 30 +++++++
io_uring/io-wq.c | 49 +++++++++++
io_uring/io-wq.h | 3 +
io_uring/io_uring.c | 28 ++++--
io_uring/io_uring.h | 1 +
io_uring/net.c | 114 ++++++++++++++++--------
io_uring/net.h | 2 +
io_uring/opdef.c | 7 ++
io_uring/opdef.h | 1 +
io_uring/poll.c | 12 +++
io_uring/poll.h | 2 +
io_uring/rsrc.c | 148 ++++++++++++++++++++++++++++++++
io_uring/rsrc.h | 2 +
io_uring/rw.c | 29 +++++++
io_uring/rw.h | 2 +
16 files changed, 390 insertions(+), 42 deletions(-)
base-commit: 30209debe98b6f66b13591e59e5272cb65b3945e
--
2.30.2
next reply other threads:[~2022-10-31 13:41 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-10-31 13:41 Dylan Yudaken [this message]
2022-10-31 13:41 ` [PATCH for-next 01/12] io_uring: infrastructure for retargeting rsrc nodes Dylan Yudaken
2022-10-31 16:02 ` Jens Axboe
2022-10-31 16:45 ` Dylan Yudaken
2022-11-02 11:20 ` Pavel Begunkov
2022-10-31 13:41 ` [PATCH for-next 02/12] io_uring: io-wq helper to iterate all work Dylan Yudaken
2022-10-31 13:41 ` [PATCH for-next 03/12] io_uring: support retargeting rsrc on requests in the io-wq Dylan Yudaken
2022-10-31 18:19 ` Jens Axboe
2022-10-31 13:41 ` [PATCH for-next 04/12] io_uring: reschedule retargeting at shutdown of ring Dylan Yudaken
2022-10-31 16:02 ` Jens Axboe
2022-10-31 16:44 ` Dylan Yudaken
2022-10-31 19:13 ` Jens Axboe
2022-11-01 12:09 ` Dylan Yudaken
2022-10-31 13:41 ` [PATCH for-next 05/12] io_uring: add tracing for io_uring_rsrc_retarget Dylan Yudaken
2022-10-31 13:41 ` [PATCH for-next 06/12] io_uring: add fixed file peeking function Dylan Yudaken
2022-10-31 16:04 ` Jens Axboe
2022-10-31 16:47 ` Dylan Yudaken
2022-11-02 11:23 ` Pavel Begunkov
2022-10-31 13:41 ` [PATCH for-next 07/12] io_uring: split send_zc specific struct out of io_sr_msg Dylan Yudaken
2022-11-02 11:32 ` Pavel Begunkov
2022-11-02 13:45 ` Jens Axboe
2022-11-02 14:09 ` Pavel Begunkov
2022-11-02 14:12 ` Jens Axboe
2022-10-31 13:41 ` [PATCH for-next 08/12] io_uring: recv/recvmsg retarget_rsrc support Dylan Yudaken
2022-10-31 13:41 ` [PATCH for-next 09/12] io_uring: accept " Dylan Yudaken
2022-10-31 13:41 ` [PATCH for-next 10/12] io_uring: read " Dylan Yudaken
2022-10-31 13:41 ` [PATCH for-next 11/12] io_uring: read_fixed " Dylan Yudaken
2022-10-31 13:41 ` [PATCH for-next 12/12] io_uring: poll_add " Dylan Yudaken
2022-11-02 11:44 ` [PATCH for-next 00/12] io_uring: retarget rsrc nodes periodically Pavel Begunkov
2022-11-02 13:08 ` Dylan Yudaken
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox