* [PATCH v2 1/2] io_uring/cmd: add iovec cache for commands
2025-03-21 18:04 [PATCH v2 0/2] cmd infra for caching iovec/bvec Pavel Begunkov
@ 2025-03-21 18:04 ` Pavel Begunkov
2025-03-21 18:04 ` [PATCH v2 2/2] io_uring/cmd: introduce io_uring_cmd_import_fixed_vec Pavel Begunkov
` (2 subsequent siblings)
3 siblings, 0 replies; 7+ messages in thread
From: Pavel Begunkov @ 2025-03-21 18:04 UTC (permalink / raw)
To: io-uring; +Cc: asml.silence
Add iou_vec to commands and wire caching for it, but don't expose it to
users just yet. We need the vec cleared on initial alloc, but since
we can't place it at the beginning at the moment, zero the entire
async_data. It's cached, and the performance effects only the initial
allocation, and it might be not a bad idea since we're exposing those
bits to outside drivers.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/io_uring.c | 5 +++--
io_uring/opdef.c | 1 +
io_uring/uring_cmd.c | 20 +++++++++++++++++++-
io_uring/uring_cmd.h | 5 +++++
4 files changed, 28 insertions(+), 3 deletions(-)
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 5eb9be063a7c..e1128b9551aa 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -289,7 +289,7 @@ static void io_free_alloc_caches(struct io_ring_ctx *ctx)
io_alloc_cache_free(&ctx->apoll_cache, kfree);
io_alloc_cache_free(&ctx->netmsg_cache, io_netmsg_cache_free);
io_alloc_cache_free(&ctx->rw_cache, io_rw_cache_free);
- io_alloc_cache_free(&ctx->cmd_cache, kfree);
+ io_alloc_cache_free(&ctx->cmd_cache, io_cmd_cache_free);
io_alloc_cache_free(&ctx->msg_cache, kfree);
io_futex_cache_free(ctx);
io_rsrc_cache_free(ctx);
@@ -335,7 +335,8 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)
sizeof(struct io_async_rw),
offsetof(struct io_async_rw, clear));
ret |= io_alloc_cache_init(&ctx->cmd_cache, IO_ALLOC_CACHE_MAX,
- sizeof(struct io_async_cmd), 0);
+ sizeof(struct io_async_cmd),
+ sizeof(struct io_async_cmd));
spin_lock_init(&ctx->msg_lock);
ret |= io_alloc_cache_init(&ctx->msg_cache, IO_ALLOC_CACHE_MAX,
sizeof(struct io_kiocb), 0);
diff --git a/io_uring/opdef.c b/io_uring/opdef.c
index e4aa61a414fb..489384c0438b 100644
--- a/io_uring/opdef.c
+++ b/io_uring/opdef.c
@@ -755,6 +755,7 @@ const struct io_cold_def io_cold_defs[] = {
},
[IORING_OP_URING_CMD] = {
.name = "URING_CMD",
+ .cleanup = io_uring_cmd_cleanup,
},
[IORING_OP_SEND_ZC] = {
.name = "SEND_ZC",
diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c
index 7c126ee497ea..6a21cdaaf495 100644
--- a/io_uring/uring_cmd.c
+++ b/io_uring/uring_cmd.c
@@ -16,6 +16,14 @@
#include "rsrc.h"
#include "uring_cmd.h"
+void io_cmd_cache_free(const void *entry)
+{
+ struct io_async_cmd *ac = (struct io_async_cmd *)entry;
+
+ io_vec_free(&ac->vec);
+ kfree(ac);
+}
+
static void io_req_uring_cleanup(struct io_kiocb *req, unsigned int issue_flags)
{
struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd);
@@ -29,13 +37,23 @@ static void io_req_uring_cleanup(struct io_kiocb *req, unsigned int issue_flags)
if (issue_flags & IO_URING_F_UNLOCKED)
return;
+
+ io_alloc_cache_vec_kasan(&ac->vec);
+ if (ac->vec.nr > IO_VEC_CACHE_SOFT_CAP)
+ io_vec_free(&ac->vec);
+
if (io_alloc_cache_put(&req->ctx->cmd_cache, cache)) {
ioucmd->sqe = NULL;
req->async_data = NULL;
- req->flags &= ~REQ_F_ASYNC_DATA;
+ req->flags &= ~(REQ_F_ASYNC_DATA|REQ_F_NEED_CLEANUP);
}
}
+void io_uring_cmd_cleanup(struct io_kiocb *req)
+{
+ io_req_uring_cleanup(req, 0);
+}
+
bool io_uring_try_cancel_uring_cmd(struct io_ring_ctx *ctx,
struct io_uring_task *tctx, bool cancel_all)
{
diff --git a/io_uring/uring_cmd.h b/io_uring/uring_cmd.h
index 2ec3a8785534..b45ec7cffcd1 100644
--- a/io_uring/uring_cmd.h
+++ b/io_uring/uring_cmd.h
@@ -1,13 +1,18 @@
// SPDX-License-Identifier: GPL-2.0
#include <linux/io_uring/cmd.h>
+#include <linux/io_uring_types.h>
struct io_async_cmd {
struct io_uring_cmd_data data;
+ struct iou_vec vec;
};
int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags);
int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);
+void io_uring_cmd_cleanup(struct io_kiocb *req);
bool io_uring_try_cancel_uring_cmd(struct io_ring_ctx *ctx,
struct io_uring_task *tctx, bool cancel_all);
+
+void io_cmd_cache_free(const void *entry);
--
2.48.1
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH v2 2/2] io_uring/cmd: introduce io_uring_cmd_import_fixed_vec
2025-03-21 18:04 [PATCH v2 0/2] cmd infra for caching iovec/bvec Pavel Begunkov
2025-03-21 18:04 ` [PATCH v2 1/2] io_uring/cmd: add iovec cache for commands Pavel Begunkov
@ 2025-03-21 18:04 ` Pavel Begunkov
2025-03-21 19:13 ` [PATCH v2 0/2] cmd infra for caching iovec/bvec Jens Axboe
2025-03-21 19:45 ` Jens Axboe
3 siblings, 0 replies; 7+ messages in thread
From: Pavel Begunkov @ 2025-03-21 18:04 UTC (permalink / raw)
To: io-uring; +Cc: asml.silence
io_uring_cmd_import_fixed_vec() is a cmd helper around vectored
registered buffer import functions, which caches the memory under
the hood. The lifetime of the vectore and hence the iterator is bound to
the request. Furthermore, the user is not allowed to call it multiple
times for a single request.
Signed-off-by: Pavel Begunkov <[email protected]>
---
include/linux/io_uring/cmd.h | 13 +++++++++++++
io_uring/uring_cmd.c | 19 +++++++++++++++++++
io_uring/uring_cmd.h | 6 ++++++
3 files changed, 38 insertions(+)
diff --git a/include/linux/io_uring/cmd.h b/include/linux/io_uring/cmd.h
index 598cacda4aa3..e6723fa95160 100644
--- a/include/linux/io_uring/cmd.h
+++ b/include/linux/io_uring/cmd.h
@@ -43,6 +43,11 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
struct iov_iter *iter,
struct io_uring_cmd *ioucmd,
unsigned int issue_flags);
+int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,
+ const struct iovec __user *uvec,
+ size_t uvec_segs,
+ int ddir, struct iov_iter *iter,
+ unsigned issue_flags);
/*
* Completes the request, i.e. posts an io_uring CQE and deallocates @ioucmd
@@ -76,6 +81,14 @@ io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
{
return -EOPNOTSUPP;
}
+static inline int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,
+ const struct iovec __user *uvec,
+ size_t uvec_segs,
+ int ddir, struct iov_iter *iter,
+ unsigned issue_flags)
+{
+ return -EOPNOTSUPP;
+}
static inline void io_uring_cmd_done(struct io_uring_cmd *cmd, ssize_t ret,
u64 ret2, unsigned issue_flags)
{
diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c
index 6a21cdaaf495..f2cfc371f3d0 100644
--- a/io_uring/uring_cmd.c
+++ b/io_uring/uring_cmd.c
@@ -277,6 +277,25 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
}
EXPORT_SYMBOL_GPL(io_uring_cmd_import_fixed);
+int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,
+ const struct iovec __user *uvec,
+ size_t uvec_segs,
+ int ddir, struct iov_iter *iter,
+ unsigned issue_flags)
+{
+ struct io_kiocb *req = cmd_to_io_kiocb(ioucmd);
+ struct io_async_cmd *ac = req->async_data;
+ int ret;
+
+ ret = io_prep_reg_iovec(req, &ac->vec, uvec, uvec_segs);
+ if (ret)
+ return ret;
+
+ return io_import_reg_vec(ddir, iter, req, &ac->vec, uvec_segs,
+ issue_flags);
+}
+EXPORT_SYMBOL_GPL(io_uring_cmd_import_fixed_vec);
+
void io_uring_cmd_issue_blocking(struct io_uring_cmd *ioucmd)
{
struct io_kiocb *req = cmd_to_io_kiocb(ioucmd);
diff --git a/io_uring/uring_cmd.h b/io_uring/uring_cmd.h
index b45ec7cffcd1..14e525255854 100644
--- a/io_uring/uring_cmd.h
+++ b/io_uring/uring_cmd.h
@@ -16,3 +16,9 @@ bool io_uring_try_cancel_uring_cmd(struct io_ring_ctx *ctx,
struct io_uring_task *tctx, bool cancel_all);
void io_cmd_cache_free(const void *entry);
+
+int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *ioucmd,
+ const struct iovec __user *uvec,
+ size_t uvec_segs,
+ int ddir, struct iov_iter *iter,
+ unsigned issue_flags);
--
2.48.1
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH v2 0/2] cmd infra for caching iovec/bvec
2025-03-21 18:04 [PATCH v2 0/2] cmd infra for caching iovec/bvec Pavel Begunkov
2025-03-21 18:04 ` [PATCH v2 1/2] io_uring/cmd: add iovec cache for commands Pavel Begunkov
2025-03-21 18:04 ` [PATCH v2 2/2] io_uring/cmd: introduce io_uring_cmd_import_fixed_vec Pavel Begunkov
@ 2025-03-21 19:13 ` Jens Axboe
2025-03-22 9:42 ` Ming Lei
2025-03-21 19:45 ` Jens Axboe
3 siblings, 1 reply; 7+ messages in thread
From: Jens Axboe @ 2025-03-21 19:13 UTC (permalink / raw)
To: Pavel Begunkov, io-uring; +Cc: Ming Lei
On 3/21/25 12:04 PM, Pavel Begunkov wrote:
> Add infrastructure that is going to be used by commands for importing
> vectored registered buffers. It can also be reused later for iovec
> caching.
>
> v2: clear the vec on first ->async_data allocation
> fix a memory leak
>
> Pavel Begunkov (2):
> io_uring/cmd: add iovec cache for commands
> io_uring/cmd: introduce io_uring_cmd_import_fixed_vec
>
> include/linux/io_uring/cmd.h | 13 ++++++++++++
> io_uring/io_uring.c | 5 +++--
> io_uring/opdef.c | 1 +
> io_uring/uring_cmd.c | 39 +++++++++++++++++++++++++++++++++++-
> io_uring/uring_cmd.h | 11 ++++++++++
> 5 files changed, 66 insertions(+), 3 deletions(-)
This version works for me - adding in Ming, so he can test and
verify as well.
--
Jens Axboe
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 0/2] cmd infra for caching iovec/bvec
2025-03-21 19:13 ` [PATCH v2 0/2] cmd infra for caching iovec/bvec Jens Axboe
@ 2025-03-22 9:42 ` Ming Lei
2025-03-22 12:09 ` Pavel Begunkov
0 siblings, 1 reply; 7+ messages in thread
From: Ming Lei @ 2025-03-22 9:42 UTC (permalink / raw)
To: Jens Axboe; +Cc: Pavel Begunkov, io-uring
On Fri, Mar 21, 2025 at 01:13:23PM -0600, Jens Axboe wrote:
> On 3/21/25 12:04 PM, Pavel Begunkov wrote:
> > Add infrastructure that is going to be used by commands for importing
> > vectored registered buffers. It can also be reused later for iovec
> > caching.
> >
> > v2: clear the vec on first ->async_data allocation
> > fix a memory leak
> >
> > Pavel Begunkov (2):
> > io_uring/cmd: add iovec cache for commands
> > io_uring/cmd: introduce io_uring_cmd_import_fixed_vec
> >
> > include/linux/io_uring/cmd.h | 13 ++++++++++++
> > io_uring/io_uring.c | 5 +++--
> > io_uring/opdef.c | 1 +
> > io_uring/uring_cmd.c | 39 +++++++++++++++++++++++++++++++++++-
> > io_uring/uring_cmd.h | 11 ++++++++++
> > 5 files changed, 66 insertions(+), 3 deletions(-)
>
> This version works for me - adding in Ming, so he can test and
> verify as well.
With the two patches, all ublk selftest can run to pass, and kernel doesn't
panic any more.
BTW, I meant vectored fix kernel buffer support for FS read/write, which
looks not supported yet.
And it can be useful in the following case:
https://lore.kernel.org/linux-block/[email protected]/T/#u
Thanks,
Ming
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 0/2] cmd infra for caching iovec/bvec
2025-03-22 9:42 ` Ming Lei
@ 2025-03-22 12:09 ` Pavel Begunkov
0 siblings, 0 replies; 7+ messages in thread
From: Pavel Begunkov @ 2025-03-22 12:09 UTC (permalink / raw)
To: Ming Lei, Jens Axboe; +Cc: io-uring
On 3/22/25 09:42, Ming Lei wrote:
> On Fri, Mar 21, 2025 at 01:13:23PM -0600, Jens Axboe wrote:
>> On 3/21/25 12:04 PM, Pavel Begunkov wrote:
>>> Add infrastructure that is going to be used by commands for importing
>>> vectored registered buffers. It can also be reused later for iovec
>>> caching.
>>>
>>> v2: clear the vec on first ->async_data allocation
>>> fix a memory leak
>>>
>>> Pavel Begunkov (2):
>>> io_uring/cmd: add iovec cache for commands
>>> io_uring/cmd: introduce io_uring_cmd_import_fixed_vec
>>>
>>> include/linux/io_uring/cmd.h | 13 ++++++++++++
>>> io_uring/io_uring.c | 5 +++--
>>> io_uring/opdef.c | 1 +
>>> io_uring/uring_cmd.c | 39 +++++++++++++++++++++++++++++++++++-
>>> io_uring/uring_cmd.h | 11 ++++++++++
>>> 5 files changed, 66 insertions(+), 3 deletions(-)
>>
>> This version works for me - adding in Ming, so he can test and
>> verify as well.
>
> With the two patches, all ublk selftest can run to pass, and kernel doesn't
> panic any more.
thanks for testing.
>
> BTW, I meant vectored fix kernel buffer support for FS read/write, which
> looks not supported yet.
Right, you can't use do a vectored registered buffer read/write/etc.
with a kernel buffer (ala ublk zc provided one). That will likely
need a special version of io_vec_fill_bvec() that would account
for bvec entries not being uniform in size.
--
Pavel Begunkov
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 0/2] cmd infra for caching iovec/bvec
2025-03-21 18:04 [PATCH v2 0/2] cmd infra for caching iovec/bvec Pavel Begunkov
` (2 preceding siblings ...)
2025-03-21 19:13 ` [PATCH v2 0/2] cmd infra for caching iovec/bvec Jens Axboe
@ 2025-03-21 19:45 ` Jens Axboe
3 siblings, 0 replies; 7+ messages in thread
From: Jens Axboe @ 2025-03-21 19:45 UTC (permalink / raw)
To: io-uring, Pavel Begunkov
On Fri, 21 Mar 2025 18:04:32 +0000, Pavel Begunkov wrote:
> Add infrastructure that is going to be used by commands for importing
> vectored registered buffers. It can also be reused later for iovec
> caching.
>
> v2: clear the vec on first ->async_data allocation
> fix a memory leak
>
> [...]
Applied, thanks!
[1/2] io_uring/cmd: add iovec cache for commands
commit: 3a4689ac109f18f23ea0d0c1c79e055142796858
[2/2] io_uring/cmd: introduce io_uring_cmd_import_fixed_vec
commit: ef490275297267d9461733ecd9b02bd3b798b3a4
Best regards,
--
Jens Axboe
^ permalink raw reply [flat|nested] 7+ messages in thread