* [PATCH for-next 1/8] io_uring/rsrc: remove unused io_rsrc_node::llist
2023-04-18 13:06 [PATCH for-next 0/8] another round of rsrc refactoring Pavel Begunkov
@ 2023-04-18 13:06 ` Pavel Begunkov
2023-04-18 13:06 ` [PATCH for-next 2/8] io_uring/rsrc: infer node from ctx on io_queue_rsrc_removal Pavel Begunkov
` (7 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Pavel Begunkov @ 2023-04-18 13:06 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
->llist was needed for rsrc node destruction offload, which is removed
now. Get rid of the unused field.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/rsrc.h | 1 -
1 file changed, 1 deletion(-)
diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h
index 732496afed4c..525905a30a55 100644
--- a/io_uring/rsrc.h
+++ b/io_uring/rsrc.h
@@ -44,7 +44,6 @@ struct io_rsrc_node {
struct io_rsrc_data *rsrc_data;
};
struct list_head node;
- struct llist_node llist;
int refs;
/*
--
2.40.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH for-next 2/8] io_uring/rsrc: infer node from ctx on io_queue_rsrc_removal
2023-04-18 13:06 [PATCH for-next 0/8] another round of rsrc refactoring Pavel Begunkov
2023-04-18 13:06 ` [PATCH for-next 1/8] io_uring/rsrc: remove unused io_rsrc_node::llist Pavel Begunkov
@ 2023-04-18 13:06 ` Pavel Begunkov
2023-04-18 13:06 ` [PATCH for-next 3/8] io_uring/rsrc: merge nodes and io_rsrc_put Pavel Begunkov
` (6 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Pavel Begunkov @ 2023-04-18 13:06 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
For io_queue_rsrc_removal() we should always use the current active rsrc
node, don't pass it directly but let the function grab it from the
context.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/filetable.c | 5 ++---
io_uring/rsrc.c | 9 +++++----
io_uring/rsrc.h | 3 +--
3 files changed, 8 insertions(+), 9 deletions(-)
diff --git a/io_uring/filetable.c b/io_uring/filetable.c
index 6255fa255ae2..367a44a6c8c5 100644
--- a/io_uring/filetable.c
+++ b/io_uring/filetable.c
@@ -85,8 +85,7 @@ static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file,
return ret;
old_file = (struct file *)(file_slot->file_ptr & FFS_MASK);
- ret = io_queue_rsrc_removal(ctx->file_data, slot_index,
- ctx->rsrc_node, old_file);
+ ret = io_queue_rsrc_removal(ctx->file_data, slot_index, old_file);
if (ret)
return ret;
@@ -163,7 +162,7 @@ int io_fixed_fd_remove(struct io_ring_ctx *ctx, unsigned int offset)
return -EBADF;
file = (struct file *)(file_slot->file_ptr & FFS_MASK);
- ret = io_queue_rsrc_removal(ctx->file_data, offset, ctx->rsrc_node, file);
+ ret = io_queue_rsrc_removal(ctx->file_data, offset, file);
if (ret)
return ret;
diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index db58a51d19da..3be483de613e 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -409,7 +409,7 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx,
if (file_slot->file_ptr) {
file = (struct file *)(file_slot->file_ptr & FFS_MASK);
- err = io_queue_rsrc_removal(data, i, ctx->rsrc_node, file);
+ err = io_queue_rsrc_removal(data, i, file);
if (err)
break;
file_slot->file_ptr = 0;
@@ -492,7 +492,7 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,
i = array_index_nospec(up->offset + done, ctx->nr_user_bufs);
if (ctx->user_bufs[i] != ctx->dummy_ubuf) {
err = io_queue_rsrc_removal(ctx->buf_data, i,
- ctx->rsrc_node, ctx->user_bufs[i]);
+ ctx->user_bufs[i]);
if (unlikely(err)) {
io_buffer_unmap(ctx, &imu);
break;
@@ -680,9 +680,10 @@ int io_files_update(struct io_kiocb *req, unsigned int issue_flags)
return IOU_OK;
}
-int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx,
- struct io_rsrc_node *node, void *rsrc)
+int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc)
{
+ struct io_ring_ctx *ctx = data->ctx;
+ struct io_rsrc_node *node = ctx->rsrc_node;
u64 *tag_slot = io_get_tag_slot(data, idx);
struct io_rsrc_put *prsrc;
diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h
index 525905a30a55..8ed3e6a65cf6 100644
--- a/io_uring/rsrc.h
+++ b/io_uring/rsrc.h
@@ -70,8 +70,7 @@ void io_rsrc_put_work(struct work_struct *work);
void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node);
int __io_rsrc_node_switch_start(struct io_ring_ctx *ctx);
struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx);
-int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx,
- struct io_rsrc_node *node, void *rsrc);
+int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc);
void io_rsrc_node_switch(struct io_ring_ctx *ctx,
struct io_rsrc_data *data_to_kill);
--
2.40.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH for-next 3/8] io_uring/rsrc: merge nodes and io_rsrc_put
2023-04-18 13:06 [PATCH for-next 0/8] another round of rsrc refactoring Pavel Begunkov
2023-04-18 13:06 ` [PATCH for-next 1/8] io_uring/rsrc: remove unused io_rsrc_node::llist Pavel Begunkov
2023-04-18 13:06 ` [PATCH for-next 2/8] io_uring/rsrc: infer node from ctx on io_queue_rsrc_removal Pavel Begunkov
@ 2023-04-18 13:06 ` Pavel Begunkov
2023-04-18 13:06 ` [PATCH for-next 4/8] io_uring/rsrc: add empty flag in rsrc_node Pavel Begunkov
` (5 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Pavel Begunkov @ 2023-04-18 13:06 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
struct io_rsrc_node carries a number of resources represented by struct
io_rsrc_put. That was handy before for sync overhead ammortisation, but
all complexity is gone and nodes are simple and lightweight. Let's
allocate a separate node for each resource.
Nodes and io_rsrc_put and not much different in size, and former are
cached, so node allocation should work better. That also removes some
overhead for nested iteration in io_rsrc_node_ref_zero() /
__io_rsrc_put_work().
Another reason for the patch is that it greatly reduces complexity
by moving io_rsrc_node_switch[_start]() inside io_queue_rsrc_removal(),
so users don't have to care about it.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/filetable.c | 9 -----
io_uring/rsrc.c | 91 +++++++++++---------------------------------
io_uring/rsrc.h | 22 +----------
3 files changed, 25 insertions(+), 97 deletions(-)
diff --git a/io_uring/filetable.c b/io_uring/filetable.c
index 367a44a6c8c5..0f6fa791a47d 100644
--- a/io_uring/filetable.c
+++ b/io_uring/filetable.c
@@ -80,10 +80,6 @@ static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file,
if (file_slot->file_ptr) {
struct file *old_file;
- ret = io_rsrc_node_switch_start(ctx);
- if (ret)
- return ret;
-
old_file = (struct file *)(file_slot->file_ptr & FFS_MASK);
ret = io_queue_rsrc_removal(ctx->file_data, slot_index, old_file);
if (ret)
@@ -91,7 +87,6 @@ static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file,
file_slot->file_ptr = 0;
io_file_bitmap_clear(&ctx->file_table, slot_index);
- io_rsrc_node_switch(ctx, ctx->file_data);
}
ret = io_scm_file_account(ctx, file);
@@ -152,9 +147,6 @@ int io_fixed_fd_remove(struct io_ring_ctx *ctx, unsigned int offset)
return -ENXIO;
if (offset >= ctx->nr_user_files)
return -EINVAL;
- ret = io_rsrc_node_switch_start(ctx);
- if (ret)
- return ret;
offset = array_index_nospec(offset, ctx->nr_user_files);
file_slot = io_fixed_file_slot(&ctx->file_table, offset);
@@ -168,7 +160,6 @@ int io_fixed_fd_remove(struct io_ring_ctx *ctx, unsigned int offset)
file_slot->file_ptr = 0;
io_file_bitmap_clear(&ctx->file_table, offset);
- io_rsrc_node_switch(ctx, ctx->file_data);
return 0;
}
diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index 3be483de613e..a54a222a20b8 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -153,17 +153,10 @@ static void io_rsrc_put_work_one(struct io_rsrc_data *rsrc_data,
static void __io_rsrc_put_work(struct io_rsrc_node *ref_node)
{
struct io_rsrc_data *rsrc_data = ref_node->rsrc_data;
- struct io_rsrc_put *prsrc, *tmp;
- if (ref_node->inline_items)
+ if (likely(ref_node->inline_items))
io_rsrc_put_work_one(rsrc_data, &ref_node->item);
- list_for_each_entry_safe(prsrc, tmp, &ref_node->item_list, list) {
- list_del(&prsrc->list);
- io_rsrc_put_work_one(rsrc_data, prsrc);
- kfree(prsrc);
- }
-
io_rsrc_node_destroy(rsrc_data->ctx, ref_node);
}
@@ -206,53 +199,29 @@ struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx)
}
ref_node->rsrc_data = NULL;
- ref_node->refs = 1;
- INIT_LIST_HEAD(&ref_node->node);
- INIT_LIST_HEAD(&ref_node->item_list);
ref_node->inline_items = 0;
+ ref_node->refs = 1;
return ref_node;
}
-void io_rsrc_node_switch(struct io_ring_ctx *ctx,
- struct io_rsrc_data *data_to_kill)
- __must_hold(&ctx->uring_lock)
-{
- struct io_rsrc_node *node = ctx->rsrc_node;
- struct io_rsrc_node *backup = io_rsrc_node_alloc(ctx);
-
- if (WARN_ON_ONCE(!backup))
- return;
-
- node->rsrc_data = data_to_kill;
- list_add_tail(&node->node, &ctx->rsrc_ref_list);
- /* put master ref */
- io_put_rsrc_node(ctx, node);
- ctx->rsrc_node = backup;
-}
-
-int __io_rsrc_node_switch_start(struct io_ring_ctx *ctx)
-{
- struct io_rsrc_node *node = kzalloc(sizeof(*node), GFP_KERNEL);
-
- if (!node)
- return -ENOMEM;
- io_alloc_cache_put(&ctx->rsrc_node_cache, &node->cache);
- return 0;
-}
-
__cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data,
struct io_ring_ctx *ctx)
{
+ struct io_rsrc_node *backup;
DEFINE_WAIT(we);
int ret;
- /* As we may drop ->uring_lock, other task may have started quiesce */
+ /* As We may drop ->uring_lock, other task may have started quiesce */
if (data->quiesce)
return -ENXIO;
- ret = io_rsrc_node_switch_start(ctx);
- if (ret)
- return ret;
- io_rsrc_node_switch(ctx, data);
+
+ backup = io_rsrc_node_alloc(ctx);
+ if (!backup)
+ return -ENOMEM;
+ ctx->rsrc_node->rsrc_data = data;
+ list_add_tail(&ctx->rsrc_node->node, &ctx->rsrc_ref_list);
+ io_put_rsrc_node(ctx, ctx->rsrc_node);
+ ctx->rsrc_node = backup;
if (list_empty(&ctx->rsrc_ref_list))
return 0;
@@ -382,7 +351,6 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx,
struct file *file;
int fd, i, err = 0;
unsigned int done;
- bool needs_switch = false;
if (!ctx->file_data)
return -ENXIO;
@@ -414,7 +382,6 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx,
break;
file_slot->file_ptr = 0;
io_file_bitmap_clear(&ctx->file_table, i);
- needs_switch = true;
}
if (fd != -1) {
file = fget(fd);
@@ -445,9 +412,6 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx,
io_file_bitmap_set(&ctx->file_table, i);
}
}
-
- if (needs_switch)
- io_rsrc_node_switch(ctx, data);
return done ? done : err;
}
@@ -458,7 +422,6 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,
u64 __user *tags = u64_to_user_ptr(up->tags);
struct iovec iov, __user *iovs = u64_to_user_ptr(up->data);
struct page *last_hpage = NULL;
- bool needs_switch = false;
__u32 done;
int i, err;
@@ -498,15 +461,11 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,
break;
}
ctx->user_bufs[i] = ctx->dummy_ubuf;
- needs_switch = true;
}
ctx->user_bufs[i] = imu;
*io_get_tag_slot(ctx->buf_data, i) = tag;
}
-
- if (needs_switch)
- io_rsrc_node_switch(ctx, ctx->buf_data);
return done ? done : err;
}
@@ -515,15 +474,11 @@ static int __io_register_rsrc_update(struct io_ring_ctx *ctx, unsigned type,
unsigned nr_args)
{
__u32 tmp;
- int err;
lockdep_assert_held(&ctx->uring_lock);
if (check_add_overflow(up->offset, nr_args, &tmp))
return -EOVERFLOW;
- err = io_rsrc_node_switch_start(ctx);
- if (err)
- return err;
switch (type) {
case IORING_RSRC_FILE:
@@ -685,21 +640,21 @@ int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc)
struct io_ring_ctx *ctx = data->ctx;
struct io_rsrc_node *node = ctx->rsrc_node;
u64 *tag_slot = io_get_tag_slot(data, idx);
- struct io_rsrc_put *prsrc;
- if (!node->inline_items) {
- prsrc = &node->item;
- node->inline_items++;
- } else {
- prsrc = kzalloc(sizeof(*prsrc), GFP_KERNEL);
- if (!prsrc)
- return -ENOMEM;
- list_add(&prsrc->list, &node->item_list);
+ ctx->rsrc_node = io_rsrc_node_alloc(ctx);
+ if (unlikely(!ctx->rsrc_node)) {
+ ctx->rsrc_node = node;
+ return -ENOMEM;
}
- prsrc->tag = *tag_slot;
+ node->item.rsrc = rsrc;
+ node->item.tag = *tag_slot;
+ node->inline_items = 1;
*tag_slot = 0;
- prsrc->rsrc = rsrc;
+
+ node->rsrc_data = data;
+ list_add_tail(&node->node, &ctx->rsrc_ref_list);
+ io_put_rsrc_node(ctx, node);
return 0;
}
diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h
index 8ed3e6a65cf6..bad7103f5033 100644
--- a/io_uring/rsrc.h
+++ b/io_uring/rsrc.h
@@ -18,7 +18,6 @@ enum {
};
struct io_rsrc_put {
- struct list_head list;
u64 tag;
union {
void *rsrc;
@@ -43,17 +42,10 @@ struct io_rsrc_node {
struct io_cache_entry cache;
struct io_rsrc_data *rsrc_data;
};
- struct list_head node;
int refs;
-
- /*
- * Keeps a list of struct io_rsrc_put to be completed. Each entry
- * represents one rsrc (e.g. file or buffer), but all of them should've
- * came from the same table and so are of the same type.
- */
- struct list_head item_list;
- struct io_rsrc_put item;
int inline_items;
+ struct list_head node;
+ struct io_rsrc_put item;
};
struct io_mapped_ubuf {
@@ -68,11 +60,8 @@ void io_rsrc_put_tw(struct callback_head *cb);
void io_rsrc_node_ref_zero(struct io_rsrc_node *node);
void io_rsrc_put_work(struct work_struct *work);
void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node);
-int __io_rsrc_node_switch_start(struct io_ring_ctx *ctx);
struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx);
int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc);
-void io_rsrc_node_switch(struct io_ring_ctx *ctx,
- struct io_rsrc_data *data_to_kill);
int io_import_fixed(int ddir, struct iov_iter *iter,
struct io_mapped_ubuf *imu,
@@ -109,13 +98,6 @@ static inline int io_scm_file_account(struct io_ring_ctx *ctx,
return __io_scm_file_account(ctx, file);
}
-static inline int io_rsrc_node_switch_start(struct io_ring_ctx *ctx)
-{
- if (unlikely(io_alloc_cache_empty(&ctx->rsrc_node_cache)))
- return __io_rsrc_node_switch_start(ctx);
- return 0;
-}
-
int io_register_files_update(struct io_ring_ctx *ctx, void __user *arg,
unsigned nr_args);
int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg,
--
2.40.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH for-next 4/8] io_uring/rsrc: add empty flag in rsrc_node
2023-04-18 13:06 [PATCH for-next 0/8] another round of rsrc refactoring Pavel Begunkov
` (2 preceding siblings ...)
2023-04-18 13:06 ` [PATCH for-next 3/8] io_uring/rsrc: merge nodes and io_rsrc_put Pavel Begunkov
@ 2023-04-18 13:06 ` Pavel Begunkov
2023-04-18 13:06 ` [PATCH for-next 5/8] io_uring/rsrc: inline io_rsrc_put_work() Pavel Begunkov
` (4 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Pavel Begunkov @ 2023-04-18 13:06 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
Unless a node was flushed by io_rsrc_ref_quiesce(), it'll carry a
resource. Replace ->inline_items with an empty flag, which is
initialised to false and only raised in io_rsrc_ref_quiesce().
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/rsrc.c | 6 +++---
io_uring/rsrc.h | 2 +-
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index a54a222a20b8..127bd602131e 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -154,7 +154,7 @@ static void __io_rsrc_put_work(struct io_rsrc_node *ref_node)
{
struct io_rsrc_data *rsrc_data = ref_node->rsrc_data;
- if (likely(ref_node->inline_items))
+ if (likely(!ref_node->empty))
io_rsrc_put_work_one(rsrc_data, &ref_node->item);
io_rsrc_node_destroy(rsrc_data->ctx, ref_node);
@@ -199,7 +199,7 @@ struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx)
}
ref_node->rsrc_data = NULL;
- ref_node->inline_items = 0;
+ ref_node->empty = 0;
ref_node->refs = 1;
return ref_node;
}
@@ -218,6 +218,7 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data,
backup = io_rsrc_node_alloc(ctx);
if (!backup)
return -ENOMEM;
+ ctx->rsrc_node->empty = true;
ctx->rsrc_node->rsrc_data = data;
list_add_tail(&ctx->rsrc_node->node, &ctx->rsrc_ref_list);
io_put_rsrc_node(ctx, ctx->rsrc_node);
@@ -649,7 +650,6 @@ int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc)
node->item.rsrc = rsrc;
node->item.tag = *tag_slot;
- node->inline_items = 1;
*tag_slot = 0;
node->rsrc_data = data;
diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h
index bad7103f5033..f3fe455c6c71 100644
--- a/io_uring/rsrc.h
+++ b/io_uring/rsrc.h
@@ -43,7 +43,7 @@ struct io_rsrc_node {
struct io_rsrc_data *rsrc_data;
};
int refs;
- int inline_items;
+ bool empty;
struct list_head node;
struct io_rsrc_put item;
};
--
2.40.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH for-next 5/8] io_uring/rsrc: inline io_rsrc_put_work()
2023-04-18 13:06 [PATCH for-next 0/8] another round of rsrc refactoring Pavel Begunkov
` (3 preceding siblings ...)
2023-04-18 13:06 ` [PATCH for-next 4/8] io_uring/rsrc: add empty flag in rsrc_node Pavel Begunkov
@ 2023-04-18 13:06 ` Pavel Begunkov
2023-04-18 13:06 ` [PATCH for-next 6/8] io_uring/rsrc: pass node to io_rsrc_put_work() Pavel Begunkov
` (3 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Pavel Begunkov @ 2023-04-18 13:06 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
io_rsrc_put_work() is simple enough to be open coded into its only
caller.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/rsrc.c | 19 ++++++-------------
io_uring/rsrc.h | 1 -
2 files changed, 6 insertions(+), 14 deletions(-)
diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index 127bd602131e..d1167b0643b7 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -140,8 +140,8 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo
*slot = NULL;
}
-static void io_rsrc_put_work_one(struct io_rsrc_data *rsrc_data,
- struct io_rsrc_put *prsrc)
+static void io_rsrc_put_work(struct io_rsrc_data *rsrc_data,
+ struct io_rsrc_put *prsrc)
{
struct io_ring_ctx *ctx = rsrc_data->ctx;
@@ -150,16 +150,6 @@ static void io_rsrc_put_work_one(struct io_rsrc_data *rsrc_data,
rsrc_data->do_put(ctx, prsrc);
}
-static void __io_rsrc_put_work(struct io_rsrc_node *ref_node)
-{
- struct io_rsrc_data *rsrc_data = ref_node->rsrc_data;
-
- if (likely(!ref_node->empty))
- io_rsrc_put_work_one(rsrc_data, &ref_node->item);
-
- io_rsrc_node_destroy(rsrc_data->ctx, ref_node);
-}
-
void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *node)
{
if (!io_alloc_cache_put(&ctx->rsrc_node_cache, &node->cache))
@@ -178,7 +168,10 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node)
if (node->refs)
break;
list_del(&node->node);
- __io_rsrc_put_work(node);
+
+ if (likely(!node->empty))
+ io_rsrc_put_work(node->rsrc_data, &node->item);
+ io_rsrc_node_destroy(ctx, node);
}
if (list_empty(&ctx->rsrc_ref_list) && unlikely(ctx->rsrc_quiesce))
wake_up_all(&ctx->rsrc_quiesce_wq);
diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h
index f3fe455c6c71..232079363f6a 100644
--- a/io_uring/rsrc.h
+++ b/io_uring/rsrc.h
@@ -58,7 +58,6 @@ struct io_mapped_ubuf {
void io_rsrc_put_tw(struct callback_head *cb);
void io_rsrc_node_ref_zero(struct io_rsrc_node *node);
-void io_rsrc_put_work(struct work_struct *work);
void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node);
struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx);
int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc);
--
2.40.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH for-next 6/8] io_uring/rsrc: pass node to io_rsrc_put_work()
2023-04-18 13:06 [PATCH for-next 0/8] another round of rsrc refactoring Pavel Begunkov
` (4 preceding siblings ...)
2023-04-18 13:06 ` [PATCH for-next 5/8] io_uring/rsrc: inline io_rsrc_put_work() Pavel Begunkov
@ 2023-04-18 13:06 ` Pavel Begunkov
2023-04-18 13:06 ` [PATCH for-next 7/8] io_uring/rsrc: devirtualise rsrc put callbacks Pavel Begunkov
` (2 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Pavel Begunkov @ 2023-04-18 13:06 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
Instead of passing rsrc_data and a resource to io_rsrc_put_work() just
forward node, that's all the function needs.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/rsrc.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index d1167b0643b7..9378691d49f5 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -140,14 +140,14 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo
*slot = NULL;
}
-static void io_rsrc_put_work(struct io_rsrc_data *rsrc_data,
- struct io_rsrc_put *prsrc)
+static void io_rsrc_put_work(struct io_rsrc_node *node)
{
- struct io_ring_ctx *ctx = rsrc_data->ctx;
+ struct io_rsrc_data *data = node->rsrc_data;
+ struct io_rsrc_put *prsrc = &node->item;
if (prsrc->tag)
- io_post_aux_cqe(ctx, prsrc->tag, 0, 0);
- rsrc_data->do_put(ctx, prsrc);
+ io_post_aux_cqe(data->ctx, prsrc->tag, 0, 0);
+ data->do_put(data->ctx, prsrc);
}
void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *node)
@@ -170,7 +170,7 @@ void io_rsrc_node_ref_zero(struct io_rsrc_node *node)
list_del(&node->node);
if (likely(!node->empty))
- io_rsrc_put_work(node->rsrc_data, &node->item);
+ io_rsrc_put_work(node);
io_rsrc_node_destroy(ctx, node);
}
if (list_empty(&ctx->rsrc_ref_list) && unlikely(ctx->rsrc_quiesce))
--
2.40.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH for-next 7/8] io_uring/rsrc: devirtualise rsrc put callbacks
2023-04-18 13:06 [PATCH for-next 0/8] another round of rsrc refactoring Pavel Begunkov
` (5 preceding siblings ...)
2023-04-18 13:06 ` [PATCH for-next 6/8] io_uring/rsrc: pass node to io_rsrc_put_work() Pavel Begunkov
@ 2023-04-18 13:06 ` Pavel Begunkov
2023-04-18 13:06 ` [PATCH for-next 8/8] io_uring/rsrc: disassociate nodes and rsrc_data Pavel Begunkov
2023-04-19 1:39 ` [PATCH for-next 0/8] another round of rsrc refactoring Jens Axboe
8 siblings, 0 replies; 10+ messages in thread
From: Pavel Begunkov @ 2023-04-18 13:06 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
We only have two rsrc types, buffers and files, replace virtual
callbacks for putting resources down with a switch..case.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/rsrc.c | 25 +++++++++++++++++++------
io_uring/rsrc.h | 2 +-
2 files changed, 20 insertions(+), 7 deletions(-)
diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index 9378691d49f5..62988b3aa927 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -23,6 +23,8 @@ struct io_rsrc_update {
u32 offset;
};
+static void io_rsrc_buf_put(struct io_ring_ctx *ctx, struct io_rsrc_put *prsrc);
+static void io_rsrc_file_put(struct io_ring_ctx *ctx, struct io_rsrc_put *prsrc);
static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov,
struct io_mapped_ubuf **pimu,
struct page **last_hpage);
@@ -147,7 +149,18 @@ static void io_rsrc_put_work(struct io_rsrc_node *node)
if (prsrc->tag)
io_post_aux_cqe(data->ctx, prsrc->tag, 0, 0);
- data->do_put(data->ctx, prsrc);
+
+ switch (data->rsrc_type) {
+ case IORING_RSRC_FILE:
+ io_rsrc_file_put(data->ctx, prsrc);
+ break;
+ case IORING_RSRC_BUFFER:
+ io_rsrc_buf_put(data->ctx, prsrc);
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ break;
+ }
}
void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *node)
@@ -297,8 +310,8 @@ static __cold void **io_alloc_page_table(size_t size)
return table;
}
-__cold static int io_rsrc_data_alloc(struct io_ring_ctx *ctx,
- rsrc_put_fn *do_put, u64 __user *utags,
+__cold static int io_rsrc_data_alloc(struct io_ring_ctx *ctx, int type,
+ u64 __user *utags,
unsigned nr, struct io_rsrc_data **pdata)
{
struct io_rsrc_data *data;
@@ -316,7 +329,7 @@ __cold static int io_rsrc_data_alloc(struct io_ring_ctx *ctx,
data->nr = nr;
data->ctx = ctx;
- data->do_put = do_put;
+ data->rsrc_type = type;
if (utags) {
ret = -EFAULT;
for (i = 0; i < nr; i++) {
@@ -849,7 +862,7 @@ int io_sqe_files_register(struct io_ring_ctx *ctx, void __user *arg,
return -EMFILE;
if (nr_args > rlimit(RLIMIT_NOFILE))
return -EMFILE;
- ret = io_rsrc_data_alloc(ctx, io_rsrc_file_put, tags, nr_args,
+ ret = io_rsrc_data_alloc(ctx, IORING_RSRC_FILE, tags, nr_args,
&ctx->file_data);
if (ret)
return ret;
@@ -1184,7 +1197,7 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,
return -EBUSY;
if (!nr_args || nr_args > IORING_MAX_REG_BUFFERS)
return -EINVAL;
- ret = io_rsrc_data_alloc(ctx, io_rsrc_buf_put, tags, nr_args, &data);
+ ret = io_rsrc_data_alloc(ctx, IORING_RSRC_BUFFER, tags, nr_args, &data);
if (ret)
return ret;
ret = io_buffers_map_alloc(ctx, nr_args);
diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h
index 232079363f6a..5d0733c4c08d 100644
--- a/io_uring/rsrc.h
+++ b/io_uring/rsrc.h
@@ -33,7 +33,7 @@ struct io_rsrc_data {
u64 **tags;
unsigned int nr;
- rsrc_put_fn *do_put;
+ u16 rsrc_type;
bool quiesce;
};
--
2.40.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH for-next 8/8] io_uring/rsrc: disassociate nodes and rsrc_data
2023-04-18 13:06 [PATCH for-next 0/8] another round of rsrc refactoring Pavel Begunkov
` (6 preceding siblings ...)
2023-04-18 13:06 ` [PATCH for-next 7/8] io_uring/rsrc: devirtualise rsrc put callbacks Pavel Begunkov
@ 2023-04-18 13:06 ` Pavel Begunkov
2023-04-19 1:39 ` [PATCH for-next 0/8] another round of rsrc refactoring Jens Axboe
8 siblings, 0 replies; 10+ messages in thread
From: Pavel Begunkov @ 2023-04-18 13:06 UTC (permalink / raw)
To: io-uring; +Cc: Jens Axboe, asml.silence
Make rsrc nodes independent from rsrd_data, for that we keep ctx and
rsrc type in nodes.
Signed-off-by: Pavel Begunkov <[email protected]>
---
io_uring/rsrc.c | 20 +++++++++-----------
io_uring/rsrc.h | 3 ++-
2 files changed, 11 insertions(+), 12 deletions(-)
diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index 62988b3aa927..20dcc7668cb0 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -144,18 +144,17 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo
static void io_rsrc_put_work(struct io_rsrc_node *node)
{
- struct io_rsrc_data *data = node->rsrc_data;
struct io_rsrc_put *prsrc = &node->item;
if (prsrc->tag)
- io_post_aux_cqe(data->ctx, prsrc->tag, 0, 0);
+ io_post_aux_cqe(node->ctx, prsrc->tag, 0, 0);
- switch (data->rsrc_type) {
+ switch (node->type) {
case IORING_RSRC_FILE:
- io_rsrc_file_put(data->ctx, prsrc);
+ io_rsrc_file_put(node->ctx, prsrc);
break;
case IORING_RSRC_BUFFER:
- io_rsrc_buf_put(data->ctx, prsrc);
+ io_rsrc_buf_put(node->ctx, prsrc);
break;
default:
WARN_ON_ONCE(1);
@@ -170,9 +169,9 @@ void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *node)
}
void io_rsrc_node_ref_zero(struct io_rsrc_node *node)
- __must_hold(&node->rsrc_data->ctx->uring_lock)
+ __must_hold(&node->ctx->uring_lock)
{
- struct io_ring_ctx *ctx = node->rsrc_data->ctx;
+ struct io_ring_ctx *ctx = node->ctx;
while (!list_empty(&ctx->rsrc_ref_list)) {
node = list_first_entry(&ctx->rsrc_ref_list,
@@ -204,7 +203,7 @@ struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx)
return NULL;
}
- ref_node->rsrc_data = NULL;
+ ref_node->ctx = ctx;
ref_node->empty = 0;
ref_node->refs = 1;
return ref_node;
@@ -225,7 +224,7 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data,
if (!backup)
return -ENOMEM;
ctx->rsrc_node->empty = true;
- ctx->rsrc_node->rsrc_data = data;
+ ctx->rsrc_node->type = -1;
list_add_tail(&ctx->rsrc_node->node, &ctx->rsrc_ref_list);
io_put_rsrc_node(ctx, ctx->rsrc_node);
ctx->rsrc_node = backup;
@@ -655,10 +654,9 @@ int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc)
}
node->item.rsrc = rsrc;
+ node->type = data->rsrc_type;
node->item.tag = *tag_slot;
*tag_slot = 0;
-
- node->rsrc_data = data;
list_add_tail(&node->node, &ctx->rsrc_ref_list);
io_put_rsrc_node(ctx, node);
return 0;
diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h
index 5d0733c4c08d..0a8a95e9b99e 100644
--- a/io_uring/rsrc.h
+++ b/io_uring/rsrc.h
@@ -40,10 +40,11 @@ struct io_rsrc_data {
struct io_rsrc_node {
union {
struct io_cache_entry cache;
- struct io_rsrc_data *rsrc_data;
+ struct io_ring_ctx *ctx;
};
int refs;
bool empty;
+ u16 type;
struct list_head node;
struct io_rsrc_put item;
};
--
2.40.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH for-next 0/8] another round of rsrc refactoring
2023-04-18 13:06 [PATCH for-next 0/8] another round of rsrc refactoring Pavel Begunkov
` (7 preceding siblings ...)
2023-04-18 13:06 ` [PATCH for-next 8/8] io_uring/rsrc: disassociate nodes and rsrc_data Pavel Begunkov
@ 2023-04-19 1:39 ` Jens Axboe
8 siblings, 0 replies; 10+ messages in thread
From: Jens Axboe @ 2023-04-19 1:39 UTC (permalink / raw)
To: io-uring, Pavel Begunkov
On Tue, 18 Apr 2023 14:06:33 +0100, Pavel Begunkov wrote:
> Further simplify rsrc infrastructure, and make it a little bit
> faster.
>
> The main part is Patch 3, which establishes 1:1 relation between
> struct io_rsrc_put and nodes, which removes io_rsrc_node_switch() /
> io_rsrc_node_switch_start() and all the additional complexity with
> pre allocations. Note, it doesn't change any guarantees as
> io_queue_rsrc_removal() was doing allocations anyway and could
> always fail.
>
> [...]
Applied, thanks!
[1/8] io_uring/rsrc: remove unused io_rsrc_node::llist
commit: 2e6f45ac0e640bbd49296adfa0982c84f85fa342
[2/8] io_uring/rsrc: infer node from ctx on io_queue_rsrc_removal
commit: 63fea89027ff4fd4f350b471ad5b9220d373eec5
[3/8] io_uring/rsrc: merge nodes and io_rsrc_put
commit: c376644fb915fbdea8c4a04f859d032a8be352fd
[4/8] io_uring/rsrc: add empty flag in rsrc_node
commit: 26147da37f3e52041d9deba189d39f27ce78a84f
[5/8] io_uring/rsrc: inline io_rsrc_put_work()
commit: 4130b49991d6b8ca0ea44cb256e710c4e48d7f01
[6/8] io_uring/rsrc: pass node to io_rsrc_put_work()
commit: 29b26c556e7439b1370ac6a59fce83a9d1521de1
[7/8] io_uring/rsrc: devirtualise rsrc put callbacks
commit: fc7f3a8d3a78503c4f3e108155fb9a233dc307a4
[8/8] io_uring/rsrc: disassociate nodes and rsrc_data
commit: 2236b3905b4d4e9cd4d149ab35767858c02bb79b
Best regards,
--
Jens Axboe
^ permalink raw reply [flat|nested] 10+ messages in thread