Re: [RFC 2/2] io_uring/net: allow to override notification tag

[Stefan Metzmacher <metze@samba.org>]

Am 16.08.22 um 09:42 schrieb Pavel Begunkov:
> Considering limited amount of slots some users struggle with
> registration time notification tag assignment as it's hard to manage
> notifications using sequence numbers. Add a simple feature that copies
> sqe->user_data of a send(+flush) request into the notification CQE it
> flushes (and only when it's flushes).
> 
> Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
> ---
>   include/uapi/linux/io_uring.h | 4 ++++
>   io_uring/net.c                | 6 +++++-
>   2 files changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h
> index 20368394870e..91e7944c9c78 100644
> --- a/include/uapi/linux/io_uring.h
> +++ b/include/uapi/linux/io_uring.h
> @@ -280,11 +280,15 @@ enum io_uring_op {
>    *
>    * IORING_RECVSEND_NOTIF_FLUSH	Flush a notification after a successful
>    *				successful. Only for zerocopy sends.
> + *
> + * IORING_RECVSEND_NOTIF_COPY_TAG Copy request's user_data into the notification
> + *				  completion even if it's flushed.
>    */
>   #define IORING_RECVSEND_POLL_FIRST	(1U << 0)
>   #define IORING_RECV_MULTISHOT		(1U << 1)
>   #define IORING_RECVSEND_FIXED_BUF	(1U << 2)
>   #define IORING_RECVSEND_NOTIF_FLUSH	(1U << 3)
> +#define IORING_RECVSEND_NOTIF_COPY_TAG	(1U << 4)
>   
>   /* cqe->res mask for extracting the notification sequence number */
>   #define IORING_NOTIF_SEQ_MASK		0xFFFFU
> diff --git a/io_uring/net.c b/io_uring/net.c
> index bd3fad9536ef..4d271a269979 100644
> --- a/io_uring/net.c
> +++ b/io_uring/net.c
> @@ -858,7 +858,9 @@ int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
>   
>   	zc->flags = READ_ONCE(sqe->ioprio);
>   	if (zc->flags & ~(IORING_RECVSEND_POLL_FIRST |
> -			  IORING_RECVSEND_FIXED_BUF | IORING_RECVSEND_NOTIF_FLUSH))
> +			  IORING_RECVSEND_FIXED_BUF |
> +			  IORING_RECVSEND_NOTIF_FLUSH |
> +			  IORING_RECVSEND_NOTIF_COPY_TAG))
>   		return -EINVAL;
>   	if (zc->flags & IORING_RECVSEND_FIXED_BUF) {
>   		unsigned idx = READ_ONCE(sqe->buf_index);
> @@ -1024,6 +1026,8 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags)
>   		if (ret == -ERESTARTSYS)
>   			ret = -EINTR;
>   	} else if (zc->flags & IORING_RECVSEND_NOTIF_FLUSH) {
> +		if (zc->flags & IORING_RECVSEND_NOTIF_COPY_TAG)
> +			notif->cqe.user_data = req->cqe.user_data;
>   		io_notif_slot_flush_submit(notif_slot, 0);
>   	}

This would work but it seems to be confusing.

Can't we have a slot-less mode, with slot_idx==U16_MAX,
where we always allocate a new notif for each request,
this would then get the same user_data and would be referenced on the
request in order to reuse the same notif on an async retry after a short send.
And this notif will always be flushed at the end of the request.

This:

struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx,
                                 struct io_notif_slot *slot)

would change to:

struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx,
                                 __u64 cqe_user_data,
				__s32 cqe_res)


and:

void io_notif_slot_flush(struct io_notif_slot *slot) __must_hold(&ctx->uring_lock)

(__must_hold looks wrong there...)

could just be:

void io_notif_flush(struct io_notif_*notif)

What do you think? It would remove the whole notif slot complexity
from caller using IORING_RECVSEND_NOTIF_FLUSH for every request anyway.

metze