Re: [RFC 6/6] io_uring/notif: implement notification stacking

[Pavel Begunkov <asml.silence@gmail.com>]

On 4/14/24 18:10, Willem de Bruijn wrote:
> Pavel Begunkov wrote:
>> The network stack allows only one ubuf_info per skb, and unlike
>> MSG_ZEROCOPY, each io_uring zerocopy send will carry a separate
>> ubuf_info. That means that send requests can't reuse a previosly
>> allocated skb and need to get one more or more of new ones. That's fine
>> for large sends, but otherwise it would spam the stack with lots of skbs
>> carrying just a little data each.
> 
> Can you give a little context why each send request has to be a
> separate ubuf_info?
> 
> This patch series aims to make that model more efficient. Would it be
> possible to just change the model instead? I assume you tried that and
> it proved unworkable, but is it easy to explain what the fundamental
> blocker is?

The uapi is so that you get a buffer completion (analogous to what you
get with recv(MSG_ERRQUEUE)) for each send request. With that, for skb
to serve multiple send requests it'd need to store a list of completions
in some way. One could try to track sockets, have one "active" ubuf_info
per socket which all sends would use, and then eventually flush the
active ubuf so it can post completions and create a new one. but io_uring
wouldn't know when it needs to "flush", whenever in the net stack it
happens naturally when it pushes skbs from the queue. Not to mention
that socket tracking has its own complications.

As for uapi, in early versions of io_uring's SEND_ZC, ubuf_info and
requests weren't entangled, roughly speaking, the user could choose
that this request should use this ubuf_info (I can elaborate if
interesting). It wasn't too complex, but all feedback was pointing
that it's much easier to use hot it is now, and honestly it does
buy with simplicity.

I'm not sure what a different model would give. We wouldn't win
in efficiency comparing to this patch, I can go into details
how there are no extra atomics/locks/kmalloc/etc., the only bit
is waking up waiting tasks, but that still would need to happen.
I can even optimise / ammortise ubuf refcounting if that would
matter.

> MSG_ZEROCOPY uses uarg->len to identify multiple consecutive send
> operations that can be notified at once.

-- 
Pavel Begunkov