[RFC 0/9] optimise zcrx refs cache bouncing

[Pavel Begunkov <asml.silence@gmail.com>]

zcrx works well when user space and NAPI run on the same CPU but loses a
chunk of performance otherwise. It's caused by cache bounces from
1) zcrx "user" refs, which track whether buffers are given to the user
2) niov refs, as zcrx bumps them on recv(), and hence
   skb_attempt_defer_free() doesn't help.

In this patchset, zcrx steals received skbs, delays their destruction
similar to skb_attempt_defer_free(), and processes them in
io_pp_zc_alloc_netmems(). This moves all aforementioned refs
modifications for the hot path to the NAPI context.

For the networking side the most interesting bits are patches 1 and 8,
and patch 6 around the call to tcp_read_sock_steal_skb(). I'm looking
to get opinions on whether tcp_read_sock_steal_skb() is fine or what
kind of helpers / API would work better.

Tested with liburing/examples/{zcrx + send-zerocopy},
200Gbit/s NICs, rx_page=4KB

before: MB/s=18948
CPU    %usr    %sys %iowait    %irq   %soft  %idle
  0    4.92   63.68    0.00    1.64    2.84  26.91
  7    0.00    0.00    0.00    0.30   89.50  10.20

after: MB/s=21034
CPU    %usr    %sys %iowait    %irq   %soft  %idle
  0    5.59   50.18    0.00    2.26    2.73  39.24
  7    0.00    0.00    0.00    0.20   87.49  12.31

Helps in a similar way to 32KB rx page size, and also improves numbers
when NAPI and user space run on the same CPU.

kernel:
url: https://github.com/isilence/linux/tree/zcrx/skb-stealing
git: https://github.com/isilence/linux.git zcrx/skb-stealing

liburing (can be used any other version):
url: https://github.com/isilence/liburing/tree/zcrx/test-skb-steal
git: https://github.com/isilence/liburing.git zcrx/test-skb-steal

Pavel Begunkov (9):
  net: allow __tcp_read_sock actors to steal skbs
  net: add provider specific net_iov field
  io_uring/zcrx: don't save/restore count for frag skbs
  io_uring/zcrx: split frag handling loop
  io_uring/zcrx: split io_zcrx_recv_frag()
  io_uring/zcrx: implement skb stealing
  io_uring/zcrx: don't lock for single producer ptr ring
  io_uring/zcrx: steal niov refs
  io_uring/zcrx: add rq_lock cache of "user" niov refs

 include/linux/net.h  |   1 +
 include/net/netmem.h |   1 +
 include/net/tcp.h    |  13 +++
 io_uring/zcrx.c      | 210 ++++++++++++++++++++++++++++++++++---------
 io_uring/zcrx.h      |   4 +
 net/ipv4/tcp.c       |  11 +++
 6 files changed, 200 insertions(+), 40 deletions(-)

-- 
2.54.0