public inbox for [email protected]
 help / color / mirror / Atom feed
* [RFC RESEND 00/11] Zero copy network RX using io_uring
@ 2023-08-26  1:19 David Wei
  2023-08-26  1:19 ` [PATCH 01/11] io_uring: add interface queue David Wei
                   ` (11 more replies)
  0 siblings, 12 replies; 18+ messages in thread
From: David Wei @ 2023-08-26  1:19 UTC (permalink / raw)
  To: Jens Axboe, Pavel Begunkov; +Cc: io-uring, netdev, Mina Almasry, Jakub Kicinski

From: David Wei <[email protected]>

This patchset is a proposal that adds zero copy network RX to io_uring.
With it, userspace can register a region of host memory for receiving
data directly from a NIC using DMA, without needing a kernel to user
copy.

Software support is added to the Broadcom BNXT driver. Hardware support
for receive flow steering and header splitting is required.

On the userspace side, a sample server is added in this branch of
liburing:
https://github.com/spikeh/liburing/tree/zcrx2

Build liburing as normal, and run examples/zcrx. Then, set flow steering
rules using ethtool. A sample shell script is included in
examples/zcrx_flow.sh, but you need to change the source IP. Finally,
connect a client using e.g. netcat and send data.

This patchset + userspace code was tested on an Intel Xeon Platinum
8321HC CPU and Broadcom BCM57504 NIC.

Early benchmarks using this prototype, with iperf3 as a load generator,
showed a ~50% reduction in overall system memory bandwidth as measured
using perf counters. Note that DDIO must be disabled on Intel systems.

Mina et al. from Google and Kuba are collaborating on a similar proposal
to ZC from NIC to devmem. There are many shared functionality in netdev
that we can collaborate on e.g.:
* Page pool memory provider backend and resource registration
* Page pool refcounted iov/buf representation and lifecycle
* Setting receive flow steering

As mentioned earlier, this is an early prototype. It is brittle, some
functionality is missing and there's little optimisation. We're looking
for feedback on the overall approach and points of collaboration in
netdev.
* No copy fallback, if payload ends up in linear part of skb then the
  code will not work
* No way to pin an RX queue to a specific CPU
* Only one ifq, one pool region, on RX queue...

This patchset is based on the work by Jonathan Lemon
<[email protected]>:
https://lore.kernel.org/io-uring/[email protected]/

David Wei (11):
  io_uring: add interface queue
  io_uring: add mmap support for shared ifq ringbuffers
  netdev: add XDP_SETUP_ZC_RX command
  io_uring: setup ZC for an RX queue when registering an ifq
  io_uring: add ZC buf and pool
  io_uring: add ZC pool API
  skbuff: add SKBFL_FIXED_FRAG and skb_fixed()
  io_uring: allocate a uarg for freeing zero copy skbs
  io_uring: delay ZC pool destruction
  netdev/bnxt: add data pool and use it in BNXT driver
  io_uring: add io_recvzc request

 drivers/net/ethernet/broadcom/bnxt/bnxt.c     |  59 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt.h     |   4 +
 drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c |   3 +
 include/linux/io_uring.h                      |  31 +
 include/linux/io_uring_types.h                |   6 +
 include/linux/netdevice.h                     |  11 +
 include/linux/skbuff.h                        |  10 +-
 include/net/data_pool.h                       |  96 +++
 include/uapi/linux/io_uring.h                 |  53 ++
 io_uring/Makefile                             |   3 +-
 io_uring/io_uring.c                           |  13 +
 io_uring/kbuf.c                               |  30 +
 io_uring/kbuf.h                               |   5 +
 io_uring/net.c                                |  83 +-
 io_uring/opdef.c                              |  16 +
 io_uring/zc_rx.c                              | 723 ++++++++++++++++++
 io_uring/zc_rx.h                              |  42 +
 17 files changed, 1168 insertions(+), 20 deletions(-)
 create mode 100644 include/net/data_pool.h
 create mode 100644 io_uring/zc_rx.c
 create mode 100644 io_uring/zc_rx.h

-- 
2.39.3


^ permalink raw reply	[flat|nested] 18+ messages in thread

end of thread, other threads:[~2023-10-23  3:35 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-08-26  1:19 [RFC RESEND 00/11] Zero copy network RX using io_uring David Wei
2023-08-26  1:19 ` [PATCH 01/11] io_uring: add interface queue David Wei
2023-08-26  1:19 ` [PATCH 02/11] io_uring: add mmap support for shared ifq ringbuffers David Wei
2023-08-26  1:19 ` [PATCH 03/11] netdev: add XDP_SETUP_ZC_RX command David Wei
2023-08-26  2:21   ` David Ahern
2023-08-26 21:37     ` David Wei
2023-08-26  1:19 ` [PATCH 04/11] io_uring: setup ZC for an RX queue when registering an ifq David Wei
2023-08-26  2:26   ` David Ahern
2023-08-26 22:00     ` David Wei
2023-08-26  1:19 ` [PATCH 05/11] io_uring: add ZC buf and pool David Wei
2023-08-26  1:19 ` [PATCH 06/11] io_uring: add ZC pool API David Wei
2023-08-26  1:19 ` [PATCH 07/11] skbuff: add SKBFL_FIXED_FRAG and skb_fixed() David Wei
2023-08-26  1:19 ` [PATCH 08/11] io_uring: allocate a uarg for freeing zero copy skbs David Wei
2023-08-26  1:19 ` [PATCH 09/11] io_uring: delay ZC pool destruction David Wei
2023-08-26  1:19 ` [PATCH 10/11] netdev/bnxt: add data pool and use it in BNXT driver David Wei
2023-08-26  1:19 ` [PATCH 11/11] io_uring: add io_recvzc request David Wei
2023-10-22 19:06 ` [RFC RESEND 00/11] Zero copy network RX using io_uring Gal Pressman
2023-10-23  3:35   ` David Wei

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox