Re: [RFC PATCH v2 00/13] zero-copy RX for io_uring

[Ziyang Zhang <ZiyangZhang@linux.alibaba.com>]

On 2022/10/19 03:15, Jonathan Lemon wrote:
> This series is a RFC for io_uring/zctap.  This is an evolution of
> the earlier zctap work, re-targeted to use io_uring as the userspace
> API.  The current code is intended to provide a zero-copy RX path for
> upper-level networking protocols (aka TCP and UDP).  The current draft
> focuses on host-provided memory (not GPU memory).
> 
> This RFC contains the upper-level core code required for operation,
> with the intent of soliciting feedback on the general API.  This does
> not contain the network driver side changes required for complete
> operation.  Also please note that as an RFC, there are some things
> which are incomplete or in need of rework.
> 
> The intent is to use a network driver which provides header/data
> splitting, so the frame header (which is processed by the networking
> stack) does not reside in user memory.
> 
> The code is successfully receiving a zero-copy TCP stream from a
> remote sender.  An RFC, the intent is to solicit feedback on the
> API and overall design.  The current code will also work with
> system pages, copying the data out to the application - this is
> intended as a fallback/testing path.
> 
> Performance numbers are coming soon!
> 
> High level description:
> 
> The application allocates a frame backing store, and provides this
> to the kernel for use.  An interface queue is requested from the
> networking device, and incoming frames are deposited into the provided
> memory region.
> 
> Responsibility for correctly steering incoming frames to the queue
> is outside the scope of this work - it is assumed that the user 
> has set steering rules up separately.
> 
> Incoming frames are sent up the stack as skb's and eventually
> land in the application's socket receive queue.  This differs
> from AF_XDP, which receives raw frames directly to userspace,
> without protocol processing.
> 
> The RECV_ZC opcode then returns an iov[] style vector which points
> to the data in userspace memory.  When the application has completed
> processing of the data, the buffer is returned back to the kernel
> through a fill ring for reuse.
> 
> Changelog:
>  v1: initial version
>  v2: Remove separate PROVIDE_REGION opcode, fold this functionality
>      into REGISTER_IFQ.  Remove page_pool hooks, as it appears the 
>      page pool is currently incompatible with user-mapped memory.
>      Add io_zctap_buffers and network driver API.
> .
> 
> Jonathan Lemon (13):
>   io_uring: add zctap ifq definition
>   netdevice: add SETUP_ZCTAP to the netdev_bpf structure
>   io_uring: add register ifq opcode
>   io_uring: create a zctap region for a mapped buffer
>   io_uring: create page freelist for the ifq region
>   io_uring: Provide driver API for zctap packet buffers.
>   io_uring: Allocate the zctap buffers for the device
>   io_uring: Add zctap buffer get/put functions and refcounting.
>   skbuff: Introduce SKBFL_FIXED_FRAG and skb_fixed()
>   io_uring: Allocate a uarg for use by the ifq RX
>   io_uring: Define the zctap iov[] returned to the user.
>   io_uring: add OP_RECV_ZC command.
>   io_uring: Make remove_ifq_region a delayed work call
> 
>  include/linux/io_uring.h       |  35 ++
>  include/linux/io_uring_types.h |  11 +
>  include/linux/netdevice.h      |   6 +
>  include/linux/skbuff.h         |  10 +-
>  include/uapi/linux/io_uring.h  |  24 +
>  io_uring/Makefile              |   3 +-
>  io_uring/io_uring.c            |   8 +
>  io_uring/kbuf.c                |  13 +
>  io_uring/kbuf.h                |   2 +
>  io_uring/net.c                 | 123 +++++
>  io_uring/opdef.c               |  15 +
>  io_uring/zctap.c               | 842 +++++++++++++++++++++++++++++++++
>  io_uring/zctap.h               |  16 +
>  13 files changed, 1106 insertions(+), 2 deletions(-)
>  create mode 100644 io_uring/zctap.c
>  create mode 100644 io_uring/zctap.h
> 

Hi, Jonathan

We are interested in your work, too. I think the API is better than V1.
I have a question: Is this patchset still incomplete? We'd like to know how to
split msg header and body by XDP with io_uring ZC_RECV. Or could you please
share one runnable demo which could run with your previous liburing patch.

Regards,
Zhang.