From: Gal Pressman <[email protected]>
To: David Wei <[email protected]>, Jens Axboe <[email protected]>,
Pavel Begunkov <[email protected]>
Cc: [email protected], [email protected],
Mina Almasry <[email protected]>,
Jakub Kicinski <[email protected]>
Subject: Re: [RFC RESEND 00/11] Zero copy network RX using io_uring
Date: Sun, 22 Oct 2023 22:06:24 +0300 [thread overview]
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
On 26/08/2023 4:19, David Wei wrote:
> From: David Wei <[email protected]>
>
> This patchset is a proposal that adds zero copy network RX to io_uring.
> With it, userspace can register a region of host memory for receiving
> data directly from a NIC using DMA, without needing a kernel to user
> copy.
>
> Software support is added to the Broadcom BNXT driver. Hardware support
> for receive flow steering and header splitting is required.
>
> On the userspace side, a sample server is added in this branch of
> liburing:
> https://github.com/spikeh/liburing/tree/zcrx2
>
> Build liburing as normal, and run examples/zcrx. Then, set flow steering
> rules using ethtool. A sample shell script is included in
> examples/zcrx_flow.sh, but you need to change the source IP. Finally,
> connect a client using e.g. netcat and send data.
>
> This patchset + userspace code was tested on an Intel Xeon Platinum
> 8321HC CPU and Broadcom BCM57504 NIC.
>
> Early benchmarks using this prototype, with iperf3 as a load generator,
> showed a ~50% reduction in overall system memory bandwidth as measured
> using perf counters. Note that DDIO must be disabled on Intel systems.
>
> Mina et al. from Google and Kuba are collaborating on a similar proposal
> to ZC from NIC to devmem. There are many shared functionality in netdev
> that we can collaborate on e.g.:
> * Page pool memory provider backend and resource registration
> * Page pool refcounted iov/buf representation and lifecycle
> * Setting receive flow steering
>
> As mentioned earlier, this is an early prototype. It is brittle, some
> functionality is missing and there's little optimisation. We're looking
> for feedback on the overall approach and points of collaboration in
> netdev.
> * No copy fallback, if payload ends up in linear part of skb then the
> code will not work
> * No way to pin an RX queue to a specific CPU
> * Only one ifq, one pool region, on RX queue...
>
> This patchset is based on the work by Jonathan Lemon
> <[email protected]>:
> https://lore.kernel.org/io-uring/[email protected]/
Hello David,
This work looks interesting, is there anywhere I can read about it some
more? Maybe it was presented (and hopefully recorded) in a recent
conference?
Maybe something geared towards adding more drivers support?
I took a brief look at the bnxt patch and saw you converted the page
pool allocation to data pool allocation, I assume this is done for data
pages only, right? Headers are still allocated on page pool pages?
Thanks
next prev parent reply other threads:[~2023-10-22 19:06 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-08-26 1:19 [RFC RESEND 00/11] Zero copy network RX using io_uring David Wei
2023-08-26 1:19 ` [PATCH 01/11] io_uring: add interface queue David Wei
2023-08-26 1:19 ` [PATCH 02/11] io_uring: add mmap support for shared ifq ringbuffers David Wei
2023-08-26 1:19 ` [PATCH 03/11] netdev: add XDP_SETUP_ZC_RX command David Wei
2023-08-26 2:21 ` David Ahern
2023-08-26 21:37 ` David Wei
2023-08-26 1:19 ` [PATCH 04/11] io_uring: setup ZC for an RX queue when registering an ifq David Wei
2023-08-26 2:26 ` David Ahern
2023-08-26 22:00 ` David Wei
2023-08-26 1:19 ` [PATCH 05/11] io_uring: add ZC buf and pool David Wei
2023-08-26 1:19 ` [PATCH 06/11] io_uring: add ZC pool API David Wei
2023-08-26 1:19 ` [PATCH 07/11] skbuff: add SKBFL_FIXED_FRAG and skb_fixed() David Wei
2023-08-26 1:19 ` [PATCH 08/11] io_uring: allocate a uarg for freeing zero copy skbs David Wei
2023-08-26 1:19 ` [PATCH 09/11] io_uring: delay ZC pool destruction David Wei
2023-08-26 1:19 ` [PATCH 10/11] netdev/bnxt: add data pool and use it in BNXT driver David Wei
2023-08-26 1:19 ` [PATCH 11/11] io_uring: add io_recvzc request David Wei
2023-10-22 19:06 ` Gal Pressman [this message]
2023-10-23 3:35 ` [RFC RESEND 00/11] Zero copy network RX using io_uring David Wei
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox