Re: [RFC v1 00/22] Large rx buffer support for zcrx

[Stanislav Fomichev <stfomichev@gmail.com>]

On 07/28, Pavel Begunkov wrote:
> On 7/28/25 21:21, Stanislav Fomichev wrote:
> > On 07/28, Pavel Begunkov wrote:
> > > On 7/28/25 18:13, Stanislav Fomichev wrote:
> ...>>> Supporting big buffers is the right direction, but I have the same
> > > > feedback:
> > > 
> > > Let me actually check the feedback for the queue config RFC...
> > > 
> > > it would be nice to fit a cohesive story for the devmem as well.
> > > 
> > > Only the last patch is zcrx specific, the rest is agnostic,
> > > devmem can absolutely reuse that. I don't think there are any
> > > issues wiring up devmem?
> > 
> > Right, but the patch number 2 exposes per-queue rx-buf-len which
> > I'm not sure is the right fit for devmem, see below. If all you
> 
> I guess you're talking about uapi setting it, because as an
> internal per queue parameter IMHO it does make sense for devmem.
> 
> > care is exposing it via io_uring, maybe don't expose it from netlink for
> 
> Sure, I can remove the set operation.
> 
> > now? Although I'm not sure I understand why you're also passing
> > this per-queue value via io_uring. Can you not inherit it from the
> > queue config?
> 
> It's not a great option. It complicates user space with netlink.
> And there are convenience configuration features in the future
> that requires io_uring to parse memory first. E.g. instead of
> user specifying a particular size, it can say "choose the largest
> length under 32K that the backing memory allows".

Don't you already need a bunch of netlink to setup rss and flow
steering? And if we end up adding queue api, you'll have to call that
one over netlink also.

> > > > We should also aim for another use-case where we allocate page pool
> > > > chunks from the huge page(s),
> > > 
> > > Separate huge page pool is a bit beyond the scope of this series.
> > > 
> > > this should push the perf even more.
> > > 
> > > And not sure about "even more" is from, you can already
> > > register a huge page with zcrx, and this will allow to chunk
> > > them to 32K or so for hardware. Is it in terms of applicability
> > > or you have some perf optimisation ideas?
> > 
> > What I'm looking for is a generic system-wide solution where we can
> > set up the host to use huge pages to back all (even non-zc) networking queues.
> > Not necessary needed, but might be an option to try.
> 
> Probably like what Jakub was once suggesting with the initial memory
> provider patch, got it.
> 
> > > > We need some way to express these things from the UAPI point of view.
> > > 
> > > Can you elaborate?
> > > 
> > > > Flipping the rx-buf-len value seems too fragile - there needs to be
> > > > something to request 32K chunks only for devmem case, not for the (default)
> > > > CPU memory. And the queues should go back to default 4K pages when the dmabuf
> > > > is detached from the queue.
> > > 
> > > That's what the per-queue config is solving. It's not default, zcrx
> > > configures it only for the specific queue it allocated, and the value
> > > is cleared on restart in netdev_rx_queue_restart(), if not even too
> > > aggressively. Maybe I should just stash it into mp_params to make
> > > sure it's not cleared if a provider is still attached on a spurious
> > > restart.
> > 
> > If we assume that at some point niov can be backed up by chunks larger
> > than PAGE_SIZE, the assumed workflow for devemem is:
> > 1. change rx-buf-len to 32K
> >    - this is needed only for devmem, but not for CPU RAM, but we'll have
> >      to refill the queues from the main memory anyway
> 
> Urgh, that's another reason why I prefer to just pass it through
> zcrx and not netlink. So maybe you can just pass the len to devmem
> on creation, and internally it sets up its queues with it.

But you still need to solve MAX_PAGE_ORDER/PAGE_ALLOC_COSTLY_ORDER I
think? We don't want the drivers to do PAGE_ALLOC_COSTLY_ORDER costly
allocation presumably?