Re: io_uring zero-copy send test results

[Pavel Begunkov <asml.silence@gmail.com>]

On 4/5/25 17:58, vitalif@yourcmc.ru wrote:
> Hi!
> 
> We ran some io_uring send-zerocopy tests with our colleagues by using the `send-zerocopy` utility from liburing examples (https://github.com/axboe/liburing/blob/master/examples/send-zerocopy.c).
> 
> And the results, especially with EPYC, were rather disappointing. :-(
> 
> The tests were run using `./send-zerocopy tcp -4 -R` at the server side and `time ./send-zerocopy tcp (-z 0)? (-b 0)? -4 -s 65435 -D 10.252.4.81` at the client side. 65435 was replaced by different buffer sizes.

fwiw, -z1 -b1 is the default, i.e. zc and fixed buffers

> Conclusion:
> - zerocopy send is beneficial for Xeon with at least 12 kb registered buffers and at least 16 kb normal buffers
> - worst thing is that with EPYCs, zerocopy send is slower than non-zerocopy in every single test... :-(
> 
> Profiling with perf shows that it spends most time in iommu related functions.
> 
> So I have a question: are these results expected? Or do I have to tune something to get better results?

Sounds like another case of iommu being painfully slow. The difference
is that while copying normal sends coalesce data into nice big contig
buffers, but zerocopy has to deal with whatever pages it's given. That's
32KB vs 4KB, and the worst case scenario you get 8x more frags (and skbs)
and 8x iommu mappings for zerocopy.

Try huge pages and see if it helps, it's -l1 in the benchmark. I can
also take a look at adding pre-mapped buffers again.

Perf profiles would also be useful to have if you can grab and post
them.

-- 
Pavel Begunkov