io_uring zero-copy send test results

[vitalif@yourcmc.ru]

Hi!

We ran some io_uring send-zerocopy tests with our colleagues by using the `send-zerocopy` utility from liburing examples (https://github.com/axboe/liburing/blob/master/examples/send-zerocopy.c).

And the results, especially with EPYC, were rather disappointing. :-(

The tests were run using `./send-zerocopy tcp -4 -R` at the server side and `time ./send-zerocopy tcp (-z 0)? (-b 0)? -4 -s 65435 -D 10.252.4.81` at the client side. 65435 was replaced by different buffer sizes.

Conclusion:
- zerocopy send is beneficial for Xeon with at least 12 kb registered buffers and at least 16 kb normal buffers
- worst thing is that with EPYCs, zerocopy send is slower than non-zerocopy in every single test... :-(

Profiling with perf shows that it spends most time in iommu related functions.

So I have a question: are these results expected? Or do I have to tune something to get better results?

1) Xeon Gold 6330 + Mellanox ConnectX-6 DC

-b 1 (fixed buffers, default):

           4096  8192  10000  12000  16384  65435
zc MB/s    1673  2939  2926   2948   2946   2944
zc CPU     100%  80%   58%    43%    31%    14%
send MB/s  2946  2945  2949   2948   2948   2947
send CPU   80%   57%   52%    46%    44%    42%

-b 0:

           4096  8192  10000  12000  16384  65435
zc MB/s    1682  2940  2925   2934   2945   2923
zc CPU     99%   85%   71%    54%    38%    17%
send MB/s  2949  2947  2950   2945   2946   2949
send CPU   74%   55%   48%    47%    45%    39%

2) AMD EPYC GENOA 9554 + Mellanox ConnectX-5

-b 1:

           4096  8192  10000  12000  16384  65435
zc MB/s    864   1495  1646   1714   1790   2266
zc CPU     99%   93%   81%    86%    75%    57%
send MB/s  1799  2167  2265   2285   2248   2286
send CPU   90%   58%   54%    54%    52%    42%

-b 0:

           4096  8192  10000  12000  16384  65435
zc MB/s    778   1274  1476   1732   1798   2246
zc CPU     99%   89%   92%    81%    80%    54%
send MB/s  1791  2069  2239   2233   2194   2253
send CPU   88%   73%   55%    52%    59%    38%

3) AMD EPYC MILAN 7313 + Mellanox ConnectX-5

           4096  8192  10000  12000  16384  65435
zc MB/s    732   1130  1284   1247   1425   1713
zc CPU     99%   81%   82%    77%    62%    33%
send MB/s  1157  1522  1779   1720   1710   1684
send CPU   95%   58%   43%    39%    36%    27%

-- 
Vitaliy Filippov