Re: io_uring zero-copy send test results

[vitalif@yourcmc.ru]

Hi,

> Nice! Is ~2400 MB/s a hardware bottleneck? Seems like the t-put
> converges to that, while I'd expect the gap to widen as we increase
> the size to 64K.

I'm not sure, it doesn't seem so, these servers are connected with a 2x100G bonded connection. Iperf -P8 (8 threads) shows 100 Gbit/s...

>> Didn't have time to repeat tests with perf on those servers yet, but I can check dmesg logs. In the
>> default iommu mode, /sys/class/iommu is empty and dmesg includes the following lines:
>> DMAR-IR: IOAPIC id 8 under DRHD base 0x9b7fc000 IOMMU 9
>> iommu: Default domain type: Translated
>> iommu: DMA domain TLB invalidation policy: lazy mode
>> With iommu=pt, dmesg has:
>> DMAR-IR: IOAPIC id 8 under DRHD base 0x9b7fc000 IOMMU 9
>> iommu: Default domain type: Passthrough (set via kernel command line)

You're probably right, it was just a random mistake, I repeated the test again with iommu=pt and without it and the new results are very close to each other, and iommu=pt result is even slightly better. So never mind :)

Xeon Gold 6342 + Mellanox ConnectX-6 Dx + iommu=pt, run 2

           4096  8192  10000  12000  16384  32768  65435
zc MB/s    2681  2947  2927   2935   2949   2947   2945
zc CPU     99%   66%   41%    48%    22%    13%    11%
send MB/s  2950  2951  2950   2950   2949   2950   2949
send CPU   48%   35%   31%    34%    28%    29%    25%

Xeon Gold 6342 + Mellanox ConnectX-6 Dx, run 2

           4096  8192  10000  12000  16384  32768  65435
zc MB/s    2262  2948  2925   2935   2946   2947   2947
zc CPU     99%   52%   60%    44%    24%    17%    17%
send MB/s  2950  2949  2950   2950   2950   2950   2950
send CPU   48%   38%   36%    31%    33%    26%    29%