Re: io_uring zero-copy send test results

[Pavel Begunkov <asml.silence@gmail.com>]

On 4/8/25 13:43, vitalif@yourcmc.ru wrote:
>> What kernel version you use? I'm specifically interested whether it has:
>>
>> 6fe4220912d19 ("io_uring/notif: implement notification stacking")
>>
>> That would explain why it's slow even with huge pages.
> 
> It was Linux 6.8.12-4-pve (proxmox), so yeah, it didn't include that commit.
> 
> We repeated tests with Linux 6.11 also from proxmox:
> 
> AMD EPYC GENOA 9554 MELLANOX CX-5, iommu=pt, Linux 6.11
> 
> 4096 8192 10000 12000 16384 65435
> zc MB/s 2288 2422 2149 2396 2506 2476
> zc CPU 90% 67% 56% 56% 57% 44%
> send MB/s 1685 2033 2389 2343 2281 2415
> send CPU 95% 87% 49% 48% 62% 38%
> 
> AMD EPYC GENOA 9554 MELLANOX CX-5, iommu=pt, -l1, Linux 6.11
> 
> 4096 8192 10000 12000 16384 65435
> zc MB/s 2359 2509 2351 2508 2384 2424
> zc CPU 85% 58% 52% 45% 37% 18%
> send MB/s 1503 1892 2325 2447 2434 2440
> send CPU 99% 96% 50% 49% 57% 37%
> 
> Now it's nice and quick even without huge pages and even with 4k buffers!

Nice! Is ~2400 MB/s a hardware bottleneck? Seems like the t-put
converges to that, while I'd expect the gap to widen as we increase
the size to 64K.

>> That doesn't make sense. Do you see anything odd in the profile?
> 
> Didn't have time to repeat tests with perf on those servers yet, but I can check dmesg logs. In the default iommu mode, /sys/class/iommu is empty and dmesg includes the following lines:
> 
> DMAR-IR: IOAPIC id 8 under DRHD base  0x9b7fc000 IOMMU 9
> iommu: Default domain type: Translated
> iommu: DMA domain TLB invalidation policy: lazy mode
> 
> With iommu=pt, dmesg has:
> 
> DMAR-IR: IOAPIC id 8 under DRHD base  0x9b7fc000 IOMMU 9
> iommu: Default domain type: Passthrough (set via kernel command line)

-- 
Pavel Begunkov