Re: io_uring zero-copy send test results

[vitalif@yourcmc.ru]

> What kernel version you use? I'm specifically interested whether it has:
> 
> 6fe4220912d19 ("io_uring/notif: implement notification stacking")
> 
> That would explain why it's slow even with huge pages.

It was Linux 6.8.12-4-pve (proxmox), so yeah, it didn't include that commit.

We repeated tests with Linux 6.11 also from proxmox:

AMD EPYC GENOA 9554 MELLANOX CX-5, iommu=pt, Linux 6.11

4096 8192 10000 12000 16384 65435
zc MB/s 2288 2422 2149 2396 2506 2476
zc CPU 90% 67% 56% 56% 57% 44%
send MB/s 1685 2033 2389 2343 2281 2415
send CPU 95% 87% 49% 48% 62% 38%

AMD EPYC GENOA 9554 MELLANOX CX-5, iommu=pt, -l1, Linux 6.11

4096 8192 10000 12000 16384 65435
zc MB/s 2359 2509 2351 2508 2384 2424
zc CPU 85% 58% 52% 45% 37% 18%
send MB/s 1503 1892 2325 2447 2434 2440
send CPU 99% 96% 50% 49% 57% 37%

Now it's nice and quick even without huge pages and even with 4k buffers!

> That doesn't make sense. Do you see anything odd in the profile?

Didn't have time to repeat tests with perf on those servers yet, but I can check dmesg logs. In the default iommu mode, /sys/class/iommu is empty and dmesg includes the following lines:

DMAR-IR: IOAPIC id 8 under DRHD base  0x9b7fc000 IOMMU 9
iommu: Default domain type: Translated 
iommu: DMA domain TLB invalidation policy: lazy mode 

With iommu=pt, dmesg has:

DMAR-IR: IOAPIC id 8 under DRHD base  0x9b7fc000 IOMMU 9
iommu: Default domain type: Passthrough (set via kernel command line)