* io_uring networking performance degradation
@ 2021-04-19 9:13 Michael Stoler
2021-04-19 10:20 ` Pavel Begunkov
0 siblings, 1 reply; 7+ messages in thread
From: Michael Stoler @ 2021-04-19 9:13 UTC (permalink / raw)
To: io-uring
We are trying to reproduce reported on page
https://github.com/frevib/io_uring-echo-server/blob/master/benchmarks/benchmarks.md
results with a more realistic environment:
1. Internode networking in AWS cluster with i3.16xlarge nodes type(25
Gigabit network connection between client and server)
2. 128 and 2048 packet sizes, to simulate typical payloads
3. 10 clients to get 75-95% CPU utilization by server to simulate
server's normal load
4. 20 clients to get 100% CPU utilization by server to simulate
server's hard load
Software:
1. OS: Ubuntu 20.04.2 LTS HWE with 5.8.0-45-generic kernel with latest liburing
2. io_uring-echo-server: https://github.com/frevib/io_uring-echo-server
3. epoll-echo-server: https://github.com/frevib/epoll-echo-server
4. benchmark: https://github.com/haraldh/rust_echo_bench
5. all commands runs with "hwloc-bind os=eth1"
The results are confusing, epoll_echo_server shows stable advantage
over io_uring-echo-server, despite reported advantage of
io_uring-echo-server:
128 bytes packet size, 10 clients, 75-95% CPU core utilization by server:
echo_bench --address '172.22.117.67:7777' -c 10 -t 60 -l 128
epoll_echo_server: Speed: 80999 request/sec, 80999 response/sec
io_uring_echo_server: Speed: 74488 request/sec, 74488 response/sec
epoll_echo_server is 8% faster
128 bytes packet size, 20 clients, 100% CPU core utilization by server:
echo_bench --address '172.22.117.67:7777' -c 20 -t 60 -l 128
epoll_echo_server: Speed: 129063 request/sec, 129063 response/sec
io_uring_echo_server: Speed: 102681 request/sec, 102681 response/sec
epoll_echo_server is 25% faster
2048 bytes packet size, 10 clients, 75-95% CPU core utilization by server:
echo_bench --address '172.22.117.67:7777' -c 10 -t 60 -l 2048
epoll_echo_server: Speed: 74421 request/sec, 74421 response/sec
io_uring_echo_server: Speed: 66510 request/sec, 66510 response/sec
epoll_echo_server is 11% faster
2048 bytes packet size, 20 clients, 100% CPU core utilization by server:
echo_bench --address '172.22.117.67:7777' -c 20 -t 60 -l 2048
epoll_echo_server: Speed: 108704 request/sec, 108704 response/sec
io_uring_echo_server: Speed: 85536 request/sec, 85536 response/sec
epoll_echo_server is 27% faster
Why io_uring shows consistent performance degradation? What is going wrong?
Regards
Michael Stoler
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: io_uring networking performance degradation 2021-04-19 9:13 io_uring networking performance degradation Michael Stoler @ 2021-04-19 10:20 ` Pavel Begunkov 2021-04-19 14:27 ` Michael Stoler 0 siblings, 1 reply; 7+ messages in thread From: Pavel Begunkov @ 2021-04-19 10:20 UTC (permalink / raw) To: Michael Stoler, io-uring, Jens Axboe On 4/19/21 10:13 AM, Michael Stoler wrote: > We are trying to reproduce reported on page > https://github.com/frevib/io_uring-echo-server/blob/master/benchmarks/benchmarks.md > results with a more realistic environment: > 1. Internode networking in AWS cluster with i3.16xlarge nodes type(25 > Gigabit network connection between client and server) > 2. 128 and 2048 packet sizes, to simulate typical payloads > 3. 10 clients to get 75-95% CPU utilization by server to simulate > server's normal load > 4. 20 clients to get 100% CPU utilization by server to simulate > server's hard load > > Software: > 1. OS: Ubuntu 20.04.2 LTS HWE with 5.8.0-45-generic kernel with latest liburing > 2. io_uring-echo-server: https://github.com/frevib/io_uring-echo-server > 3. epoll-echo-server: https://github.com/frevib/epoll-echo-server > 4. benchmark: https://github.com/haraldh/rust_echo_bench > 5. all commands runs with "hwloc-bind os=eth1" > > The results are confusing, epoll_echo_server shows stable advantage > over io_uring-echo-server, despite reported advantage of > io_uring-echo-server: > > 128 bytes packet size, 10 clients, 75-95% CPU core utilization by server: > echo_bench --address '172.22.117.67:7777' -c 10 -t 60 -l 128 > epoll_echo_server: Speed: 80999 request/sec, 80999 response/sec > io_uring_echo_server: Speed: 74488 request/sec, 74488 response/sec > epoll_echo_server is 8% faster > > 128 bytes packet size, 20 clients, 100% CPU core utilization by server: > echo_bench --address '172.22.117.67:7777' -c 20 -t 60 -l 128 > epoll_echo_server: Speed: 129063 request/sec, 129063 response/sec > io_uring_echo_server: Speed: 102681 request/sec, 102681 response/sec > epoll_echo_server is 25% faster > > 2048 bytes packet size, 10 clients, 75-95% CPU core utilization by server: > echo_bench --address '172.22.117.67:7777' -c 10 -t 60 -l 2048 > epoll_echo_server: Speed: 74421 request/sec, 74421 response/sec > io_uring_echo_server: Speed: 66510 request/sec, 66510 response/sec > epoll_echo_server is 11% faster > > 2048 bytes packet size, 20 clients, 100% CPU core utilization by server: > echo_bench --address '172.22.117.67:7777' -c 20 -t 60 -l 2048 > epoll_echo_server: Speed: 108704 request/sec, 108704 response/sec > io_uring_echo_server: Speed: 85536 request/sec, 85536 response/sec > epoll_echo_server is 27% faster > > Why io_uring shows consistent performance degradation? What is going wrong? 5.8 is pretty old, and I'm not sure all the performance problems were addressed there. Apart from missing common optimisations as you may have seen in the thread, it looks to me it doesn't have sighd(?) lock hammering fix. Jens, knows better it has been backported or not. So, things you can do: 1) try out 5.12 2) attach `perf top` output or some other profiling for your 5.8 3) to have a more complete picture do 2) with 5.12 Let's find what's gone wrong -- Pavel Begunkov ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: io_uring networking performance degradation 2021-04-19 10:20 ` Pavel Begunkov @ 2021-04-19 14:27 ` Michael Stoler 2021-04-20 10:44 ` Michael Stoler 0 siblings, 1 reply; 7+ messages in thread From: Michael Stoler @ 2021-04-19 14:27 UTC (permalink / raw) To: Pavel Begunkov; +Cc: io-uring, Jens Axboe 1) linux-5.12-rc8 shows generally same picture: average load, 70-85% CPU core usage, 128 bytes packets echo_bench --address '172.22.150.170:7777' --number 10 --duration 60 --length 128` epoll-echo-server: Speed: 71513 request/sec, 71513 response/sec io_uring_echo_server: Speed: 64091 request/sec, 64091 response/sec epoll-echo-server is 11% faster high load, 95-100% CPU core usage, 128 bytes packets echo_bench --address '172.22.150.170:7777' --number 20 --duration 60 --length 128` epoll-echo-server: Speed: 130186 request/sec, 130186 response/sec io_uring_echo_server: Speed: 109793 request/sec, 109793 response/sec epoll-echo-server is 18% faster average load, 70-85% CPU core usage, 2048 bytes packets echo_bench --address '172.22.150.170:7777' --number 10 --duration 60 --length 2048` epoll-echo-server: Speed: 63082 request/sec, 63082 response/sec io_uring_echo_server: Speed: 59449 request/sec, 59449 response/sec epoll-echo-server is 6% faster high load, 95-100% CPU core usage, 2048 bytes packets echo_bench --address '172.22.150.170:7777' --number 20 --duration 60 --length 2048` epoll-echo-server: Speed: 110402 request/sec, 110402 response/sec io_uring_echo_server: Speed: 88718 request/sec, 88718 response/sec epoll-echo-server is 24% faster 2-3) The "perf top" doesn't work stable with Ubuntu over AWS. All the time it shows errors: "Uhhuh. NMI received for unknown reason", "Do you have a strange power saving mode enabled?", "Dazed and confused, but trying to continue". Regards Michael Stoler On Mon, Apr 19, 2021 at 1:20 PM Pavel Begunkov <[email protected]> wrote: > > On 4/19/21 10:13 AM, Michael Stoler wrote: > > We are trying to reproduce reported on page > > https://github.com/frevib/io_uring-echo-server/blob/master/benchmarks/benchmarks.md > > results with a more realistic environment: > > 1. Internode networking in AWS cluster with i3.16xlarge nodes type(25 > > Gigabit network connection between client and server) > > 2. 128 and 2048 packet sizes, to simulate typical payloads > > 3. 10 clients to get 75-95% CPU utilization by server to simulate > > server's normal load > > 4. 20 clients to get 100% CPU utilization by server to simulate > > server's hard load > > > > Software: > > 1. OS: Ubuntu 20.04.2 LTS HWE with 5.8.0-45-generic kernel with latest liburing > > 2. io_uring-echo-server: https://github.com/frevib/io_uring-echo-server > > 3. epoll-echo-server: https://github.com/frevib/epoll-echo-server > > 4. benchmark: https://github.com/haraldh/rust_echo_bench > > 5. all commands runs with "hwloc-bind os=eth1" > > > > The results are confusing, epoll_echo_server shows stable advantage > > over io_uring-echo-server, despite reported advantage of > > io_uring-echo-server: > > > > 128 bytes packet size, 10 clients, 75-95% CPU core utilization by server: > > echo_bench --address '172.22.117.67:7777' -c 10 -t 60 -l 128 > > epoll_echo_server: Speed: 80999 request/sec, 80999 response/sec > > io_uring_echo_server: Speed: 74488 request/sec, 74488 response/sec > > epoll_echo_server is 8% faster > > > > 128 bytes packet size, 20 clients, 100% CPU core utilization by server: > > echo_bench --address '172.22.117.67:7777' -c 20 -t 60 -l 128 > > epoll_echo_server: Speed: 129063 request/sec, 129063 response/sec > > io_uring_echo_server: Speed: 102681 request/sec, 102681 response/sec > > epoll_echo_server is 25% faster > > > > 2048 bytes packet size, 10 clients, 75-95% CPU core utilization by server: > > echo_bench --address '172.22.117.67:7777' -c 10 -t 60 -l 2048 > > epoll_echo_server: Speed: 74421 request/sec, 74421 response/sec > > io_uring_echo_server: Speed: 66510 request/sec, 66510 response/sec > > epoll_echo_server is 11% faster > > > > 2048 bytes packet size, 20 clients, 100% CPU core utilization by server: > > echo_bench --address '172.22.117.67:7777' -c 20 -t 60 -l 2048 > > epoll_echo_server: Speed: 108704 request/sec, 108704 response/sec > > io_uring_echo_server: Speed: 85536 request/sec, 85536 response/sec > > epoll_echo_server is 27% faster > > > > Why io_uring shows consistent performance degradation? What is going wrong? > > 5.8 is pretty old, and I'm not sure all the performance problems were > addressed there. Apart from missing common optimisations as you may > have seen in the thread, it looks to me it doesn't have sighd(?) lock > hammering fix. Jens, knows better it has been backported or not. > > So, things you can do: > 1) try out 5.12 > 2) attach `perf top` output or some other profiling for your 5.8 > 3) to have a more complete picture do 2) with 5.12 > > Let's find what's gone wrong > > -- > Pavel Begunkov -- Michael Stoler ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: io_uring networking performance degradation 2021-04-19 14:27 ` Michael Stoler @ 2021-04-20 10:44 ` Michael Stoler 2021-04-25 9:52 ` Michael Stoler 0 siblings, 1 reply; 7+ messages in thread From: Michael Stoler @ 2021-04-20 10:44 UTC (permalink / raw) To: Pavel Begunkov; +Cc: io-uring, Jens Axboe Hi, perf data and tops for linux-5.8 are here: http://rdxdownloads.rdxdyn.com/michael_stoler_perf_data.tgz Regards Michael Stoler On Mon, Apr 19, 2021 at 5:27 PM Michael Stoler <[email protected]> wrote: > > 1) linux-5.12-rc8 shows generally same picture: > > average load, 70-85% CPU core usage, 128 bytes packets > echo_bench --address '172.22.150.170:7777' --number 10 --duration > 60 --length 128` > epoll-echo-server: Speed: 71513 request/sec, 71513 response/sec > io_uring_echo_server: Speed: 64091 request/sec, 64091 response/sec > epoll-echo-server is 11% faster > > high load, 95-100% CPU core usage, 128 bytes packets > echo_bench --address '172.22.150.170:7777' --number 20 --duration > 60 --length 128` > epoll-echo-server: Speed: 130186 request/sec, 130186 response/sec > io_uring_echo_server: Speed: 109793 request/sec, 109793 response/sec > epoll-echo-server is 18% faster > > average load, 70-85% CPU core usage, 2048 bytes packets > echo_bench --address '172.22.150.170:7777' --number 10 --duration > 60 --length 2048` > epoll-echo-server: Speed: 63082 request/sec, 63082 response/sec > io_uring_echo_server: Speed: 59449 request/sec, 59449 response/sec > epoll-echo-server is 6% faster > > high load, 95-100% CPU core usage, 2048 bytes packets > echo_bench --address '172.22.150.170:7777' --number 20 --duration > 60 --length 2048` > epoll-echo-server: Speed: 110402 request/sec, 110402 response/sec > io_uring_echo_server: Speed: 88718 request/sec, 88718 response/sec > epoll-echo-server is 24% faster > > > 2-3) The "perf top" doesn't work stable with Ubuntu over AWS. All the > time it shows errors: "Uhhuh. NMI received for unknown reason", "Do > you have a strange power saving mode enabled?", "Dazed and confused, > but trying to continue". > > Regards > Michael Stoler > > On Mon, Apr 19, 2021 at 1:20 PM Pavel Begunkov <[email protected]> wrote: > > > > On 4/19/21 10:13 AM, Michael Stoler wrote: > > > We are trying to reproduce reported on page > > > https://github.com/frevib/io_uring-echo-server/blob/master/benchmarks/benchmarks.md > > > results with a more realistic environment: > > > 1. Internode networking in AWS cluster with i3.16xlarge nodes type(25 > > > Gigabit network connection between client and server) > > > 2. 128 and 2048 packet sizes, to simulate typical payloads > > > 3. 10 clients to get 75-95% CPU utilization by server to simulate > > > server's normal load > > > 4. 20 clients to get 100% CPU utilization by server to simulate > > > server's hard load > > > > > > Software: > > > 1. OS: Ubuntu 20.04.2 LTS HWE with 5.8.0-45-generic kernel with latest liburing > > > 2. io_uring-echo-server: https://github.com/frevib/io_uring-echo-server > > > 3. epoll-echo-server: https://github.com/frevib/epoll-echo-server > > > 4. benchmark: https://github.com/haraldh/rust_echo_bench > > > 5. all commands runs with "hwloc-bind os=eth1" > > > > > > The results are confusing, epoll_echo_server shows stable advantage > > > over io_uring-echo-server, despite reported advantage of > > > io_uring-echo-server: > > > > > > 128 bytes packet size, 10 clients, 75-95% CPU core utilization by server: > > > echo_bench --address '172.22.117.67:7777' -c 10 -t 60 -l 128 > > > epoll_echo_server: Speed: 80999 request/sec, 80999 response/sec > > > io_uring_echo_server: Speed: 74488 request/sec, 74488 response/sec > > > epoll_echo_server is 8% faster > > > > > > 128 bytes packet size, 20 clients, 100% CPU core utilization by server: > > > echo_bench --address '172.22.117.67:7777' -c 20 -t 60 -l 128 > > > epoll_echo_server: Speed: 129063 request/sec, 129063 response/sec > > > io_uring_echo_server: Speed: 102681 request/sec, 102681 response/sec > > > epoll_echo_server is 25% faster > > > > > > 2048 bytes packet size, 10 clients, 75-95% CPU core utilization by server: > > > echo_bench --address '172.22.117.67:7777' -c 10 -t 60 -l 2048 > > > epoll_echo_server: Speed: 74421 request/sec, 74421 response/sec > > > io_uring_echo_server: Speed: 66510 request/sec, 66510 response/sec > > > epoll_echo_server is 11% faster > > > > > > 2048 bytes packet size, 20 clients, 100% CPU core utilization by server: > > > echo_bench --address '172.22.117.67:7777' -c 20 -t 60 -l 2048 > > > epoll_echo_server: Speed: 108704 request/sec, 108704 response/sec > > > io_uring_echo_server: Speed: 85536 request/sec, 85536 response/sec > > > epoll_echo_server is 27% faster > > > > > > Why io_uring shows consistent performance degradation? What is going wrong? > > > > 5.8 is pretty old, and I'm not sure all the performance problems were > > addressed there. Apart from missing common optimisations as you may > > have seen in the thread, it looks to me it doesn't have sighd(?) lock > > hammering fix. Jens, knows better it has been backported or not. > > > > So, things you can do: > > 1) try out 5.12 > > 2) attach `perf top` output or some other profiling for your 5.8 > > 3) to have a more complete picture do 2) with 5.12 > > > > Let's find what's gone wrong > > > > -- > > Pavel Begunkov > > > > -- > Michael Stoler -- Michael Stoler ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: io_uring networking performance degradation 2021-04-20 10:44 ` Michael Stoler @ 2021-04-25 9:52 ` Michael Stoler 2021-04-26 11:49 ` Pavel Begunkov 0 siblings, 1 reply; 7+ messages in thread From: Michael Stoler @ 2021-04-25 9:52 UTC (permalink / raw) To: Pavel Begunkov; +Cc: io-uring, Jens Axboe Because of unstable working of perf over AWS VM I recheck test on physical machine: Ubuntu 20.04, 5.8.0-50-generic kernel, CPU AMD EPYC 7272 12-Core Processor 3200MHz, BogoMIPS 5789.39, NIC melanox 5, Speed: 25000Mb/s Full Duplex. Over physical machine performance degradation is much less pronounced but still exists: io_uring-echo-server Speed: 143081 request/sec, 143081 response/sec epoll-echo-server Speed: 150692 request/sec, 150692 response/sec epoll-echo-server is 5% faster "perf top" with io_uring-echo-server: PerfTop: 16481 irqs/sec kernel:98.5% exact: 99.8% lost: 0/0 drop: 0/0 [4000Hz cycles], (all, 24 CPUs) ------------------------------------------------------------------------------- 8.66% [kernel] [k] __x86_indirect_thunk_rax 8.49% [kernel] [k] copy_user_generic_string 5.57% [kernel] [k] memset 2.81% [kernel] [k] tcp_rate_skb_sent 2.32% [kernel] [k] __alloc_skb 2.16% [kernel] [k] __check_object_size 1.44% [unknown] [k] 0xffffffffc100c296 1.28% [kernel] [k] tcp_write_xmit 1.22% [kernel] [k] iommu_dma_map_page 1.16% [kernel] [k] kmem_cache_free 1.14% [kernel] [k] __softirqentry_text_start 1.06% [unknown] [k] 0xffffffffc1008a7e 1.03% [kernel] [k] __skb_datagram_iter 0.97% [kernel] [k] __dev_queue_xmit 0.86% [kernel] [k] ipv4_mtu 0.85% [kernel] [k] tcp_schedule_loss_probe 0.80% [kernel] [k] tcp_release_cb 0.78% [unknown] [k] 0xffffffffc100c290 0.77% [unknown] [k] 0xffffffffc100c295 0.76% perf [.] __symbols__insert "perf top" with epoll-echo-server: PerfTop: 24255 irqs/sec kernel:98.3% exact: 99.6% lost: 0/0 drop: 0/0 [4000Hz cycles], (all, 24 CPUs) ------------------------------------------------------------------------------- 8.77% [kernel] [k] __x86_indirect_thunk_rax 7.50% [kernel] [k] copy_user_generic_string 4.10% [kernel] [k] memset 2.70% [kernel] [k] tcp_rate_skb_sent 2.18% [kernel] [k] __check_object_size 2.09% [kernel] [k] __alloc_skb 1.61% [unknown] [k] 0xffffffffc100c296 1.47% [kernel] [k] __virt_addr_valid 1.40% [kernel] [k] iommu_dma_map_page 1.37% [unknown] [k] 0xffffffffc1008a7e 1.22% [kernel] [k] tcp_poll 1.16% [kernel] [k] __softirqentry_text_start 1.15% [kernel] [k] tcp_stream_memory_free 1.07% [kernel] [k] tcp_write_xmit 1.06% [kernel] [k] kmem_cache_free 1.03% [kernel] [k] tcp_release_cb 0.96% [kernel] [k] syscall_return_via_sysret 0.90% [kernel] [k] __lock_text_start 0.82% [kernel] [k] __copy_skb_header 0.81% [kernel] [k] amd_iommu_map Regards Michael Stoler On Tue, Apr 20, 2021 at 1:44 PM Michael Stoler <[email protected]> wrote: > > Hi, perf data and tops for linux-5.8 are here: > http://rdxdownloads.rdxdyn.com/michael_stoler_perf_data.tgz > > Regards > Michael Stoler > > On Mon, Apr 19, 2021 at 5:27 PM Michael Stoler <[email protected]> wrote: > > > > 1) linux-5.12-rc8 shows generally same picture: > > > > average load, 70-85% CPU core usage, 128 bytes packets > > echo_bench --address '172.22.150.170:7777' --number 10 --duration > > 60 --length 128` > > epoll-echo-server: Speed: 71513 request/sec, 71513 response/sec > > io_uring_echo_server: Speed: 64091 request/sec, 64091 response/sec > > epoll-echo-server is 11% faster > > > > high load, 95-100% CPU core usage, 128 bytes packets > > echo_bench --address '172.22.150.170:7777' --number 20 --duration > > 60 --length 128` > > epoll-echo-server: Speed: 130186 request/sec, 130186 response/sec > > io_uring_echo_server: Speed: 109793 request/sec, 109793 response/sec > > epoll-echo-server is 18% faster > > > > average load, 70-85% CPU core usage, 2048 bytes packets > > echo_bench --address '172.22.150.170:7777' --number 10 --duration > > 60 --length 2048` > > epoll-echo-server: Speed: 63082 request/sec, 63082 response/sec > > io_uring_echo_server: Speed: 59449 request/sec, 59449 response/sec > > epoll-echo-server is 6% faster > > > > high load, 95-100% CPU core usage, 2048 bytes packets > > echo_bench --address '172.22.150.170:7777' --number 20 --duration > > 60 --length 2048` > > epoll-echo-server: Speed: 110402 request/sec, 110402 response/sec > > io_uring_echo_server: Speed: 88718 request/sec, 88718 response/sec > > epoll-echo-server is 24% faster > > > > > > 2-3) The "perf top" doesn't work stable with Ubuntu over AWS. All the > > time it shows errors: "Uhhuh. NMI received for unknown reason", "Do > > you have a strange power saving mode enabled?", "Dazed and confused, > > but trying to continue". > > > > Regards > > Michael Stoler > > > > On Mon, Apr 19, 2021 at 1:20 PM Pavel Begunkov <[email protected]> wrote: > > > > > > On 4/19/21 10:13 AM, Michael Stoler wrote: > > > > We are trying to reproduce reported on page > > > > https://github.com/frevib/io_uring-echo-server/blob/master/benchmarks/benchmarks.md > > > > results with a more realistic environment: > > > > 1. Internode networking in AWS cluster with i3.16xlarge nodes type(25 > > > > Gigabit network connection between client and server) > > > > 2. 128 and 2048 packet sizes, to simulate typical payloads > > > > 3. 10 clients to get 75-95% CPU utilization by server to simulate > > > > server's normal load > > > > 4. 20 clients to get 100% CPU utilization by server to simulate > > > > server's hard load > > > > > > > > Software: > > > > 1. OS: Ubuntu 20.04.2 LTS HWE with 5.8.0-45-generic kernel with latest liburing > > > > 2. io_uring-echo-server: https://github.com/frevib/io_uring-echo-server > > > > 3. epoll-echo-server: https://github.com/frevib/epoll-echo-server > > > > 4. benchmark: https://github.com/haraldh/rust_echo_bench > > > > 5. all commands runs with "hwloc-bind os=eth1" > > > > > > > > The results are confusing, epoll_echo_server shows stable advantage > > > > over io_uring-echo-server, despite reported advantage of > > > > io_uring-echo-server: > > > > > > > > 128 bytes packet size, 10 clients, 75-95% CPU core utilization by server: > > > > echo_bench --address '172.22.117.67:7777' -c 10 -t 60 -l 128 > > > > epoll_echo_server: Speed: 80999 request/sec, 80999 response/sec > > > > io_uring_echo_server: Speed: 74488 request/sec, 74488 response/sec > > > > epoll_echo_server is 8% faster > > > > > > > > 128 bytes packet size, 20 clients, 100% CPU core utilization by server: > > > > echo_bench --address '172.22.117.67:7777' -c 20 -t 60 -l 128 > > > > epoll_echo_server: Speed: 129063 request/sec, 129063 response/sec > > > > io_uring_echo_server: Speed: 102681 request/sec, 102681 response/sec > > > > epoll_echo_server is 25% faster > > > > > > > > 2048 bytes packet size, 10 clients, 75-95% CPU core utilization by server: > > > > echo_bench --address '172.22.117.67:7777' -c 10 -t 60 -l 2048 > > > > epoll_echo_server: Speed: 74421 request/sec, 74421 response/sec > > > > io_uring_echo_server: Speed: 66510 request/sec, 66510 response/sec > > > > epoll_echo_server is 11% faster > > > > > > > > 2048 bytes packet size, 20 clients, 100% CPU core utilization by server: > > > > echo_bench --address '172.22.117.67:7777' -c 20 -t 60 -l 2048 > > > > epoll_echo_server: Speed: 108704 request/sec, 108704 response/sec > > > > io_uring_echo_server: Speed: 85536 request/sec, 85536 response/sec > > > > epoll_echo_server is 27% faster > > > > > > > > Why io_uring shows consistent performance degradation? What is going wrong? > > > > > > 5.8 is pretty old, and I'm not sure all the performance problems were > > > addressed there. Apart from missing common optimisations as you may > > > have seen in the thread, it looks to me it doesn't have sighd(?) lock > > > hammering fix. Jens, knows better it has been backported or not. > > > > > > So, things you can do: > > > 1) try out 5.12 > > > 2) attach `perf top` output or some other profiling for your 5.8 > > > 3) to have a more complete picture do 2) with 5.12 > > > > > > Let's find what's gone wrong > > > > > > -- > > > Pavel Begunkov > > > > > > > > -- > > Michael Stoler > > > > -- > Michael Stoler -- Michael Stoler ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: io_uring networking performance degradation 2021-04-25 9:52 ` Michael Stoler @ 2021-04-26 11:49 ` Pavel Begunkov 2021-04-26 13:07 ` Michael Stoler 0 siblings, 1 reply; 7+ messages in thread From: Pavel Begunkov @ 2021-04-26 11:49 UTC (permalink / raw) To: Michael Stoler; +Cc: io-uring, Jens Axboe On 4/25/21 10:52 AM, Michael Stoler wrote: > Because of unstable working of perf over AWS VM I recheck test on > physical machine: Ubuntu 20.04, 5.8.0-50-generic kernel, CPU AMD EPYC > 7272 12-Core Processor 3200MHz, BogoMIPS 5789.39, NIC melanox 5, > Speed: 25000Mb/s Full Duplex. > Over physical machine performance degradation is much less pronounced > but still exists: > io_uring-echo-server Speed: 143081 request/sec, 143081 response/sec > epoll-echo-server Speed: 150692 request/sec, 150692 response/sec > epoll-echo-server is 5% faster Have to note that I haven't check the userspace programs, so not sure it's a fair comparison (may be or may be not). So, with it being said: 1) The last report had lot of idle time, so it may be a question of latency but not throughput for it. 2) Did you do proper pinning to a CPU/core? taskset or cset? Also, did it saturate the CPU/core you used in the most recent post? 3) Looking at __skb_datagram_iter taking 1%, seems there are other tasks taking a relatively good share of CPU/NIC resources. What is this datagram? UDP on the same NIC? Is something else using your NIC/interface? 4) don't see even close anything related to io_uring in the recent run, and it was only a small fraction in previous ones. So it's definitely not the overhead on submit/complete. If there is a io_uring problem, it could be the difference in polling / iowq punting comparing with epoll. It may be interesting to look into. And related thing I'm curious about is to compare FAST_POLL requests with io_uring multi-shot polling + send/recv. > > "perf top" with io_uring-echo-server: > PerfTop: 16481 irqs/sec kernel:98.5% exact: 99.8% lost: 0/0 drop: > 0/0 [4000Hz cycles], (all, 24 CPUs) > ------------------------------------------------------------------------------- > 8.66% [kernel] [k] __x86_indirect_thunk_rax > 8.49% [kernel] [k] copy_user_generic_string > 5.57% [kernel] [k] memset > 2.81% [kernel] [k] tcp_rate_skb_sent > 2.32% [kernel] [k] __alloc_skb > 2.16% [kernel] [k] __check_object_size > 1.44% [unknown] [k] 0xffffffffc100c296 > 1.28% [kernel] [k] tcp_write_xmit > 1.22% [kernel] [k] iommu_dma_map_page > 1.16% [kernel] [k] kmem_cache_free > 1.14% [kernel] [k] __softirqentry_text_start > 1.06% [unknown] [k] 0xffffffffc1008a7e > 1.03% [kernel] [k] __skb_datagram_iter > 0.97% [kernel] [k] __dev_queue_xmit > 0.86% [kernel] [k] ipv4_mtu > 0.85% [kernel] [k] tcp_schedule_loss_probe > 0.80% [kernel] [k] tcp_release_cb > 0.78% [unknown] [k] 0xffffffffc100c290 > 0.77% [unknown] [k] 0xffffffffc100c295 > 0.76% perf [.] __symbols__insert > > "perf top" with epoll-echo-server: > PerfTop: 24255 irqs/sec kernel:98.3% exact: 99.6% lost: 0/0 drop: > 0/0 [4000Hz cycles], (all, 24 CPUs) > ------------------------------------------------------------------------------- > 8.77% [kernel] [k] __x86_indirect_thunk_rax > 7.50% [kernel] [k] copy_user_generic_string > 4.10% [kernel] [k] memset > 2.70% [kernel] [k] tcp_rate_skb_sent > 2.18% [kernel] [k] __check_object_size > 2.09% [kernel] [k] __alloc_skb > 1.61% [unknown] [k] 0xffffffffc100c296 > 1.47% [kernel] [k] __virt_addr_valid > 1.40% [kernel] [k] iommu_dma_map_page > 1.37% [unknown] [k] 0xffffffffc1008a7e > 1.22% [kernel] [k] tcp_poll > 1.16% [kernel] [k] __softirqentry_text_start > 1.15% [kernel] [k] tcp_stream_memory_free > 1.07% [kernel] [k] tcp_write_xmit > 1.06% [kernel] [k] kmem_cache_free > 1.03% [kernel] [k] tcp_release_cb > 0.96% [kernel] [k] syscall_return_via_sysret > 0.90% [kernel] [k] __lock_text_start > 0.82% [kernel] [k] __copy_skb_header > 0.81% [kernel] [k] amd_iommu_map > > Regards > Michael Stoler > > On Tue, Apr 20, 2021 at 1:44 PM Michael Stoler <[email protected]> wrote: >> >> Hi, perf data and tops for linux-5.8 are here: >> http://rdxdownloads.rdxdyn.com/michael_stoler_perf_data.tgz >> >> Regards >> Michael Stoler >> >> On Mon, Apr 19, 2021 at 5:27 PM Michael Stoler <[email protected]> wrote: >>> >>> 1) linux-5.12-rc8 shows generally same picture: >>> >>> average load, 70-85% CPU core usage, 128 bytes packets >>> echo_bench --address '172.22.150.170:7777' --number 10 --duration >>> 60 --length 128` >>> epoll-echo-server: Speed: 71513 request/sec, 71513 response/sec >>> io_uring_echo_server: Speed: 64091 request/sec, 64091 response/sec >>> epoll-echo-server is 11% faster >>> >>> high load, 95-100% CPU core usage, 128 bytes packets >>> echo_bench --address '172.22.150.170:7777' --number 20 --duration >>> 60 --length 128` >>> epoll-echo-server: Speed: 130186 request/sec, 130186 response/sec >>> io_uring_echo_server: Speed: 109793 request/sec, 109793 response/sec >>> epoll-echo-server is 18% faster >>> >>> average load, 70-85% CPU core usage, 2048 bytes packets >>> echo_bench --address '172.22.150.170:7777' --number 10 --duration >>> 60 --length 2048` >>> epoll-echo-server: Speed: 63082 request/sec, 63082 response/sec >>> io_uring_echo_server: Speed: 59449 request/sec, 59449 response/sec >>> epoll-echo-server is 6% faster >>> >>> high load, 95-100% CPU core usage, 2048 bytes packets >>> echo_bench --address '172.22.150.170:7777' --number 20 --duration >>> 60 --length 2048` >>> epoll-echo-server: Speed: 110402 request/sec, 110402 response/sec >>> io_uring_echo_server: Speed: 88718 request/sec, 88718 response/sec >>> epoll-echo-server is 24% faster >>> >>> >>> 2-3) The "perf top" doesn't work stable with Ubuntu over AWS. All the >>> time it shows errors: "Uhhuh. NMI received for unknown reason", "Do >>> you have a strange power saving mode enabled?", "Dazed and confused, >>> but trying to continue". >>> >>> Regards >>> Michael Stoler >>> >>> On Mon, Apr 19, 2021 at 1:20 PM Pavel Begunkov <[email protected]> wrote: >>>> >>>> On 4/19/21 10:13 AM, Michael Stoler wrote: >>>>> We are trying to reproduce reported on page >>>>> https://github.com/frevib/io_uring-echo-server/blob/master/benchmarks/benchmarks.md >>>>> results with a more realistic environment: >>>>> 1. Internode networking in AWS cluster with i3.16xlarge nodes type(25 >>>>> Gigabit network connection between client and server) >>>>> 2. 128 and 2048 packet sizes, to simulate typical payloads >>>>> 3. 10 clients to get 75-95% CPU utilization by server to simulate >>>>> server's normal load >>>>> 4. 20 clients to get 100% CPU utilization by server to simulate >>>>> server's hard load >>>>> >>>>> Software: >>>>> 1. OS: Ubuntu 20.04.2 LTS HWE with 5.8.0-45-generic kernel with latest liburing >>>>> 2. io_uring-echo-server: https://github.com/frevib/io_uring-echo-server >>>>> 3. epoll-echo-server: https://github.com/frevib/epoll-echo-server >>>>> 4. benchmark: https://github.com/haraldh/rust_echo_bench >>>>> 5. all commands runs with "hwloc-bind os=eth1" >>>>> >>>>> The results are confusing, epoll_echo_server shows stable advantage >>>>> over io_uring-echo-server, despite reported advantage of >>>>> io_uring-echo-server: >>>>> >>>>> 128 bytes packet size, 10 clients, 75-95% CPU core utilization by server: >>>>> echo_bench --address '172.22.117.67:7777' -c 10 -t 60 -l 128 >>>>> epoll_echo_server: Speed: 80999 request/sec, 80999 response/sec >>>>> io_uring_echo_server: Speed: 74488 request/sec, 74488 response/sec >>>>> epoll_echo_server is 8% faster >>>>> >>>>> 128 bytes packet size, 20 clients, 100% CPU core utilization by server: >>>>> echo_bench --address '172.22.117.67:7777' -c 20 -t 60 -l 128 >>>>> epoll_echo_server: Speed: 129063 request/sec, 129063 response/sec >>>>> io_uring_echo_server: Speed: 102681 request/sec, 102681 response/sec >>>>> epoll_echo_server is 25% faster >>>>> >>>>> 2048 bytes packet size, 10 clients, 75-95% CPU core utilization by server: >>>>> echo_bench --address '172.22.117.67:7777' -c 10 -t 60 -l 2048 >>>>> epoll_echo_server: Speed: 74421 request/sec, 74421 response/sec >>>>> io_uring_echo_server: Speed: 66510 request/sec, 66510 response/sec >>>>> epoll_echo_server is 11% faster >>>>> >>>>> 2048 bytes packet size, 20 clients, 100% CPU core utilization by server: >>>>> echo_bench --address '172.22.117.67:7777' -c 20 -t 60 -l 2048 >>>>> epoll_echo_server: Speed: 108704 request/sec, 108704 response/sec >>>>> io_uring_echo_server: Speed: 85536 request/sec, 85536 response/sec >>>>> epoll_echo_server is 27% faster >>>>> >>>>> Why io_uring shows consistent performance degradation? What is going wrong? >>>> >>>> 5.8 is pretty old, and I'm not sure all the performance problems were >>>> addressed there. Apart from missing common optimisations as you may >>>> have seen in the thread, it looks to me it doesn't have sighd(?) lock >>>> hammering fix. Jens, knows better it has been backported or not. >>>> >>>> So, things you can do: >>>> 1) try out 5.12 >>>> 2) attach `perf top` output or some other profiling for your 5.8 >>>> 3) to have a more complete picture do 2) with 5.12 >>>> -- Pavel Begunkov ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: io_uring networking performance degradation 2021-04-26 11:49 ` Pavel Begunkov @ 2021-04-26 13:07 ` Michael Stoler 0 siblings, 0 replies; 7+ messages in thread From: Michael Stoler @ 2021-04-26 13:07 UTC (permalink / raw) To: Pavel Begunkov; +Cc: io-uring, Jens Axboe 1. I am of the same opinion, but cannot prove it. bpftrace is too intrusive and rough to measure read/write path latency. 2. No, the test wasn't bound to a particular CPU/core. It was bound to NIC's node only by hwloc-bind os=eth_name ... 3. __skb_datagram_iter is very strange. I didn't see any activity in top during tests. In any case all test was performed over dedicated NIC. Regards Michael Stoler On Mon, Apr 26, 2021 at 2:49 PM Pavel Begunkov <[email protected]> wrote: > > On 4/25/21 10:52 AM, Michael Stoler wrote: > > Because of unstable working of perf over AWS VM I recheck test on > > physical machine: Ubuntu 20.04, 5.8.0-50-generic kernel, CPU AMD EPYC > > 7272 12-Core Processor 3200MHz, BogoMIPS 5789.39, NIC melanox 5, > > Speed: 25000Mb/s Full Duplex. > > Over physical machine performance degradation is much less pronounced > > but still exists: > > io_uring-echo-server Speed: 143081 request/sec, 143081 response/sec > > epoll-echo-server Speed: 150692 request/sec, 150692 response/sec > > epoll-echo-server is 5% faster > > Have to note that I haven't check the userspace programs, so not sure > it's a fair comparison (may be or may be not). So, with it being said: > > 1) The last report had lot of idle time, so it may be a question of > latency but not throughput for it. > > 2) Did you do proper pinning to a CPU/core? taskset or cset? Also, > did it saturate the CPU/core you used in the most recent post? > > 3) Looking at __skb_datagram_iter taking 1%, seems there are other > tasks taking a relatively good share of CPU/NIC resources. What is > this datagram? UDP on the same NIC? Is something else using your > NIC/interface? > > 4) don't see even close anything related to io_uring in the recent > run, and it was only a small fraction in previous ones. So it's > definitely not the overhead on submit/complete. If there is a > io_uring problem, it could be the difference in polling / iowq > punting comparing with epoll. It may be interesting to look into. > > And related thing I'm curious about is to compare FAST_POLL > requests with io_uring multi-shot polling + send/recv. > > > > > > "perf top" with io_uring-echo-server: > > PerfTop: 16481 irqs/sec kernel:98.5% exact: 99.8% lost: 0/0 drop: > > 0/0 [4000Hz cycles], (all, 24 CPUs) > > ------------------------------------------------------------------------------- > > 8.66% [kernel] [k] __x86_indirect_thunk_rax > > 8.49% [kernel] [k] copy_user_generic_string > > 5.57% [kernel] [k] memset > > 2.81% [kernel] [k] tcp_rate_skb_sent > > 2.32% [kernel] [k] __alloc_skb > > 2.16% [kernel] [k] __check_object_size > > 1.44% [unknown] [k] 0xffffffffc100c296 > > 1.28% [kernel] [k] tcp_write_xmit > > 1.22% [kernel] [k] iommu_dma_map_page > > 1.16% [kernel] [k] kmem_cache_free > > 1.14% [kernel] [k] __softirqentry_text_start > > 1.06% [unknown] [k] 0xffffffffc1008a7e > > 1.03% [kernel] [k] __skb_datagram_iter > > 0.97% [kernel] [k] __dev_queue_xmit > > 0.86% [kernel] [k] ipv4_mtu > > 0.85% [kernel] [k] tcp_schedule_loss_probe > > 0.80% [kernel] [k] tcp_release_cb > > 0.78% [unknown] [k] 0xffffffffc100c290 > > 0.77% [unknown] [k] 0xffffffffc100c295 > > 0.76% perf [.] __symbols__insert > > > > "perf top" with epoll-echo-server: > > PerfTop: 24255 irqs/sec kernel:98.3% exact: 99.6% lost: 0/0 drop: > > 0/0 [4000Hz cycles], (all, 24 CPUs) > > ------------------------------------------------------------------------------- > > 8.77% [kernel] [k] __x86_indirect_thunk_rax > > 7.50% [kernel] [k] copy_user_generic_string > > 4.10% [kernel] [k] memset > > 2.70% [kernel] [k] tcp_rate_skb_sent > > 2.18% [kernel] [k] __check_object_size > > 2.09% [kernel] [k] __alloc_skb > > 1.61% [unknown] [k] 0xffffffffc100c296 > > 1.47% [kernel] [k] __virt_addr_valid > > 1.40% [kernel] [k] iommu_dma_map_page > > 1.37% [unknown] [k] 0xffffffffc1008a7e > > 1.22% [kernel] [k] tcp_poll > > 1.16% [kernel] [k] __softirqentry_text_start > > 1.15% [kernel] [k] tcp_stream_memory_free > > 1.07% [kernel] [k] tcp_write_xmit > > 1.06% [kernel] [k] kmem_cache_free > > 1.03% [kernel] [k] tcp_release_cb > > 0.96% [kernel] [k] syscall_return_via_sysret > > 0.90% [kernel] [k] __lock_text_start > > 0.82% [kernel] [k] __copy_skb_header > > 0.81% [kernel] [k] amd_iommu_map > > > > Regards > > Michael Stoler > > > > On Tue, Apr 20, 2021 at 1:44 PM Michael Stoler <[email protected]> wrote: > >> > >> Hi, perf data and tops for linux-5.8 are here: > >> http://rdxdownloads.rdxdyn.com/michael_stoler_perf_data.tgz > >> > >> Regards > >> Michael Stoler > >> > >> On Mon, Apr 19, 2021 at 5:27 PM Michael Stoler <[email protected]> wrote: > >>> > >>> 1) linux-5.12-rc8 shows generally same picture: > >>> > >>> average load, 70-85% CPU core usage, 128 bytes packets > >>> echo_bench --address '172.22.150.170:7777' --number 10 --duration > >>> 60 --length 128` > >>> epoll-echo-server: Speed: 71513 request/sec, 71513 response/sec > >>> io_uring_echo_server: Speed: 64091 request/sec, 64091 response/sec > >>> epoll-echo-server is 11% faster > >>> > >>> high load, 95-100% CPU core usage, 128 bytes packets > >>> echo_bench --address '172.22.150.170:7777' --number 20 --duration > >>> 60 --length 128` > >>> epoll-echo-server: Speed: 130186 request/sec, 130186 response/sec > >>> io_uring_echo_server: Speed: 109793 request/sec, 109793 response/sec > >>> epoll-echo-server is 18% faster > >>> > >>> average load, 70-85% CPU core usage, 2048 bytes packets > >>> echo_bench --address '172.22.150.170:7777' --number 10 --duration > >>> 60 --length 2048` > >>> epoll-echo-server: Speed: 63082 request/sec, 63082 response/sec > >>> io_uring_echo_server: Speed: 59449 request/sec, 59449 response/sec > >>> epoll-echo-server is 6% faster > >>> > >>> high load, 95-100% CPU core usage, 2048 bytes packets > >>> echo_bench --address '172.22.150.170:7777' --number 20 --duration > >>> 60 --length 2048` > >>> epoll-echo-server: Speed: 110402 request/sec, 110402 response/sec > >>> io_uring_echo_server: Speed: 88718 request/sec, 88718 response/sec > >>> epoll-echo-server is 24% faster > >>> > >>> > >>> 2-3) The "perf top" doesn't work stable with Ubuntu over AWS. All the > >>> time it shows errors: "Uhhuh. NMI received for unknown reason", "Do > >>> you have a strange power saving mode enabled?", "Dazed and confused, > >>> but trying to continue". > >>> > >>> Regards > >>> Michael Stoler > >>> > >>> On Mon, Apr 19, 2021 at 1:20 PM Pavel Begunkov <[email protected]> wrote: > >>>> > >>>> On 4/19/21 10:13 AM, Michael Stoler wrote: > >>>>> We are trying to reproduce reported on page > >>>>> https://github.com/frevib/io_uring-echo-server/blob/master/benchmarks/benchmarks.md > >>>>> results with a more realistic environment: > >>>>> 1. Internode networking in AWS cluster with i3.16xlarge nodes type(25 > >>>>> Gigabit network connection between client and server) > >>>>> 2. 128 and 2048 packet sizes, to simulate typical payloads > >>>>> 3. 10 clients to get 75-95% CPU utilization by server to simulate > >>>>> server's normal load > >>>>> 4. 20 clients to get 100% CPU utilization by server to simulate > >>>>> server's hard load > >>>>> > >>>>> Software: > >>>>> 1. OS: Ubuntu 20.04.2 LTS HWE with 5.8.0-45-generic kernel with latest liburing > >>>>> 2. io_uring-echo-server: https://github.com/frevib/io_uring-echo-server > >>>>> 3. epoll-echo-server: https://github.com/frevib/epoll-echo-server > >>>>> 4. benchmark: https://github.com/haraldh/rust_echo_bench > >>>>> 5. all commands runs with "hwloc-bind os=eth1" > >>>>> > >>>>> The results are confusing, epoll_echo_server shows stable advantage > >>>>> over io_uring-echo-server, despite reported advantage of > >>>>> io_uring-echo-server: > >>>>> > >>>>> 128 bytes packet size, 10 clients, 75-95% CPU core utilization by server: > >>>>> echo_bench --address '172.22.117.67:7777' -c 10 -t 60 -l 128 > >>>>> epoll_echo_server: Speed: 80999 request/sec, 80999 response/sec > >>>>> io_uring_echo_server: Speed: 74488 request/sec, 74488 response/sec > >>>>> epoll_echo_server is 8% faster > >>>>> > >>>>> 128 bytes packet size, 20 clients, 100% CPU core utilization by server: > >>>>> echo_bench --address '172.22.117.67:7777' -c 20 -t 60 -l 128 > >>>>> epoll_echo_server: Speed: 129063 request/sec, 129063 response/sec > >>>>> io_uring_echo_server: Speed: 102681 request/sec, 102681 response/sec > >>>>> epoll_echo_server is 25% faster > >>>>> > >>>>> 2048 bytes packet size, 10 clients, 75-95% CPU core utilization by server: > >>>>> echo_bench --address '172.22.117.67:7777' -c 10 -t 60 -l 2048 > >>>>> epoll_echo_server: Speed: 74421 request/sec, 74421 response/sec > >>>>> io_uring_echo_server: Speed: 66510 request/sec, 66510 response/sec > >>>>> epoll_echo_server is 11% faster > >>>>> > >>>>> 2048 bytes packet size, 20 clients, 100% CPU core utilization by server: > >>>>> echo_bench --address '172.22.117.67:7777' -c 20 -t 60 -l 2048 > >>>>> epoll_echo_server: Speed: 108704 request/sec, 108704 response/sec > >>>>> io_uring_echo_server: Speed: 85536 request/sec, 85536 response/sec > >>>>> epoll_echo_server is 27% faster > >>>>> > >>>>> Why io_uring shows consistent performance degradation? What is going wrong? > >>>> > >>>> 5.8 is pretty old, and I'm not sure all the performance problems were > >>>> addressed there. Apart from missing common optimisations as you may > >>>> have seen in the thread, it looks to me it doesn't have sighd(?) lock > >>>> hammering fix. Jens, knows better it has been backported or not. > >>>> > >>>> So, things you can do: > >>>> 1) try out 5.12 > >>>> 2) attach `perf top` output or some other profiling for your 5.8 > >>>> 3) to have a more complete picture do 2) with 5.12 > >>>> > > > -- > Pavel Begunkov -- Michael Stoler ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-04-26 13:07 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-04-19 9:13 io_uring networking performance degradation Michael Stoler 2021-04-19 10:20 ` Pavel Begunkov 2021-04-19 14:27 ` Michael Stoler 2021-04-20 10:44 ` Michael Stoler 2021-04-25 9:52 ` Michael Stoler 2021-04-26 11:49 ` Pavel Begunkov 2021-04-26 13:07 ` Michael Stoler
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox