From: Michael Stoler <[email protected]>
To: Pavel Begunkov <[email protected]>
Cc: [email protected], Jens Axboe <[email protected]>
Subject: Re: io_uring networking performance degradation
Date: Tue, 20 Apr 2021 13:44:36 +0300 [thread overview]
Message-ID: <CAN633emWn60bbFqCF7Yxu4XDFiBXq5Z_BE5NZdX2qwQP8ju=_A@mail.gmail.com> (raw)
In-Reply-To: <CAN633emhHEo3x6S77ZszzNRr4F6Xt8aP3VDn6w3GSRtNz5Qeww@mail.gmail.com>
Hi, perf data and tops for linux-5.8 are here:
http://rdxdownloads.rdxdyn.com/michael_stoler_perf_data.tgz
Regards
Michael Stoler
On Mon, Apr 19, 2021 at 5:27 PM Michael Stoler <[email protected]> wrote:
>
> 1) linux-5.12-rc8 shows generally same picture:
>
> average load, 70-85% CPU core usage, 128 bytes packets
> echo_bench --address '172.22.150.170:7777' --number 10 --duration
> 60 --length 128`
> epoll-echo-server: Speed: 71513 request/sec, 71513 response/sec
> io_uring_echo_server: Speed: 64091 request/sec, 64091 response/sec
> epoll-echo-server is 11% faster
>
> high load, 95-100% CPU core usage, 128 bytes packets
> echo_bench --address '172.22.150.170:7777' --number 20 --duration
> 60 --length 128`
> epoll-echo-server: Speed: 130186 request/sec, 130186 response/sec
> io_uring_echo_server: Speed: 109793 request/sec, 109793 response/sec
> epoll-echo-server is 18% faster
>
> average load, 70-85% CPU core usage, 2048 bytes packets
> echo_bench --address '172.22.150.170:7777' --number 10 --duration
> 60 --length 2048`
> epoll-echo-server: Speed: 63082 request/sec, 63082 response/sec
> io_uring_echo_server: Speed: 59449 request/sec, 59449 response/sec
> epoll-echo-server is 6% faster
>
> high load, 95-100% CPU core usage, 2048 bytes packets
> echo_bench --address '172.22.150.170:7777' --number 20 --duration
> 60 --length 2048`
> epoll-echo-server: Speed: 110402 request/sec, 110402 response/sec
> io_uring_echo_server: Speed: 88718 request/sec, 88718 response/sec
> epoll-echo-server is 24% faster
>
>
> 2-3) The "perf top" doesn't work stable with Ubuntu over AWS. All the
> time it shows errors: "Uhhuh. NMI received for unknown reason", "Do
> you have a strange power saving mode enabled?", "Dazed and confused,
> but trying to continue".
>
> Regards
> Michael Stoler
>
> On Mon, Apr 19, 2021 at 1:20 PM Pavel Begunkov <[email protected]> wrote:
> >
> > On 4/19/21 10:13 AM, Michael Stoler wrote:
> > > We are trying to reproduce reported on page
> > > https://github.com/frevib/io_uring-echo-server/blob/master/benchmarks/benchmarks.md
> > > results with a more realistic environment:
> > > 1. Internode networking in AWS cluster with i3.16xlarge nodes type(25
> > > Gigabit network connection between client and server)
> > > 2. 128 and 2048 packet sizes, to simulate typical payloads
> > > 3. 10 clients to get 75-95% CPU utilization by server to simulate
> > > server's normal load
> > > 4. 20 clients to get 100% CPU utilization by server to simulate
> > > server's hard load
> > >
> > > Software:
> > > 1. OS: Ubuntu 20.04.2 LTS HWE with 5.8.0-45-generic kernel with latest liburing
> > > 2. io_uring-echo-server: https://github.com/frevib/io_uring-echo-server
> > > 3. epoll-echo-server: https://github.com/frevib/epoll-echo-server
> > > 4. benchmark: https://github.com/haraldh/rust_echo_bench
> > > 5. all commands runs with "hwloc-bind os=eth1"
> > >
> > > The results are confusing, epoll_echo_server shows stable advantage
> > > over io_uring-echo-server, despite reported advantage of
> > > io_uring-echo-server:
> > >
> > > 128 bytes packet size, 10 clients, 75-95% CPU core utilization by server:
> > > echo_bench --address '172.22.117.67:7777' -c 10 -t 60 -l 128
> > > epoll_echo_server: Speed: 80999 request/sec, 80999 response/sec
> > > io_uring_echo_server: Speed: 74488 request/sec, 74488 response/sec
> > > epoll_echo_server is 8% faster
> > >
> > > 128 bytes packet size, 20 clients, 100% CPU core utilization by server:
> > > echo_bench --address '172.22.117.67:7777' -c 20 -t 60 -l 128
> > > epoll_echo_server: Speed: 129063 request/sec, 129063 response/sec
> > > io_uring_echo_server: Speed: 102681 request/sec, 102681 response/sec
> > > epoll_echo_server is 25% faster
> > >
> > > 2048 bytes packet size, 10 clients, 75-95% CPU core utilization by server:
> > > echo_bench --address '172.22.117.67:7777' -c 10 -t 60 -l 2048
> > > epoll_echo_server: Speed: 74421 request/sec, 74421 response/sec
> > > io_uring_echo_server: Speed: 66510 request/sec, 66510 response/sec
> > > epoll_echo_server is 11% faster
> > >
> > > 2048 bytes packet size, 20 clients, 100% CPU core utilization by server:
> > > echo_bench --address '172.22.117.67:7777' -c 20 -t 60 -l 2048
> > > epoll_echo_server: Speed: 108704 request/sec, 108704 response/sec
> > > io_uring_echo_server: Speed: 85536 request/sec, 85536 response/sec
> > > epoll_echo_server is 27% faster
> > >
> > > Why io_uring shows consistent performance degradation? What is going wrong?
> >
> > 5.8 is pretty old, and I'm not sure all the performance problems were
> > addressed there. Apart from missing common optimisations as you may
> > have seen in the thread, it looks to me it doesn't have sighd(?) lock
> > hammering fix. Jens, knows better it has been backported or not.
> >
> > So, things you can do:
> > 1) try out 5.12
> > 2) attach `perf top` output or some other profiling for your 5.8
> > 3) to have a more complete picture do 2) with 5.12
> >
> > Let's find what's gone wrong
> >
> > --
> > Pavel Begunkov
>
>
>
> --
> Michael Stoler
--
Michael Stoler
next prev parent reply other threads:[~2021-04-20 10:44 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-19 9:13 io_uring networking performance degradation Michael Stoler
2021-04-19 10:20 ` Pavel Begunkov
2021-04-19 14:27 ` Michael Stoler
2021-04-20 10:44 ` Michael Stoler [this message]
2021-04-25 9:52 ` Michael Stoler
2021-04-26 11:49 ` Pavel Begunkov
2021-04-26 13:07 ` Michael Stoler
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAN633emWn60bbFqCF7Yxu4XDFiBXq5Z_BE5NZdX2qwQP8ju=_A@mail.gmail.com' \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox