public inbox for [email protected]
 help / color / mirror / Atom feed
From: Jens Axboe <[email protected]>
To: Olivier Langlois <[email protected]>
Cc: Pavel Begunkov <[email protected]>,
	Hao Xu <[email protected]>,
	io-uring <[email protected]>,
	linux-kernel <[email protected]>
Subject: Re: [PATCH v1] io_uring: Add support for napi_busy_poll
Date: Sat, 19 Feb 2022 17:22:13 -0700	[thread overview]
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>

On 2/19/22 2:42 PM, Olivier Langlois wrote:
> One side effect that I have discovered from testing the napi_busy_poll
> patch, despite improving the network timing of the threads performing
> the busy poll, it is the networking performance degradation that it has
> on the rest of the system.
> 
> I dedicate isolated CPUS to specific threads of my program. My kernel
> is compiled with CONFIG_NO_HZ_FULL. One thing that I have never really
> understood is why there were still kernel threads assigned to the
> isolated CPUs.
> 
> $ CORENUM=2; ps -L -e -o pid,psr,cpu,cmd | grep -E 
> "^[[:space:]]+[[:digit:]]+[[:space:]]+${CORENUM}"
>      24   2   - [cpuhp/2]
>      25   2   - [idle_inject/2]
>      26   2   - [migration/2]
>      27   2   - [ksoftirqd/2]
>      28   2   - [kworker/2:0-events]
>      29   2   - [kworker/2:0H]
>      83   2   - [kworker/2:1-mm_percpu_wq]
> 
> It is very hard to keep the CPU 100% tickless if there are still tasks
> assigned to isolated CPUs by the kernel.
> 
> This question isn't really answered anywhere AFAIK:
> https://www.kernel.org/doc/html/latest/timers/no_hz.html
> https://jeremyeder.com/2013/11/15/nohz_fullgodmode/
> 
> Those threads running on their dedicated CPUS are the ones doing the
> NAPI busy polling. Because of that, those CPUs usage ramp up to 100%
> and running ping on the side is now having horrible numbers:
> 
> [2022-02-19 07:27:54] INFO SOCKPP/ping ping results for 10 loops:
> 0. 104.16.211.191 rtt min/avg/max/mdev = 9.926/34.987/80.048/17.016 ms
> 1. 104.16.212.191 rtt min/avg/max/mdev = 9.861/34.934/79.986/17.019 ms
> 2. 104.16.213.191 rtt min/avg/max/mdev = 9.876/34.949/79.965/16.997 ms
> 3. 104.16.214.191 rtt min/avg/max/mdev = 9.852/34.927/79.977/17.019 ms
> 4. 104.16.215.191 rtt min/avg/max/mdev = 9.869/34.943/79.958/16.997 ms
> 
> Doing this:
> echo 990000 > /proc/sys/kernel/sched_rt_runtime_us
> 
> as instructed here:
> https://www.kernel.org/doc/html/latest/scheduler/sched-rt-group.html
> 
> fix the problem:
> 
> $ ping 104.16.211.191
> PING 104.16.211.191 (104.16.211.191) 56(84) bytes of data.
> 64 bytes from 104.16.211.191: icmp_seq=1 ttl=62 time=1.05 ms
> 64 bytes from 104.16.211.191: icmp_seq=2 ttl=62 time=0.812 ms
> 64 bytes from 104.16.211.191: icmp_seq=3 ttl=62 time=0.864 ms
> 64 bytes from 104.16.211.191: icmp_seq=4 ttl=62 time=0.846 ms
> 64 bytes from 104.16.211.191: icmp_seq=5 ttl=62 time=1.23 ms
> 64 bytes from 104.16.211.191: icmp_seq=6 ttl=62 time=0.957 ms
> 64 bytes from 104.16.211.191: icmp_seq=7 ttl=62 time=1.10 ms
> ^C
> --- 104.16.211.191 ping statistics ---
> 7 packets transmitted, 7 received, 0% packet loss, time 6230ms
> rtt min/avg/max/mdev = 0.812/0.979/1.231/0.142 ms
> 
> If I was to guess, I would say that it is ksoftirqd on those CPUs that
> is starving and is not servicing the network packets but I wish that I
> had a better understanding of what is really happening and know if it
> would be possible to keep 100% those processors dedicated to my tasks
> and have the network softirqs handled somewhere else to not have to
> tweak /proc/sys/kernel/sched_rt_runtime_us to fix the issue...

Outside of this, I was hoping to see some performance numbers in the
main patch. Sounds like you have them, can you share?

-- 
Jens Axboe


  reply	other threads:[~2022-02-20  0:22 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-02-19  8:03 [PATCH v1] io_uring: Add support for napi_busy_poll Olivier Langlois
2022-02-19 21:42 ` Olivier Langlois
2022-02-20  0:22   ` Jens Axboe [this message]
2022-02-20 18:37     ` Olivier Langlois
2022-02-20 19:38       ` Jens Axboe
2022-02-21 19:29         ` Olivier Langlois
2022-02-21  5:25       ` Hao Xu
2022-02-20 20:51 ` kernel test robot
2022-02-20 21:53 ` kernel test robot
2022-02-20 21:53 ` kernel test robot
2022-02-21  5:23 ` Hao Xu
2022-02-25  5:32   ` Olivier Langlois
2022-02-25 15:32     ` Olivier Langlois
2022-02-28 18:34       ` Hao Xu
2022-02-28 21:20         ` Olivier Langlois
2022-03-01  3:53           ` Hao Xu
2022-02-28 18:26     ` Hao Xu
2022-02-28 21:01       ` Olivier Langlois
2022-03-01  8:23         ` Hao Xu

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox