public inbox for [email protected]
 help / color / mirror / Atom feed
From: Xiaoguang Wang <[email protected]>
To: [email protected]
Cc: [email protected], [email protected]
Subject: [PATCH v2 0/2] improve SQPOLL handling
Date: Tue,  3 Nov 2020 14:15:58 +0800	[thread overview]
Message-ID: <[email protected]> (raw)

The first patch tries to improve various issues in current implementation:
  The prepare_to_wait() usage in __io_sq_thread() is weird. If multiple ctxs
share one same poll thread, one ctx will put poll thread in TASK_INTERRUPTIBLE,
but if other ctxs have work to do, we don't need to change task's stat at all.
I think only if all ctxs don't have work to do, we can do it.
  We use round-robin strategy to make multiple ctxs share one same poll thread,
but there are various condition in __io_sq_thread(), which seems complicated and
may affect round-robin strategy.

The second patch adds a IORING_SETUP_SQPOLL_PERCPU flag, for those rings which
have SQPOLL enabled and are willing to be bound to one same cpu, hence share
one same poll thread, add a capability that these rings can share one poll thread
by specifying a new IORING_SETUP_SQPOLL_PERCPU flag. FIO tool can integrate this
feature easily, so we can test multiple rings to share same poll thread easily.


TEST:
  This patch set have passed liburing test cases.

  I also make fio support IORING_SETUP_SQPOLL_PERCPU flag, and make some
io stress tests, no errors or performance regression. See below fio job file:

First in unpatched kernel, I test a fio file which only contains one job
with iodepth being 128, see below:
[global]
ioengine=io_uring
sqthread_poll=1
registerfiles=1
fixedbufs=1
hipri=1
thread=1
bs=4k
direct=1
rw=randread
time_based=1
runtime=120
ramp_time=0
randrepeat=0
group_reporting=1
filename=/dev/nvme0n1
sqthread_poll_cpu=15

[job0]
cpus_allowed=5
iodepth=128
sqthread_poll_cpu=9

performance data: IOPS: 453k, avg lat: 282.37usec


Second in unpatched kernel, I test a fio file which contains 4 jobs
with each iodepth being 32, see below:
[global]
ioengine=io_uring
sqthread_poll=1
registerfiles=1
fixedbufs=1
hipri=1
thread=1
bs=4k
direct=1
rw=randread
time_based=1
runtime=120
ramp_time=0
randrepeat=0
group_reporting=1
filename=/dev/nvme0n1
sqthread_poll_cpu=15

[job0]
cpus_allowed=5
iodepth=32
sqthread_poll_cpu=9

[job1]
cpus_allowed=6
iodepth=32
sqthread_poll_cpu=9

[job2]
cpus_allowed=7
iodepth=32
sqthread_poll_cpu=9

[job3]
cpus_allowed=8
iodepth=32
sqthread_poll_cpu=9
performance data: IOPS: 254k, avg lat: 503.80 usec, obvious performance
drop.

Finally in patched kernel, I test a fio file which contains 4 jobs
with each iodepth being 32, and now we enable sqthread_poll_percpu
flag, see blow:

[global]
ioengine=io_uring
sqthread_poll=1
registerfiles=1
fixedbufs=1
hipri=1
thread=1
bs=4k
direct=1
rw=randread
time_based=1
runtime=120
ramp_time=0
randrepeat=0
group_reporting=1
filename=/dev/nvme0n1
#sqthread_poll_cpu=15
sqthread_poll_percpu=1  # enable percpu feature

[job0]
cpus_allowed=5
iodepth=32
sqthread_poll_cpu=9

[job1]
cpus_allowed=6
iodepth=32
sqthread_poll_cpu=9

[job2]
cpus_allowed=7
iodepth=32
sqthread_poll_cpu=9

performance data: IOPS: 438k, avg lat: 291.69usec


From above teses, we can see that IORING_SETUP_SQPOLL_PERCPU is easy to
use, and no obvious performance regression.
Note I don't test IORING_SETUP_ATTACH_WQ in above three test cases, it's
a little hard to support IORING_SETUP_ATTACH_WQ in fio.

Xiaoguang Wang (2):
  io_uring: refactor io_sq_thread() handling
  io_uring: support multiple rings to share same poll thread by
    specifying same cpu

 fs/io_uring.c                 | 289 +++++++++++++++++++---------------
 include/uapi/linux/io_uring.h |   1 +
 2 files changed, 166 insertions(+), 124 deletions(-)

-- 
2.17.2


             reply	other threads:[~2020-11-03  6:16 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-03  6:15 Xiaoguang Wang [this message]
2020-11-03  6:15 ` [PATCH v2 1/2] io_uring: refactor io_sq_thread() handling Xiaoguang Wang
2020-11-03  6:16 ` [PATCH v2 2/2] io_uring: support multiple rings to share same poll thread by specifying same cpu Xiaoguang Wang
2020-11-08 14:16 ` [PATCH v2 0/2] improve SQPOLL handling Xiaoguang Wang
2020-11-09 14:42   ` Jens Axboe
2020-11-10  2:31     ` Xiaoguang Wang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201103061600.11053-1-xiaoguang.wang@linux.alibaba.com \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox