public inbox for [email protected]
 help / color / mirror / Atom feed
From: Jens Axboe <[email protected]>
To: Xiaoguang Wang <[email protected]>,
	[email protected]
Cc: Pavel Begunkov <[email protected]>,
	joseph qi <[email protected]>,
	Jiufei Xue <[email protected]>
Subject: Re: Is io_uring framework becoming bloated gradually? and introduces performace regression
Date: Fri, 8 May 2020 10:37:49 -0600	[thread overview]
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>

On 5/8/20 9:18 AM, Xiaoguang Wang wrote:
> hi,
> 
> This issue was found when I tested IORING_FEAT_FAST_POLL feature, with
> the newest upstream codes, indeed I find that io_uring's performace
> improvement is not obvious compared to epoll in my test environment,
> most of the time they are similar.  Test cases basically comes from:
> https://github.com/frevib/io_uring-echo-server/blob/io-uring-feat-fast-poll/benchmarks/benchmarks.md.
> In above url, the author's test results shows that io_uring will get a
> big performace improvement compared to epoll. I'm still looking into
> why I don't get the big improvement, currently don't know why, but I
> find some obvious regression issue.
> 
> I wrote a simple tool based io_uring nop operation to evaluate
> io_uring framework in v5.1 and 5.7.0-rc4+(jens's io_uring-5.7 branch),
> I see a obvious performace regression:
>
> v5.1 kernel:
>      $sudo taskset -c 60 ./io_uring_nop_stress -r 300 # run 300 seconds
>      total ios: 1832524960
>      IOPS:      6108416
> 5.7.0-rc4+
>      $sudo taskset -c 60 ./io_uring_nop_stress -r 300
>      total ios: 1597672304
>      IOPS:      5325574
> it's about 12% performance regression.

For sure there's a bit more bloat in 5.7+ compared to the initial slim
version, and focus has been on features to a certain extent recently.
The poll rework for 5.7 will really improve performance for the
networked side though, so it's not like it's just piling on features
that add bloat.

That said, I do think it's time for a revisit on overhead. It's been a
while since I've done my disk IO testing. The nop testing isn't _that_
interesting by itself, as a micro benchmark it does yield some results
though. Are you running on bare metal or in a VM?

> Using perf can see many performance bottlenecks, for example,
> io_submit_sqes is one.  For now, I did't make many analysis yet, just
> have a look at io_submit_sqes(), there are many assignment operations
> in io_init_req(), but I'm not sure whether they are all needed when
> req is not needed to be punt to io-wq, for example,
> INIT_IO_WORK(&req->work, io_wq_submit_work); # a whole struct
> assignment from perf annotate tool, it's an expensive operation, I
> think reqs that use fast poll feature use task-work function, so the
> INIT_IO_WORK maybe not necessary.

I'm sure there's some low hanging fruit there, and I'd love to take
patches for it.

> Above is just one issue, what I worry is that whether io_uring is
> becoming more bloated gradually, and will not that better to aio. In
> https://kernel.dk/io_uring.pdf, it says that io_uring will eliminate
> 104 bytes copy compared to aio, but see currenct io_init_req(),
> io_uring maybe copy more, introducing more overhead? Or does we need
> to carefully re-design struct io_kiocb, to reduce overhead as soon as
> possible.

The copy refers to the data structures coming in and out, both io_uring
and io_uring inititalize their main io_kiocb/aio_kiocb structure as
well. The io_uring is slightly bigger, but not much, and it's the same
number of cachelines. So should not be a huge difference there. The
copying on the aio side is basically first the pointer copy, then the
user side kiocb structure. io_uring doesn't need to do that. The
completion side is also slimmer. We also don't need as many system calls
to do the same thing, for example.

So no, we should always been substantially slimmer than aio, just by the
very nature of the API.

One major thing I've been thinking about for io_uring is io_kiocb
recycling. We're hitting the memory allocator for alloc+free for each
request, even though that can be somewhat amortized by doing batched
submissions, and polling for instance can also do batched frees. But I'm
pretty sure we can find some gains here by having some io_kiocb caching
that is persistent across operations.

Outside of that, runtime analysis and speeding up the normal path
through io_uring will probably also easily yield us an extra 5% (or
more).

-- 
Jens Axboe


  reply	other threads:[~2020-05-08 16:37 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-08 15:18 Is io_uring framework becoming bloated gradually? and introduces performace regression Xiaoguang Wang
2020-05-08 16:37 ` Jens Axboe [this message]
2020-05-10 18:10   ` H. de Vries
2020-05-11  5:43     ` Xiaoguang Wang
2020-05-11  3:30   ` Xiaoguang Wang
2020-05-11 18:08     ` Jens Axboe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox