public inbox for [email protected]
 help / color / mirror / Atom feed
From: Xiaoguang Wang <[email protected]>
To: Jens Axboe <[email protected]>, [email protected]
Cc: Pavel Begunkov <[email protected]>,
	joseph qi <[email protected]>,
	Jiufei Xue <[email protected]>
Subject: Re: Is io_uring framework becoming bloated gradually? and introduces performace regression
Date: Mon, 11 May 2020 11:30:27 +0800	[thread overview]
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>

hi,

> On 5/8/20 9:18 AM, Xiaoguang Wang wrote:
>> hi,
>>
>> This issue was found when I tested IORING_FEAT_FAST_POLL feature, with
>> the newest upstream codes, indeed I find that io_uring's performace
>> improvement is not obvious compared to epoll in my test environment,
>> most of the time they are similar.  Test cases basically comes from:
>> https://github.com/frevib/io_uring-echo-server/blob/io-uring-feat-fast-poll/benchmarks/benchmarks.md.
>> In above url, the author's test results shows that io_uring will get a
>> big performace improvement compared to epoll. I'm still looking into
>> why I don't get the big improvement, currently don't know why, but I
>> find some obvious regression issue.
>>
>> I wrote a simple tool based io_uring nop operation to evaluate
>> io_uring framework in v5.1 and 5.7.0-rc4+(jens's io_uring-5.7 branch),
>> I see a obvious performace regression:
>>
>> v5.1 kernel:
>>       $sudo taskset -c 60 ./io_uring_nop_stress -r 300 # run 300 seconds
>>       total ios: 1832524960
>>       IOPS:      6108416
>> 5.7.0-rc4+
>>       $sudo taskset -c 60 ./io_uring_nop_stress -r 300
>>       total ios: 1597672304
>>       IOPS:      5325574
>> it's about 12% performance regression.
> 
> For sure there's a bit more bloat in 5.7+ compared to the initial slim
> version, and focus has been on features to a certain extent recently.
> The poll rework for 5.7 will really improve performance for the
> networked side though, so it's not like it's just piling on features
> that add bloat.
> 
> That said, I do think it's time for a revisit on overhead. It's been a
> while since I've done my disk IO testing. The nop testing isn't _that_
> interesting by itself, as a micro benchmark it does yield some results
> though. Are you running on bare metal or in a VM?
I run it on bare metal.

> 
>> Using perf can see many performance bottlenecks, for example,
>> io_submit_sqes is one.  For now, I did't make many analysis yet, just
>> have a look at io_submit_sqes(), there are many assignment operations
>> in io_init_req(), but I'm not sure whether they are all needed when
>> req is not needed to be punt to io-wq, for example,
>> INIT_IO_WORK(&req->work, io_wq_submit_work); # a whole struct
>> assignment from perf annotate tool, it's an expensive operation, I
>> think reqs that use fast poll feature use task-work function, so the
>> INIT_IO_WORK maybe not necessary.
> 
> I'm sure there's some low hanging fruit there, and I'd love to take
> patches for it.
Ok, I'll try to improve it a bit, thanks.

> 
>> Above is just one issue, what I worry is that whether io_uring is
>> becoming more bloated gradually, and will not that better to aio. In
>> https://kernel.dk/io_uring.pdf, it says that io_uring will eliminate
>> 104 bytes copy compared to aio, but see currenct io_init_req(),
>> io_uring maybe copy more, introducing more overhead? Or does we need
>> to carefully re-design struct io_kiocb, to reduce overhead as soon as
>> possible.
> 
> The copy refers to the data structures coming in and out, both io_uring
> and io_uring inititalize their main io_kiocb/aio_kiocb structure as
> well. The io_uring is slightly bigger, but not much, and it's the same
> number of cachelines. So should not be a huge difference there. The
> copying on the aio side is basically first the pointer copy, then the
> user side kiocb structure. io_uring doesn't need to do that. The
> completion side is also slimmer. We also don't need as many system calls
> to do the same thing, for example.
> 
> So no, we should always been substantially slimmer than aio, just by the
> very nature of the API.
Really thanks for detailed explanations, now I see, feel good:)

> 
> One major thing I've been thinking about for io_uring is io_kiocb
> recycling. We're hitting the memory allocator for alloc+free for each
> request, even though that can be somewhat amortized by doing batched
> submissions, and polling for instance can also do batched frees. But I'm
> pretty sure we can find some gains here by having some io_kiocb caching
> that is persistent across operations.
Yes, agree. From my above nop operation tests, using perf, it shows
kmem_cache_free() and kmem_cache_alloc_bulk() introduce an considerable
overhead.

Regards,
Xiaoguang Wang
> 
> Outside of that, runtime analysis and speeding up the normal path
> through io_uring will probably also easily yield us an extra 5% (or
> more).
> 

  parent reply	other threads:[~2020-05-11  3:30 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-08 15:18 Is io_uring framework becoming bloated gradually? and introduces performace regression Xiaoguang Wang
2020-05-08 16:37 ` Jens Axboe
2020-05-10 18:10   ` H. de Vries
2020-05-11  5:43     ` Xiaoguang Wang
2020-05-11  3:30   ` Xiaoguang Wang [this message]
2020-05-11 18:08     ` Jens Axboe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=113c34c8-9f17-7890-d12a-e35c7922432e@linux.alibaba.com \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox