public inbox for [email protected]
 help / color / mirror / Atom feed
From: Dmitry Shulyak <[email protected]>
To: Jens Axboe <[email protected]>
Cc: [email protected]
Subject: Re: Large number of empty reads on 5.9-rc2 under moderate load
Date: Mon, 24 Aug 2020 14:09:33 +0300	[thread overview]
Message-ID: <CAF-ewDpvLwkiZ3sJMT64e=efCRFYVkt2Z71==1FztLg=vZN8fg@mail.gmail.com> (raw)
In-Reply-To: <[email protected]>

library that i am using https://github.com/dshulyak/uring
It requires golang 1.14, if installed, benchmark can be run with:
go test ./fs -run=xx -bench=BenchmarkReadAt/uring_8 -benchtime=1000000x
go test ./fs -run=xx -bench=BenchmarkReadAt/uring_5 -benchtime=8000000x

note that it will setup uring instance per cpu, with shared worker pool.
it will take me too much time to implement repro in c, but in general
i am simply submitting multiple concurrent
read requests and watching read rate.



On Mon, 24 Aug 2020 at 13:46, Jens Axboe <[email protected]> wrote:
>
> On 8/24/20 4:40 AM, Dmitry Shulyak wrote:
> > In the program, I am submitting a large number of concurrent read
> > requests with o_direct. In both scenarios the number of concurrent
> > read requests is limited to 20 000, with only difference being that
> > for 512b total number of reads is 8millions and for 8kb - 1million. On
> > 5.8.3 I didn't see any empty reads at all.
> >
> > BenchmarkReadAt/uring_512-8              8000000              1879
> > ns/op         272.55 MB/s
> > BenchmarkReadAt/uring_8192-8             1000000             18178
> > ns/op         450.65 MB/s
> >
> > I am seeing the same numbers in iotop, so pretty confident that the
> > benchmark is fine. Below is a version with regular syscalls and
> > threads (note that this is with golang):
> >
> > BenchmarkReadAt/os_512-256               8000000              4393
> > ns/op         116.55 MB/s
> > BenchmarkReadAt/os_8192-256              1000000             18811
> > ns/op         435.48 MB/s
> >
> > I run the same program on 5.9-rc.2 and noticed that for workload with
> > 8kb buffer and 1mill reads I had to make more than 7 millions retries,
> > which obviously makes the program very slow. For 512b and 8million
> > reads there were only 22 000 retries, but it is still very slow for
> > some other reason.
> >
> > BenchmarkReadAt/uring_512-8  8000000       8432 ns/op   60.72 MB/s
> > BenchmarkReadAt/uring_8192-8 1000000      42603 ns/op 192.29 MB/s
> >
> > In iotop i am seeing a huge increase for 8kb, actual disk read goes up
> > to 2gb/s, which looks somewhat suspicious given that my ssd should
> > support only 450mb/s. If I will lower the number of concurrent
> > requests to 1000, then there are almost no empty reads and numbers for
> > 8kb go back to the same level I saw with 5.8.3.
> >
> > Is it a regression or should I throttle submissions?
>
> Since it's performing worse than 5.8, sounds like there is. How can we
> reproduce this?
>
> --
> Jens Axboe
>

  reply	other threads:[~2020-08-24 11:09 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-08-24 10:40 Large number of empty reads on 5.9-rc2 under moderate load Dmitry Shulyak
2020-08-24 10:46 ` Jens Axboe
2020-08-24 11:09   ` Dmitry Shulyak [this message]
2020-08-24 14:06     ` Jens Axboe
2020-08-24 14:45       ` Jens Axboe
2020-08-24 15:33         ` Dmitry Shulyak
2020-08-24 16:10           ` Jens Axboe
2020-08-24 16:13             ` Dmitry Shulyak
2020-08-24 16:18               ` Jens Axboe
2020-08-24 17:44                 ` Jens Axboe
2020-08-25  8:52                   ` Dmitry Shulyak
2020-08-25 13:39                     ` Jens Axboe
2020-08-25 14:14                       ` Dmitry Shulyak

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAF-ewDpvLwkiZ3sJMT64e=efCRFYVkt2Z71==1FztLg=vZN8fg@mail.gmail.com' \
    [email protected] \
    [email protected] \
    [email protected] \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox