public inbox for [email protected]
 help / color / mirror / Atom feed
* Question about poll_multi_file
@ 2021-06-03 12:53 Hao Xu
  2021-06-03 18:01 ` Jens Axboe
  0 siblings, 1 reply; 4+ messages in thread
From: Hao Xu @ 2021-06-03 12:53 UTC (permalink / raw)
  To: Jens Axboe, Pavel Begunkov; +Cc: io-uring

Hi Jens,
I've a question about poll_multi_file in io_do_iopoll().
It keeps spinning in f_op->iopoll() if poll_multi_file is
true (and we're under the requested amount). But in my
understanding, reqs may be in different hardware queues
for blk-mq device even in this situation.
Should we consider the hardware queue number as well? Some
thing like below:

diff --git a/fs/io_uring.c b/fs/io_uring.c
index 7b0e6bc9ea3d..479a75022449 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -2533,10 +2533,13 @@ static void io_iopoll_req_issued(struct io_kiocb 
*req, bool in_async)
                 ctx->poll_multi_file = false;
         } else if (!ctx->poll_multi_file) {
                 struct io_kiocb *list_req;
+               unsigned int queue_num0, queue_num1;

                 list_req = list_first_entry(&ctx->iopoll_list, struct 
io_kiocb,
                                                 inflight_entry);
-               if (list_req->file != req->file)
+               queue_num0 = 
blk_qc_t_to_queue_num(list_req->rw.kiocb.ki_cookie);
+               queue_num1 = blk_qc_t_to_queue_num(req->rw.kiocb.ki_cookie);
+               if (list_req->file != req->file || queue_num0 != queue_num1)
                         ctx->poll_multi_file = true;
         }

^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: Question about poll_multi_file
  2021-06-03 12:53 Question about poll_multi_file Hao Xu
@ 2021-06-03 18:01 ` Jens Axboe
  2021-06-06 14:08   ` Hao Xu
  0 siblings, 1 reply; 4+ messages in thread
From: Jens Axboe @ 2021-06-03 18:01 UTC (permalink / raw)
  To: Hao Xu, Pavel Begunkov; +Cc: io-uring

On 6/3/21 6:53 AM, Hao Xu wrote:
> Hi Jens,
> I've a question about poll_multi_file in io_do_iopoll().
> It keeps spinning in f_op->iopoll() if poll_multi_file is
> true (and we're under the requested amount). But in my
> understanding, reqs may be in different hardware queues
> for blk-mq device even in this situation.
> Should we consider the hardware queue number as well? Some
> thing like below:

That looks reasonable to me - do you have any performance
numbers to go with it?

-- 
Jens Axboe


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Question about poll_multi_file
  2021-06-03 18:01 ` Jens Axboe
@ 2021-06-06 14:08   ` Hao Xu
  2021-06-07 16:45     ` Pavel Begunkov
  0 siblings, 1 reply; 4+ messages in thread
From: Hao Xu @ 2021-06-06 14:08 UTC (permalink / raw)
  To: Jens Axboe, Pavel Begunkov; +Cc: io-uring, Joseph Qi

在 2021/6/4 上午2:01, Jens Axboe 写道:
> On 6/3/21 6:53 AM, Hao Xu wrote:
>> Hi Jens,
>> I've a question about poll_multi_file in io_do_iopoll().
>> It keeps spinning in f_op->iopoll() if poll_multi_file is
>> true (and we're under the requested amount). But in my
>> understanding, reqs may be in different hardware queues
>> for blk-mq device even in this situation.
>> Should we consider the hardware queue number as well? Some
>> thing like below:
> 
> That looks reasonable to me - do you have any performance
> numbers to go with it?

Not very easy for me to construct a good case. I'm trying to
mock the below situation:
manully control uring reqs to go to 2 hardware queues, like:
    hw_queue0     hw_queue1
    heavy_req     simple_req
    heavy_req     simple_req
      ...            ...

heavy_req is some request that needs more time to complete,
while simple_req takes less time. And make the io_do_iopoll()
alway first spin on hw_queue0.
any ideas?

Thanks,
Hao
> 


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Question about poll_multi_file
  2021-06-06 14:08   ` Hao Xu
@ 2021-06-07 16:45     ` Pavel Begunkov
  0 siblings, 0 replies; 4+ messages in thread
From: Pavel Begunkov @ 2021-06-07 16:45 UTC (permalink / raw)
  To: Hao Xu, Jens Axboe; +Cc: io-uring, Joseph Qi

On 6/6/21 3:08 PM, Hao Xu wrote:
> 在 2021/6/4 上午2:01, Jens Axboe 写道:
>> On 6/3/21 6:53 AM, Hao Xu wrote:
>>> Hi Jens,
>>> I've a question about poll_multi_file in io_do_iopoll().
>>> It keeps spinning in f_op->iopoll() if poll_multi_file is
>>> true (and we're under the requested amount). But in my
>>> understanding, reqs may be in different hardware queues
>>> for blk-mq device even in this situation.
>>> Should we consider the hardware queue number as well? Some
>>> thing like below:
>>
>> That looks reasonable to me - do you have any performance
>> numbers to go with it?
> 
> Not very easy for me to construct a good case. I'm trying to
> mock the below situation:
> manully control uring reqs to go to 2 hardware queues, like:
>    hw_queue0     hw_queue1
>    heavy_req     simple_req
>    heavy_req     simple_req
>      ...            ...
> 
> heavy_req is some request that needs more time to complete,
> while simple_req takes less time. And make the io_do_iopoll()
> alway first spin on hw_queue0.
> any ideas?

- NVMe with #HW qs >= #CPUs, so HW to SW qs are 1-to-1.
- 2 threads pinned to different CPUs, so they submit to
different qs.

Then one thread is doing 512B rand reads, and the second
is doing 64-128 KB rand reads. So, I'd expect a latency
spike on some nine. Not tested, so just a suggestion.

The second can also be doing writes, but that would need
1) waiting for steady state
2) higher QD/load for writes because otherwise SSD
caches might hide waiting.

-- 
Pavel Begunkov

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2021-06-07 16:45 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-06-03 12:53 Question about poll_multi_file Hao Xu
2021-06-03 18:01 ` Jens Axboe
2021-06-06 14:08   ` Hao Xu
2021-06-07 16:45     ` Pavel Begunkov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox