Question about poll_multi_file

Question about poll_multi_file

[Hao Xu]

Hi Jens,
I've a question about poll_multi_file in io_do_iopoll().
It keeps spinning in f_op->iopoll() if poll_multi_file is
true (and we're under the requested amount). But in my
understanding, reqs may be in different hardware queues
for blk-mq device even in this situation.
Should we consider the hardware queue number as well? Some
thing like below:

diff --git a/fs/io_uring.c b/fs/io_uring.c
index 7b0e6bc9ea3d..479a75022449 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -2533,10 +2533,13 @@ static void io_iopoll_req_issued(struct io_kiocb 
*req, bool in_async)
                 ctx->poll_multi_file = false;
         } else if (!ctx->poll_multi_file) {
                 struct io_kiocb *list_req;
+               unsigned int queue_num0, queue_num1;

                 list_req = list_first_entry(&ctx->iopoll_list, struct 
io_kiocb,
                                                 inflight_entry);
-               if (list_req->file != req->file)
+               queue_num0 = 
blk_qc_t_to_queue_num(list_req->rw.kiocb.ki_cookie);
+               queue_num1 = blk_qc_t_to_queue_num(req->rw.kiocb.ki_cookie);
+               if (list_req->file != req->file || queue_num0 != queue_num1)
                         ctx->poll_multi_file = true;
         }

Re: Question about poll_multi_file

[Jens Axboe]

On 6/3/21 6:53 AM, Hao Xu wrote:
> Hi Jens,
> I've a question about poll_multi_file in io_do_iopoll().
> It keeps spinning in f_op->iopoll() if poll_multi_file is
> true (and we're under the requested amount). But in my
> understanding, reqs may be in different hardware queues
> for blk-mq device even in this situation.
> Should we consider the hardware queue number as well? Some
> thing like below:

That looks reasonable to me - do you have any performance
numbers to go with it?

-- 
Jens Axboe

Re: Question about poll_multi_file

[Hao Xu]

在 2021/6/4 上午2:01, Jens Axboe 写道:
> On 6/3/21 6:53 AM, Hao Xu wrote:
>> Hi Jens,
>> I've a question about poll_multi_file in io_do_iopoll().
>> It keeps spinning in f_op->iopoll() if poll_multi_file is
>> true (and we're under the requested amount). But in my
>> understanding, reqs may be in different hardware queues
>> for blk-mq device even in this situation.
>> Should we consider the hardware queue number as well? Some
>> thing like below:
> 
> That looks reasonable to me - do you have any performance
> numbers to go with it?

Not very easy for me to construct a good case. I'm trying to
mock the below situation:
manully control uring reqs to go to 2 hardware queues, like:
    hw_queue0     hw_queue1
    heavy_req     simple_req
    heavy_req     simple_req
      ...            ...

heavy_req is some request that needs more time to complete,
while simple_req takes less time. And make the io_do_iopoll()
alway first spin on hw_queue0.
any ideas?

Thanks,
Hao
> 

Re: Question about poll_multi_file

[Pavel Begunkov]

On 6/6/21 3:08 PM, Hao Xu wrote:
> 在 2021/6/4 上午2:01, Jens Axboe 写道:
>> On 6/3/21 6:53 AM, Hao Xu wrote:
>>> Hi Jens,
>>> I've a question about poll_multi_file in io_do_iopoll().
>>> It keeps spinning in f_op->iopoll() if poll_multi_file is
>>> true (and we're under the requested amount). But in my
>>> understanding, reqs may be in different hardware queues
>>> for blk-mq device even in this situation.
>>> Should we consider the hardware queue number as well? Some
>>> thing like below:
>>
>> That looks reasonable to me - do you have any performance
>> numbers to go with it?
> 
> Not very easy for me to construct a good case. I'm trying to
> mock the below situation:
> manully control uring reqs to go to 2 hardware queues, like:
>    hw_queue0     hw_queue1
>    heavy_req     simple_req
>    heavy_req     simple_req
>      ...            ...
> 
> heavy_req is some request that needs more time to complete,
> while simple_req takes less time. And make the io_do_iopoll()
> alway first spin on hw_queue0.
> any ideas?

- NVMe with #HW qs >= #CPUs, so HW to SW qs are 1-to-1.
- 2 threads pinned to different CPUs, so they submit to
different qs.

Then one thread is doing 512B rand reads, and the second
is doing 64-128 KB rand reads. So, I'd expect a latency
spike on some nine. Not tested, so just a suggestion.

The second can also be doing writes, but that would need
1) waiting for steady state
2) higher QD/load for writes because otherwise SSD
caches might hide waiting.

-- 
Pavel Begunkov