Re: [PATCH 0/4] iopoll support for io_uring/nvme passthrough

[Jens Axboe <axboe@kernel.dk>]

On 8/7/22 11:58 AM, Kanchan Joshi wrote:
> On Fri, Aug 05, 2022 at 12:15:24PM -0600, Jens Axboe wrote:
>> On 8/5/22 12:11 PM, Keith Busch wrote:
>>> On Fri, Aug 05, 2022 at 11:18:38AM -0600, Jens Axboe wrote:
>>>> On 8/5/22 11:04 AM, Jens Axboe wrote:
>>>>> On 8/5/22 9:42 AM, Kanchan Joshi wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Series enables async polling on io_uring command, and nvme passthrough
>>>>>> (for io-commands) is wired up to leverage that.
>>>>>>
>>>>>> 512b randread performance (KIOP) below:
>>>>>>
>>>>>> QD_batch    block    passthru    passthru-poll   block-poll
>>>>>> 1_1          80        81          158            157
>>>>>> 8_2         406       470          680            700
>>>>>> 16_4        620       656          931            920
>>>>>> 128_32      879       1056        1120            1132
>>>>>
>>>>> Curious on why passthru is slower than block-poll? Are we missing
>>>>> something here?
>>>>
>>>> I took a quick peek, running it here. List of items making it slower:
>>>>
>>>> - No fixedbufs support for passthru, each each request will go through
>>>>   get_user_pages() and put_pages() on completion. This is about a 10%
>>>>   change for me, by itself.
>>>
>>> Enabling fixed buffer support through here looks like it will take a
>>> little bit of work. The driver needs an opcode or flag to tell it the
>>> user address is a fixed buffer, and io_uring needs to export its
>>> registered buffer for a driver like nvme to get to.
>>
>> Yeah, it's not a straight forward thing. But if this will be used with
>> recycled buffers, then it'll definitely be worthwhile to look into.
> 
> Had posted bio-cache and fixedbufs in the initial round but retracted
> to get the foundation settled first.
> https://lore.kernel.org/linux-nvme/20220308152105.309618-1-joshi.k@samsung.com/
> 
> I see that you brought back bio-cache already. I can refresh fixedbufs.

Excellent, yes please bring back the fixedbufs. It's a 5-10% win,
nothing to sneeze at.

> Completion-batching seems too tightly coupled to block-path.

It's really not, in fact it'd be even simpler to do for passthru. The
rq->end_io handler just needs to know about it.

-- 
Jens Axboe