Re: [PATCH 3/3] block: enable bio allocation cache for IRQ driven IO

[Jens Axboe <axboe@kernel.dk>]

On 12/16/21 7:33 AM, Christoph Hellwig wrote:
> On Wed, Dec 15, 2021 at 09:30:09AM -0700, Jens Axboe wrote:
>> We currently cannot use the bio recycling allocation cache for IRQ driven
>> IO, as the cache isn't IRQ safe (by design).
>>
>> Add a way for the completion side to pass back a bio that needs freeing,
>> so we can do it from the io_uring side. io_uring completions always
>> run in task context.
>>
>> This is good for about a 13% improvement in IRQ driven IO, taking us from
>> around 6.3M/core to 7.1M/core IOPS.
> 
> The numbers looks great, but I really hate how it ties the caller into
> using a bio.  I'll have to think hard about a better structure.

Yeah it's definitely not the prettiest, but I haven't been able to come
up with a better approach so far. I don't think the bio association is
the worst, can't imagine we'd want to tie it to something else. I might
be wrong...

Ideally we'd flip the completion model upside down here, instead of
having ->bi_end_io() call ->ki_complete.

> Just curious:  are the numbers with retpolines or without?  Do you care
> about the cost of indirect calls with retpolines for these benchmarks?

No mitigations enabled for these tests, so no retpoline. I definitely do
care about indirect call performance. The peak testing so far has been
mostly focused on best case - various features disabled that are common
but costly, and no mitigations enabled. Mostly because it's necessary to
break down the issues one-by-one, and the intent has always been to move
towards making common options less expensive too. Most of the patches
cover both bases, but there are also cases where we want to fix just
items that are costly when features/mitigations are enabled.


-- 
Jens Axboe