[RFC] do_hashed and wq enqueue

[RFC] do_hashed and wq enqueue

[Pavel Begunkov]

[-- Attachment #1.1: Type: text/plain, Size: 476 bytes --]

Hi,

I've been looking for hashed io-wq enqueuing, but not fully understand the
issue. As I remember, it was something about getting around blocking for
buffered I/O. Is that so? Is there any write-up of a thread for this issue?

My case is 2-fd request, and I'm not sure how it should look. For splice() it's
probably Ok to hash the non-pipe end (if any), but I wonder how it should work,
if, for example, the restriction will be removed.


-- 
Pavel Begunkov


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [RFC] do_hashed and wq enqueue

[Jens Axboe]

On 2/11/20 1:20 PM, Pavel Begunkov wrote:
> Hi,
> 
> I've been looking for hashed io-wq enqueuing, but not fully understand the
> issue. As I remember, it was something about getting around blocking for
> buffered I/O. Is that so? Is there any write-up of a thread for this issue?

Not sure if there's a writeup, but the issue is that buffered writes all
need to grab the per-inode mutex. Hence if you don't serialize these writes,
you end up having potentially quite a few threads banging on the same mutex.
This causes a high level of lock contention (and scheduling). By serializing
by inode hash we avoid that, and yield the same performance.

Dave Chinner is working on lifting this restriction, at which point we'll
have to gate the hashing based on whether or not the fs is smart or dumb
when it comes to buffered writes and locking.

> My case is 2-fd request, and I'm not sure how it should look. For splice() it's
> probably Ok to hash the non-pipe end (if any), but I wonder how it should work,
> if, for example, the restriction will be removed.

Probably just do the same, but for the output fd only (and following the
same restrictions in terms of file type).

-- 
Jens Axboe

Re: [RFC] do_hashed and wq enqueue

[Pavel Begunkov]

[-- Attachment #1.1: Type: text/plain, Size: 1320 bytes --]

On 11/02/2020 23:30, Jens Axboe wrote:
> On 2/11/20 1:20 PM, Pavel Begunkov wrote:
>> Hi,
>>
>> I've been looking for hashed io-wq enqueuing, but not fully understand the
>> issue. As I remember, it was something about getting around blocking for
>> buffered I/O. Is that so? Is there any write-up of a thread for this issue?
> 
> Not sure if there's a writeup, but the issue is that buffered writes all
> need to grab the per-inode mutex. Hence if you don't serialize these writes,
> you end up having potentially quite a few threads banging on the same mutex.
> This causes a high level of lock contention (and scheduling). By serializing
> by inode hash we avoid that, and yield the same performance.
> 
> Dave Chinner is working on lifting this restriction, at which point we'll
> have to gate the hashing based on whether or not the fs is smart or dumb
> when it comes to buffered writes and locking.

Got it, thanks!

>> My case is 2-fd request, and I'm not sure how it should look. For splice() it's
>> probably Ok to hash the non-pipe end (if any), but I wonder how it should work,
>> if, for example, the restriction will be removed.
> 
> Probably just do the same, but for the output fd only (and following the
> same restrictions in terms of file type).
> 

-- 
Pavel Begunkov


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]