Re: Fixed buffers have out-dated content

[Jens Axboe <axboe@kernel.dk>]

On 1/14/21 2:50 PM, Martin Raiber wrote:
> On 10.01.2021 17:50 Martin Raiber wrote:
>> On 09.01.2021 21:32 Pavel Begunkov wrote:
>>> On 09/01/2021 16:58, Martin Raiber wrote:
>>>> On 09.01.2021 17:23 Jens Axboe wrote:
>>>>> On 1/8/21 4:39 PM, Martin Raiber wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I have a gnarly issue with io_uring and fixed buffers (fixed
>>>>>> read/write). It seems the contents of those buffers contain old 
>>>>>> data in
>>>>>> some rare cases under memory pressure after a read/during a write.
>>>>>>
>>>>>> Specifically I use io_uring with fuse and to confirm this is not some
>>>>>> user space issue let fuse print the unique id it adds to each 
>>>>>> request.
>>>>>> Fuse adds this request data to a pipe, and when the pipe buffer is 
>>>>>> later
>>>>>> copied to the io_uring fixed buffer it has the id of a fuse request
>>>>>> returned earlier using the same buffer while returning the size of 
>>>>>> the
>>>>>> new request. Or I set the unique id in the buffer, write it to 
>>>>>> fuse (via
>>>>>> writing to a pipe, then splicing) and then fuse returns with e.g.
>>>>>> ENOENT, because the unique id is not correct because in kernel it 
>>>>>> reads
>>>>>> the id of the previous, already completed, request using this buffer.
>>>>>>
>>>>>> To make reproducing this faster running memtester (which mlocks a
>>>>>> configurable amount of memory) with a large amount of user memory 
>>>>>> every
>>>>>> 30s helps. So it has something to do with swapping? It seems to not
>>>>>> occur if no swap space is active. Problem occurs without warning when
>>>>>> the kernel is build with KASAN and slab debugging.
>>>>>>
>>>>>> If I don't use the _FIXED opcodes (which is easy to do), the problem
>>>>>> does not occur.
>>>>>>
>>>>>> Problem occurs with 5.9.16 and 5.10.5.
>>>>> Can you mention more about what kind of IO you are doing, I'm assuming
>>>>> it's O_DIRECT? I'll see if I can reproduce this.
>>>> It's writing to/reading from pipes (nonblocking, no O_DIRECT).
>>> A blind guess, does it handle short reads and writes? If not, can you
>>> check whether they happen or not?
>>
>> Something like this was what I suspected at first as well. It does 
>> check for short read/writes and I added (unnecessary -- because the 
>> fuse request structure is 40 bytes and it does io in page sizes) code 
>> for retrying short reads at some point. I also checked for the pipes 
>> to be empty before they are used at some point and let the kernel log 
>> allocation failures (idea was that it was short pipe read/writes 
>> because of allocation failure or that something doesn't get rewound 
>> properly in this case). Beyond that three things that make a user 
>> space problem unlikely:
>>
>>  - occurs only when using fixed buffers and does not occur when 
>> running same code without fixed buffer opcodes
>>  - doesn't occur when there is no memory pressure
>>  - I added print(k/f) logging that pointed me in this direction as well
>>
>>>> I can reproduce it with https://github.com/uroni/fuseuring on e.g. a 
>>>> 2GB VPS. Modify bench.sh so that fio loops. Add swap, then run 1400M 
>>>> memtester while it runs (so it swaps, I guess). I can try further 
>>>> reducing the reproducer, but I wanted to avoid that work in case it 
>>>> is something obvious. The next step would be to remove fuse from the 
>>>> equation -- it does try to move the pages from the pipe when 
>>>> splicing to it, for example.
> 
> When I use 5.10.7 with 09854ba94c6aad7886996bfbee2530b3d8a7f4f4 ("mm: 
> do_wp_page() simplification"), 1a0cf26323c80e2f1c58fc04f15686de61bfab0c 
> ("mm/ksm: Remove reuse_ksm_page()") and 
> be068f29034fb00530a053d18b8cf140c32b12b3 ("mm: fix misplaced unlock_page 
> in do_wp_page()") reverted the issue doesn't seem to occur.

Linus, I'm looking into the above report, all of it should still be
intact in the quoted section. Figured it would not hurt to loop you in,
in case this is a generic problem and since Martin identified that
reverting the above changes on the mm side makes it go away. Maybe
there's already some discussion elsewhere about it that I'm not privy
to.

-- 
Jens Axboe