[Report] annoyed dma debug warning "cacheline tracking EEXIST, overlapping mappings aren't supported"

[Report] annoyed dma debug warning "cacheline tracking EEXIST, overlapping mappings aren't supported"

[Ming Lei]

Hello Guys,

I got more and more reports on DMA debug warning "cacheline tracking EEXIST,
overlapping mappings aren't supported" in storage related tests:

1) liburing
- test/iopoll-overflow.t
- test/sq-poll-dup.t

Same buffer is used in more than 1 IO.

2) raid1 driver

- same buffer is used in more than 1 bio

3) some storage utilities
- dm thin provisioning utility of thin_check
- `dt`(https://github.com/RobinTMiller/dt)

I looks like same user buffer is used in more than 1 dio.

4) some self cooked test code which does same thing with 1)

In storage stack, the buffer provider is far away from the actual DMA
controller operating code, which doesn't have the knowledge if
DMA_ATTR_SKIP_CPU_SYNC should be set.

And suggestions for avoiding this noise?

Thanks,
Ming

Re: [Report] annoyed dma debug warning "cacheline tracking EEXIST, overlapping mappings aren't supported"

[Hannes Reinecke]

On 10/14/24 03:27, Ming Lei wrote:
> Hello Guys,
> 
> I got more and more reports on DMA debug warning "cacheline tracking EEXIST,
> overlapping mappings aren't supported" in storage related tests:
> 
> 1) liburing
> - test/iopoll-overflow.t
> - test/sq-poll-dup.t
> 
> Same buffer is used in more than 1 IO.
> 
> 2) raid1 driver
> 
> - same buffer is used in more than 1 bio
> 
> 3) some storage utilities
> - dm thin provisioning utility of thin_check
> - `dt`(https://github.com/RobinTMiller/dt)
> 
> I looks like same user buffer is used in more than 1 dio.
> 
> 4) some self cooked test code which does same thing with 1)
> 
> In storage stack, the buffer provider is far away from the actual DMA
> controller operating code, which doesn't have the knowledge if
> DMA_ATTR_SKIP_CPU_SYNC should be set.
> 
> And suggestions for avoiding this noise?
> 
Can you check if this is the NULL page? Operations like 'discard' will 
create bios with several bvecs all pointing to the same NULL page.
That would be the most obvious culprit.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

Re: [Report] annoyed dma debug warning "cacheline tracking EEXIST, overlapping mappings aren't supported"

[Christoph Hellwig]

On Mon, Oct 14, 2024 at 09:23:14AM +0200, Hannes Reinecke wrote:
>> 3) some storage utilities
>> - dm thin provisioning utility of thin_check
>> - `dt`(https://github.com/RobinTMiller/dt)
>>
>> I looks like same user buffer is used in more than 1 dio.
>>
>> 4) some self cooked test code which does same thing with 1)
>>
>> In storage stack, the buffer provider is far away from the actual DMA
>> controller operating code, which doesn't have the knowledge if
>> DMA_ATTR_SKIP_CPU_SYNC should be set.
>>
>> And suggestions for avoiding this noise?
>>
> Can you check if this is the NULL page? Operations like 'discard' will 
> create bios with several bvecs all pointing to the same NULL page.
> That would be the most obvious culprit.

The only case I fully understand without looking into the details
is raid1, and that will obviously map the same data multiple times
because it writes it out multiple time.  Now mapping a buffer
multiple times for a DMA_TO_DEVICE is relatively harmless in
practice as the data is transferred to the device, but it it
still breaks the dma buffer ownership model in the dma which is
really helpful to find bugs where people don't think about this
at all.  Not sure if there is any good solution here.

Re: [Report] annoyed dma debug warning "cacheline tracking EEXIST, overlapping mappings aren't supported"

[Ming Lei]

On Mon, Oct 14, 2024 at 09:41:51AM +0200, Christoph Hellwig wrote:
> On Mon, Oct 14, 2024 at 09:23:14AM +0200, Hannes Reinecke wrote:
> >> 3) some storage utilities
> >> - dm thin provisioning utility of thin_check
> >> - `dt`(https://github.com/RobinTMiller/dt)
> >>
> >> I looks like same user buffer is used in more than 1 dio.
> >>
> >> 4) some self cooked test code which does same thing with 1)
> >>
> >> In storage stack, the buffer provider is far away from the actual DMA
> >> controller operating code, which doesn't have the knowledge if
> >> DMA_ATTR_SKIP_CPU_SYNC should be set.
> >>
> >> And suggestions for avoiding this noise?
> >>
> > Can you check if this is the NULL page? Operations like 'discard' will 
> > create bios with several bvecs all pointing to the same NULL page.
> > That would be the most obvious culprit.
> 
> The only case I fully understand without looking into the details
> is raid1, and that will obviously map the same data multiple times

The other cases should be concurrent DIOs on same userspace buffer.


Thanks,
Ming

Re: [Report] annoyed dma debug warning "cacheline tracking EEXIST, overlapping mappings aren't supported"

[Jens Axboe]

On 10/13/24 7:27 PM, Ming Lei wrote:
> Hello Guys,
> 
> I got more and more reports on DMA debug warning "cacheline tracking EEXIST,
> overlapping mappings aren't supported" in storage related tests:

This is easily triggerable with dio and reusing a buffer. But it looks
like this will only trigger if CONFIG_DMA_API_DEBUG is set, so should
not be a prod issue? If so, I'd say that debug printk should be killed.

Maybe turn it into a pr_warn_once() or something?

-- 
Jens Axboe

Re: [Report] annoyed dma debug warning "cacheline tracking EEXIST, overlapping mappings aren't supported"

[Robin Murphy]

On 14/10/2024 8:58 am, Ming Lei wrote:
> On Mon, Oct 14, 2024 at 09:41:51AM +0200, Christoph Hellwig wrote:
>> On Mon, Oct 14, 2024 at 09:23:14AM +0200, Hannes Reinecke wrote:
>>>> 3) some storage utilities
>>>> - dm thin provisioning utility of thin_check
>>>> - `dt`(https://github.com/RobinTMiller/dt)
>>>>
>>>> I looks like same user buffer is used in more than 1 dio.
>>>>
>>>> 4) some self cooked test code which does same thing with 1)
>>>>
>>>> In storage stack, the buffer provider is far away from the actual DMA
>>>> controller operating code, which doesn't have the knowledge if
>>>> DMA_ATTR_SKIP_CPU_SYNC should be set.
>>>>
>>>> And suggestions for avoiding this noise?
>>>>
>>> Can you check if this is the NULL page? Operations like 'discard' will
>>> create bios with several bvecs all pointing to the same NULL page.
>>> That would be the most obvious culprit.
>>
>> The only case I fully understand without looking into the details
>> is raid1, and that will obviously map the same data multiple times
> 
> The other cases should be concurrent DIOs on same userspace buffer.

active_cacheline_insert() does already bail out for DMA_TO_DEVICE, so it 
returning -EEXIST to tickle the warning would seem to genuinely imply 
these are DMA mappings requesting to *write* the same cacheline 
concurrently, which is indeed broken in general.

Thanks,
Robin.

Re: [Report] annoyed dma debug warning "cacheline tracking EEXIST, overlapping mappings aren't supported"

[Ming Lei]

On Mon, Oct 14, 2024 at 07:09:08PM +0100, Robin Murphy wrote:
> On 14/10/2024 8:58 am, Ming Lei wrote:
> > On Mon, Oct 14, 2024 at 09:41:51AM +0200, Christoph Hellwig wrote:
> > > On Mon, Oct 14, 2024 at 09:23:14AM +0200, Hannes Reinecke wrote:
> > > > > 3) some storage utilities
> > > > > - dm thin provisioning utility of thin_check
> > > > > - `dt`(https://github.com/RobinTMiller/dt)
> > > > > 
> > > > > I looks like same user buffer is used in more than 1 dio.
> > > > > 
> > > > > 4) some self cooked test code which does same thing with 1)
> > > > > 
> > > > > In storage stack, the buffer provider is far away from the actual DMA
> > > > > controller operating code, which doesn't have the knowledge if
> > > > > DMA_ATTR_SKIP_CPU_SYNC should be set.
> > > > > 
> > > > > And suggestions for avoiding this noise?
> > > > > 
> > > > Can you check if this is the NULL page? Operations like 'discard' will
> > > > create bios with several bvecs all pointing to the same NULL page.
> > > > That would be the most obvious culprit.
> > > 
> > > The only case I fully understand without looking into the details
> > > is raid1, and that will obviously map the same data multiple times
> > 
> > The other cases should be concurrent DIOs on same userspace buffer.
> 
> active_cacheline_insert() does already bail out for DMA_TO_DEVICE, so it
> returning -EEXIST to tickle the warning would seem to genuinely imply these
> are DMA mappings requesting to *write* the same cacheline concurrently,
> which is indeed broken in general.

The two io_uring tests are READ, and the dm thin_check are READ too.

For the raid1 case, the warning is from raid1_sync_request() which may
have both READ/WRITE IO.

Thanks,
Ming

Re: [Report] annoyed dma debug warning "cacheline tracking EEXIST, overlapping mappings aren't supported"

[Dan Williams]

Ming Lei wrote:
> On Mon, Oct 14, 2024 at 07:09:08PM +0100, Robin Murphy wrote:
> > On 14/10/2024 8:58 am, Ming Lei wrote:
> > > On Mon, Oct 14, 2024 at 09:41:51AM +0200, Christoph Hellwig wrote:
> > > > On Mon, Oct 14, 2024 at 09:23:14AM +0200, Hannes Reinecke wrote:
> > > > > > 3) some storage utilities
> > > > > > - dm thin provisioning utility of thin_check
> > > > > > - `dt`(https://github.com/RobinTMiller/dt)
> > > > > > 
> > > > > > I looks like same user buffer is used in more than 1 dio.
> > > > > > 
> > > > > > 4) some self cooked test code which does same thing with 1)
> > > > > > 
> > > > > > In storage stack, the buffer provider is far away from the actual DMA
> > > > > > controller operating code, which doesn't have the knowledge if
> > > > > > DMA_ATTR_SKIP_CPU_SYNC should be set.
> > > > > > 
> > > > > > And suggestions for avoiding this noise?
> > > > > > 
> > > > > Can you check if this is the NULL page? Operations like 'discard' will
> > > > > create bios with several bvecs all pointing to the same NULL page.
> > > > > That would be the most obvious culprit.
> > > > 
> > > > The only case I fully understand without looking into the details
> > > > is raid1, and that will obviously map the same data multiple times
> > > 
> > > The other cases should be concurrent DIOs on same userspace buffer.
> > 
> > active_cacheline_insert() does already bail out for DMA_TO_DEVICE, so it
> > returning -EEXIST to tickle the warning would seem to genuinely imply these
> > are DMA mappings requesting to *write* the same cacheline concurrently,
> > which is indeed broken in general.
> 
> The two io_uring tests are READ, and the dm thin_check are READ too.

"READ from the device" == "WRITE to the page" (DMA_FROM_DEVICE).

> For the raid1 case, the warning is from raid1_sync_request() which may
> have both READ/WRITE IO.

I don't see an easy way out of this without instrumenting archs that
can not support overlapping mappings to opt-in to bounce buffering for
these cases.

Archs that can support this can skip the opt-in and quiet this test, but
some of the value is being able to catch boundary conditions on more
widely available systems.

Re: [Report] annoyed dma debug warning "cacheline tracking EEXIST, overlapping mappings aren't supported"

[Ming Lei]

On Mon, Oct 14, 2024 at 09:41:51AM +0200, Christoph Hellwig wrote:
> On Mon, Oct 14, 2024 at 09:23:14AM +0200, Hannes Reinecke wrote:
> >> 3) some storage utilities
> >> - dm thin provisioning utility of thin_check
> >> - `dt`(https://github.com/RobinTMiller/dt)
> >>
> >> I looks like same user buffer is used in more than 1 dio.
> >>
> >> 4) some self cooked test code which does same thing with 1)
> >>
> >> In storage stack, the buffer provider is far away from the actual DMA
> >> controller operating code, which doesn't have the knowledge if
> >> DMA_ATTR_SKIP_CPU_SYNC should be set.
> >>
> >> And suggestions for avoiding this noise?
> >>
> > Can you check if this is the NULL page? Operations like 'discard' will 
> > create bios with several bvecs all pointing to the same NULL page.
> > That would be the most obvious culprit.
> 
> The only case I fully understand without looking into the details
> is raid1, and that will obviously map the same data multiple times
> because it writes it out multiple time.  Now mapping a buffer
> multiple times for a DMA_TO_DEVICE is relatively harmless in
> practice as the data is transferred to the device, but it it
> still breaks the dma buffer ownership model in the dma which is
> really helpful to find bugs where people don't think about this
> at all.  Not sure if there is any good solution here.
>

Another related topic:

Recently direct IO buffer alignment changes to just respect DMA
controller alignment which is often too relax, such as dma_alignment
is just 3 for many host controllers, then two direct IO buffers may
cross same DMA mapping cache line.

Is this one real problem?


Thanks,
Ming

Re: [Report] annoyed dma debug warning "cacheline tracking EEXIST, overlapping mappings aren't supported"

[Christoph Hellwig]

On Mon, Oct 14, 2024 at 07:09:08PM +0100, Robin Murphy wrote:
>>> The only case I fully understand without looking into the details
>>> is raid1, and that will obviously map the same data multiple times
>>
>> The other cases should be concurrent DIOs on same userspace buffer.
>
> active_cacheline_insert() does already bail out for DMA_TO_DEVICE, so it 
> returning -EEXIST to tickle the warning would seem to genuinely imply these 
> are DMA mappings requesting to *write* the same cacheline concurrently, 
> which is indeed broken in general.

Yes, active_cacheline_insert only complains for FROM_DEVICE or
BIDIRECTIONAL mappings.  I can't see how raid 1 would trigger that
given that it only reads from one leg at a time.

Ming, can you look a bit more into what is happening here?

Re: [Report] annoyed dma debug warning "cacheline tracking EEXIST, overlapping mappings aren't supported"

[Ming Lei]

On Tue, Oct 15, 2024 at 06:54:13AM +0200, Christoph Hellwig wrote:
> On Mon, Oct 14, 2024 at 07:09:08PM +0100, Robin Murphy wrote:
> >>> The only case I fully understand without looking into the details
> >>> is raid1, and that will obviously map the same data multiple times
> >>
> >> The other cases should be concurrent DIOs on same userspace buffer.
> >
> > active_cacheline_insert() does already bail out for DMA_TO_DEVICE, so it 
> > returning -EEXIST to tickle the warning would seem to genuinely imply these 
> > are DMA mappings requesting to *write* the same cacheline concurrently, 
> > which is indeed broken in general.
> 
> Yes, active_cacheline_insert only complains for FROM_DEVICE or
> BIDIRECTIONAL mappings.  I can't see how raid 1 would trigger that
> given that it only reads from one leg at a time.
> 
> Ming, can you look a bit more into what is happening here?

All should be READ IO which is FROM_DEVICE, please see my reply:

https://lore.kernel.org/linux-block/Zw3MZrK_l7DuFfFd@fedora/

And the raid1 warning is actually from raid1_sync_request().


Thanks,
Ming

Re: [Report] annoyed dma debug warning "cacheline tracking EEXIST, overlapping mappings aren't supported"

[Christoph Hellwig]

On Tue, Oct 15, 2024 at 03:40:26PM +0800, Ming Lei wrote:
> > Yes, active_cacheline_insert only complains for FROM_DEVICE or
> > BIDIRECTIONAL mappings.  I can't see how raid 1 would trigger that
> > given that it only reads from one leg at a time.
> > 
> > Ming, can you look a bit more into what is happening here?
> 
> All should be READ IO which is FROM_DEVICE, please see my reply:

Yes, reads translate to DMA_FROM_DEVICE.

> https://lore.kernel.org/linux-block/Zw3MZrK_l7DuFfFd@fedora/
> 
> And the raid1 warning is actually from raid1_sync_request().

In that case the warnings are perfectly valid because the I/O patterns
will create data corruption on non-coherent architectures.  For direct
I/O from userspace the kernel can't prevent it, but for raid1 we should
be able to do something better.  As raid1_sync_request is a convoluted
and undocumented mess I don't have a straigh shot answer to what it is
doing (wrong) and how to fix it unfortunately.

> 
> 
> Thanks,
> Ming
---end quoted text---