Re: [RFC PATCH] misc: fgds: enable GPU-NVMe direct I/O via POSIX and io_uring

["Christian König" <christian.koenig@amd.com>]

On 9/10/26 05:57, Li Wang wrote:
...
>> AMD came up with something similar, but all those approaches are so fundamentally broken that we didn't even considered upstreaming it.
>>> And you are using this as a "bypass" for the normal accel subsystem,
>>> shouldn't this be part of that subsystem instead of a custom user/kernel
>>> api like you are creating here?
>>
>> As far as I know there is a patch set under review and even already partially merged which enables exactly that functionality as general feature for DMA-buf which is vendor independent and should at least in theory work with all drivers.
>>
>> I'm really surprised that somebody is still working on the vendor specific stuff.
> As you pointed out, every vendor has been inventing their own way and interfaces to support GDS,
> introducing custom kernel modules and proprietary UAPI interfaces, with varying performance that
> leaves developers heavily frustrated. Apologies for not making this clear enough in our commit
> messages, which understandably caused some confusion. We merely borrowed the name "GDS" to describe
> the functional purpose of fgds.
> 
> In fact, we believe fgds offers four key advantages:
> (1) GPU platform independence;
> (2) POSIX/io_uring interface compatibility;
> (3) Higher performance than GDS;
> (4) Minimal kernel footprint and UAPI footprint
> 
> Regarding (1), (2), and (3), please allow me to briefly explain the design mechanism of fgds:
> fgds turns a GPU memory buffer into a POSIX/io_uring-compatible user-space virtual address via
> three main steps:
> 
> Step 1: Utilizing ZONE_DEVICE support, we remap the GPU memory exposed via PCIe BAR into struct pages
> using devm_memremap_pages();
> 
> Step 2: Utilizing dma-buf support, the GPU memory buffer is exported as a dma-buf file descriptor (fd).
> Using this fd as a bridge, we look up the corresponding DMA addresses for the GPU memory buffer inside
> the kernel;
> 
> Step 3: Through mmap, we insert the struct pages corresponding to the GPU memory buffer into the userspace
> VMA, mapping their physical/DMA addresses directly. The virtual address returned by mmap can then be directly
> passed into standard POSIX or io_uring interfaces.

Well long story short what you do here is completely broken.

Approaches like those have been suggested before and we added both documentation as well as code to prevent such hacks from working.

Please see Pavel Begunkov patch set on the LKML which adds DMA-buf support to io_uring for how to do it correctly. Just google for "Add dmabuf read/write via io_uring".

Regards,
Christian.