Re: VU NVMe commadns with io_uring? NVME_URING_CMD_IO

[Keith Busch <kbusch@kernel.org>]

On Thu, Apr 18, 2024 at 05:24:15PM +0000, Doug.Coatney@microchip.com wrote:
> HI Folks!
> 
> Trying to send down some vendor unique NVMe commands with io_uring and fixed buffers via NVME_URING_CMD_IO. 
> 
> I have no issues sending standard NVMe read/write requests, but we are working on some VU commands which have
> buffer and length in different command payload offsets. So I'm trying to find where io_uring populates the NVMe cmd 
> payload for a "write" (outbound) or "read" inbound request. 
> 
> This percolates into drivers/nvme/host/ioctl.c: nvme_uring_cmd_io() as expected where io_uring_cmd points to the 
> expected cmd payload and a local struct nvme_command is created to be encoded into a user request via 
> nvme_alloc_user_request() call. 
> 
> Within the nvme_alloc_user_request() call nvme_init_request() is called which performs a 
> memcpy(nvme_req(req)->cmd,cmd, sizeof(*cmd));  which is basically copying the local 
> struct nvme_command to the request structure. The issue though is the buffer and length 
> are not populated in the cmd payload at this time. 
>
> Instead this happens somewhere else at I/O processing time which I haven't found yet. 
> I'm trying to track that location down and was wondering if I'm on the right path here or not. 

Right after nvme_init_request(), the data payload is attached to the
struct request in nvme_map_user_request(), which works for both fixed
and unregistered buffers.

The payload gets attached to the nvme command later after dispatching
the request through the block layer. Assuming your nvme is PCI, that
happens in nvme_map_data().

> Any suggestions on where this update occurs would be incredibly useful. Also we're hoping 
> to have multiple fixed buffers for a specific command specified within the cmd payload 
> In order to provide for extended key data on the commands. Essentially providing KV support 
> for store/retrieve of keys larger than what can fit in the command payload. 
> 
> We're on a path to use fixed buffers and to be able to map them on demand for the VU 
> commands, but inserting them into the command payload is the missing point at present.