VU NVMe commadns with io_uring? NVME_URING_CMD_IO

VU NVMe commadns with io_uring? NVME_URING_CMD_IO

[Doug.Coatney]

HI Folks!

Trying to send down some vendor unique NVMe commands with io_uring and fixed buffers via NVME_URING_CMD_IO. 

I have no issues sending standard NVMe read/write requests, but we are working on some VU commands which have
buffer and length in different command payload offsets. So I'm trying to find where io_uring populates the NVMe cmd 
payload for a "write" (outbound) or "read" inbound request. 

This percolates into drivers/nvme/host/ioctl.c: nvme_uring_cmd_io() as expected where io_uring_cmd points to the 
expected cmd payload and a local struct nvme_command is created to be encoded into a user request via 
nvme_alloc_user_request() call. 

Within the nvme_alloc_user_request() call nvme_init_request() is called which performs a 
memcpy(nvme_req(req)->cmd,cmd, sizeof(*cmd));  which is basically copying the local 
struct nvme_command to the request structure. The issue though is the buffer and length 
are not populated in the cmd payload at this time. 

Instead this happens somewhere else at I/O processing time which I haven't found yet. 
I'm trying to track that location down and was wondering if I'm on the right path here or not. 
Any suggestions on where this update occurs would be incredibly useful. Also we're hoping 
to have multiple fixed buffers for a specific command specified within the cmd payload 
In order to provide for extended key data on the commands. Essentially providing KV support 
for store/retrieve of keys larger than what can fit in the command payload. 

We're on a path to use fixed buffers and to be able to map them on demand for the VU 
commands, but inserting them into the command payload is the missing point at present. 

Also are there any future plans to support VU commands within io_uring? Trying to get the 
infrastructure in place to allow that now. The long term goal would be to support a KV 
command set effectively in io_uring, but in the meantime provide it via the passthru capability. 

Thanks!
Doug


Doug Coatney
Associate Technical Fellow
Data Center Solutions Engineering
 
MICROCHIP
408.888.0105 Mobile Phone
doug.coatney@microchip.com
  
            

Re: VU NVMe commadns with io_uring? NVME_URING_CMD_IO

[Keith Busch]

On Thu, Apr 18, 2024 at 05:24:15PM +0000, Doug.Coatney@microchip.com wrote:
> HI Folks!
> 
> Trying to send down some vendor unique NVMe commands with io_uring and fixed buffers via NVME_URING_CMD_IO. 
> 
> I have no issues sending standard NVMe read/write requests, but we are working on some VU commands which have
> buffer and length in different command payload offsets. So I'm trying to find where io_uring populates the NVMe cmd 
> payload for a "write" (outbound) or "read" inbound request. 
> 
> This percolates into drivers/nvme/host/ioctl.c: nvme_uring_cmd_io() as expected where io_uring_cmd points to the 
> expected cmd payload and a local struct nvme_command is created to be encoded into a user request via 
> nvme_alloc_user_request() call. 
> 
> Within the nvme_alloc_user_request() call nvme_init_request() is called which performs a 
> memcpy(nvme_req(req)->cmd,cmd, sizeof(*cmd));  which is basically copying the local 
> struct nvme_command to the request structure. The issue though is the buffer and length 
> are not populated in the cmd payload at this time. 
>
> Instead this happens somewhere else at I/O processing time which I haven't found yet. 
> I'm trying to track that location down and was wondering if I'm on the right path here or not. 

Right after nvme_init_request(), the data payload is attached to the
struct request in nvme_map_user_request(), which works for both fixed
and unregistered buffers.

The payload gets attached to the nvme command later after dispatching
the request through the block layer. Assuming your nvme is PCI, that
happens in nvme_map_data().

> Any suggestions on where this update occurs would be incredibly useful. Also we're hoping 
> to have multiple fixed buffers for a specific command specified within the cmd payload 
> In order to provide for extended key data on the commands. Essentially providing KV support 
> for store/retrieve of keys larger than what can fit in the command payload. 
> 
> We're on a path to use fixed buffers and to be able to map them on demand for the VU 
> commands, but inserting them into the command payload is the missing point at present.