Re: [PATCH V3 00/16] io_uring/ublk: add IORING_OP_FUSED_CMD

[Ming Lei <ming.lei@redhat.com>]

On Sat, Mar 18, 2023 at 04:23:35PM +0000, Pavel Begunkov wrote:
> On 3/16/23 03:13, Xiaoguang Wang wrote:
> > > Add IORING_OP_FUSED_CMD, it is one special URING_CMD, which has to
> > > be SQE128. The 1st SQE(master) is one 64byte URING_CMD, and the 2nd
> > > 64byte SQE(slave) is another normal 64byte OP. For any OP which needs
> > > to support slave OP, io_issue_defs[op].fused_slave needs to be set as 1,
> > > and its ->issue() can retrieve/import buffer from master request's
> > > fused_cmd_kbuf. The slave OP is actually submitted from kernel, part of
> > > this idea is from Xiaoguang's ublk ebpf patchset, but this patchset
> > > submits slave OP just like normal OP issued from userspace, that said,
> > > SQE order is kept, and batching handling is done too.
> > Thanks for this great work, seems that we're now in the right direction
> > to support ublk zero copy, I believe this feature will improve io throughput
> > greatly and reduce ublk's cpu resource usage.
> > 
> > I have gone through your 2th patch, and have some little concerns here:
> > Say we have one ublk loop target device, but it has 4 backend files,
> > every file will carry 25% of device capacity and it's implemented in stripped
> > way, then for every io request, current implementation will need issed 4
> > fused_cmd, right? 4 slave sqes are necessary, but it would be better to
> > have just one master sqe, so I wonder whether we can have another
> > method. The key point is to let io_uring support register various kernel
> > memory objects, which come from kernel, such as ITER_BVEC or
> > ITER_KVEC. so how about below actions:
> > 1. add a new infrastructure in io_uring, which will support to register
> > various kernel memory objects in it, this new infrastructure could be
> > maintained in a xarray structure, every memory objects in it will have
> > a unique id. This registration could be done in a ublk uring cmd, io_uring
> > offers registration interface.
> > 2. then any sqe can use these memory objects freely, so long as it
> > passes above unique id in sqe properly.
> > Above are just rough ideas, just for your reference.
> 
> It precisely hints on what I proposed a bit earlier, that makes
> me not alone thinking that it's a good idea to have a design allowing
> 1) multiple ops using a buffer and 2) to limiting it to one single
> submission because the userspace might want to preprocess a part
> of the data, multiplex it or on the opposite divide. I was mostly
> coming from non ublk cases, and one example would be such zc recv,
> parsing the app level headers and redirecting the rest of the data
> somewhere.

Just get some time to think about zc recv.

Firstly I understand the buffer shouldn't be provided from userspace unlike
storage, given network recv can happen any time, and NIC driver has to put
received data into kernel socket recv buffer first. But if yes for some special recv
case, the use case is totally different with ublk, and impossible to share
any code with ublk.

So here suppose the zc recv means to export socket recv buffer to userspace
just like the implementation in lwn doc [1].

[1] https://lwn.net/Articles/752188/

But how does userspace pre-process this kernel buffer? mmap is expensive,
and copy won't be one option. Or the data is just simply forwarded to
somewhere(special case)?

If yes, it can become a bit similar with ublk's case in which
the device io buffer needn't to be modified and just simply forwarded to
FS or socket in most of cases. Then it could be possible to extend fused
for supporting it given the buffer lifetime model is useful for generic zero
copy, such as:

- send fused command(A) to just register buffer(socket recv buffer) with one
empty buffer index, then return the buffer index to userspace via CQE(
IORING_CQE_F_MORE), but not complete this fused command(A); but it
requires socket FS to implement ->uring_command() for providing recv
buffer.

- after getting recv SQE, userspace can use the registered buffer to
do whatever, but direct access on buffer is one problem, since it is
simply pages which have to be mapped for handling from userspace

- after userspace handles everything(includes net send over this buffer) on
the recv buffer, send another fused command or new OP to ask kernel to
release buffer by completing fused command(A). However, one corner case
is that this fuse command needs to be completed automatically when
io_uring exits since app is dead at that time.

It should be easy to extend fused command in above way(slave less)
since V4 starts to support normal 64byte SQE, and we have enough uring
command flags.

But not sure if that is what you need. If not, please explain a bit
your exact requirement.

> 
> I haven't got a chance to work on it but will return to it in
> a week. The discussion was here:
> 
> https://lore.kernel.org/all/ce96f7e7-1315-7154-f540-1a3ff0215674@gmail.com/


Thanks,
Ming