Re: [PATCH V3 00/16] io_uring/ublk: add IORING_OP_FUSED_CMD

[Ming Lei <ming.lei@redhat.com>]

On Sat, Mar 18, 2023 at 10:09:52AM -0600, Jens Axboe wrote:
> On 3/14/23 6:57?AM, Ming Lei wrote:
> > Basically userspace can specify any sub-buffer of the ublk block request
> > buffer from the fused command just by setting 'offset/len'
> > in the slave SQE for running slave OP. This way is flexible to implement
> > io mapping: mirror, stripped, ...
> > 
> > The 3th & 4th patches enable fused slave support for the following OPs:
> > 
> > 	OP_READ/OP_WRITE
> > 	OP_SEND/OP_RECV/OP_SEND_ZC
> > 
> > The other ublk patches cleans ublk driver and implement fused command
> > for supporting zero copy.
> > 
> > Follows userspace code:
> > 
> > https://github.com/ming1/ubdsrv/tree/fused-cmd-zc-v2
> 
> Ran some quick testing here with qcow2. This is just done on my laptop
> in kvm, so take them with a grain of salt, results may be better
> elsewhere.
> 
> Basline:
> 
> 64k reads       98-100K IOPS    6-6.1GB/sec     (ublk 100%, io_uring 9%)
> 4k reads        670-680K IOPS   2.6GB/sec       (ublk 65%, io_uring 44%)
> 
> and with zerocopy enabled:
> 
> 64k reads       184K IOPS       11.5GB/sec      (ublk 91%, io_uring 12%)
> 4k reads        730K IOPS       2.8GB/sec       (ublk 73%, io_uring 48%)

There are other ways to observe the boost:

1) loop over file in tmpfs
- 1~2X in my test

2) nbd with local nbd server(nbdkit memory )
- less than 1X in my test

3) null
- which won't call into fused command, but can evaluate page copy cost
- 5+X in my test

> 
> and with zerocopy and using SINGLE_ISSUER|COOP_TASKRUN for the ring:
> 
> 64k reads       205K IOPS       12.8GB/sec      (ublk 91%, io_uring 12%)
> 4k reads        730K IOPS       2.8GB/sec       (ublk 66%, io_uring 42%)

Looks SINGLE_ISSUER|COOP_TASKRUN can get ~10% improvement, will look it.


Thanks,
Ming