Re: [PATCH v3 1/2] io_uring: avoid whole io_wq_work copy for requests completed inline

[Xiaoguang Wang <xiaoguang.wang@linux.alibaba.com>]

hi，

> On 5/28/20 3:15 AM, Xiaoguang Wang wrote:
>> If requests can be submitted and completed inline, we don't need to
>> initialize whole io_wq_work in io_init_req(), which is an expensive
>> operation, add a new 'REQ_F_WORK_INITIALIZED' to control whether
>> io_wq_work is initialized.
>>
>> I use /dev/nullb0 to evaluate performance improvement in my physical
>> machine:
>>    modprobe null_blk nr_devices=1 completion_nsec=0
>>    sudo taskset -c 60 fio  -name=fiotest -filename=/dev/nullb0 -iodepth=128
>>    -thread -rw=read -ioengine=io_uring -direct=1 -bs=4k -size=100G -numjobs=1
>>    -time_based -runtime=120
>>
>> before this patch:
>> Run status group 0 (all jobs):
>>     READ: bw=724MiB/s (759MB/s), 724MiB/s-724MiB/s (759MB/s-759MB/s),
>>     io=84.8GiB (91.1GB), run=120001-120001msec
>>
>> With this patch:
>> Run status group 0 (all jobs):
>>     READ: bw=761MiB/s (798MB/s), 761MiB/s-761MiB/s (798MB/s-798MB/s),
>>     io=89.2GiB (95.8GB), run=120001-120001msec
>>
>> About 5% improvement.
> 
> I think this is a big enough of a win to warrant looking closer
> at this. Just a quick comment from me so far:
Yeah, to be honest, I did't expect that we get this some obvious improvement.
But I have run multiple rounds of same tests, I always get similar improvement,
if you have some free time, you can have a test :)

> 
>> @@ -2923,7 +2943,10 @@ static int io_fsync(struct io_kiocb *req, bool force_nonblock)
>>   {
>>   	/* fsync always requires a blocking context */
>>   	if (force_nonblock) {
>> -		req->work.func = io_fsync_finish;
>> +		if (!(req->flags & REQ_F_WORK_INITIALIZED))
>> +			init_io_work(req, io_fsync_finish);
>> +		else
>> +			req->work.func = io_fsync_finish;
> 
> This pattern is repeated enough to warrant a helper, ala:
> 
> static void io_req_init_async(req, func)
> {
> 	if (req->flags & REQ_F_WORK_INITIALIZED)
> 		req->work.func = func;
> 	else
> 		init_io_work(req, func);
> }
> 
> also swapped the conditions, I tend to find it easier to read without
> the negation.
Thanks for your suggestions. I'll prepare a V4 soon.

Regards,
Xiaoguang Wang

>