Re: [ISSUE] The time cost of IOSQE_IO_LINK

["Carter Li 李通洲" <carter.li@eoitek.com>]

Another suggestion: we should always try completing operations inline
unless IOSQE_ASYNC is specified, no matter if the operations are preceded
by a poll.

Carter

> 2020年2月13日 下午11:51，Carter Li 李通洲 <carter.li@eoitek.com> 写道：
> 
> 
> 
>> 2020年2月13日 下午11:14，Jens Axboe <axboe@kernel.dk> 写道：
>> 
>> On 2/13/20 8:08 AM, Pavel Begunkov wrote:
>>> On 2/13/2020 3:33 AM, Carter Li 李通洲 wrote:
>>>> Thanks for your reply.
>>>> 
>>>> You are right the nop isn't really a good test case. But I actually
>>>> found this issue when benchmarking my echo server, which didn't use
>>>> NOP of course.
>>> 
>>> If there are no hidden subtle issues in io_uring, your benchmark or the
>>> used pattern itself, it's probably because of overhead on async punting
>>> (copying iovecs, several extra switches, refcounts, grabbing mm/fs/etc,
>>> io-wq itself).
>>> 
>>> I was going to tune async/punting stuff anyway, so I'll look into this.
>>> And of course, there is always a good chance Jens have some bright insights
>> 
>> The main issue here is that if you do the poll->recv, then it'll be
>> an automatic punt of the recv to async context when the poll completes.
>> That's regardless of whether or not we can complete the poll inline,
>> we never attempt to recv inline from that completion. This is in contrast
>> to doing a separate poll, getting the notification, then doing another
>> sqe and io_uring_enter to perform the recv. For this case, we end up
>> doing everything inline, just with the cost of an additional system call
>> to submit the new recv.
>> 
>> It'd be really cool if we could improve on this situation, as recv (or
>> read) preceded by a poll is indeed a common use case. Or ditto for the
>> write side.
>> 
>>> BTW, what's benefit of doing poll(fd)->read(fd), but not directly read()?
>> 
>> If there's no data to begin with, then the read will go async. Hence
>> it'll be a switch to a worker thread. The above should avoid it, but
>> it doesn't.
> 
> Yes. I actually tested `directly read()` first, and found it was about 30%
> slower then poll(fd)->read(fd).
> 
> https://github.com/axboe/liburing/issues/69
> 
> So it turns out that async punting has high overhead. A (silly) question:
> could we implement read/write operations that would block as poll->read/write?
> 
> 
>> 
>> For carter's sake, it's worth nothing that the poll command is special
>> and normal requests would be more efficient with links. We just need
>> to work on making the poll linked with read/write perform much better.
> 
> Thanks
> 
>> 
>> -- 
>> Jens Axboe