Non sequential linked chains and IO_LINK support

Non sequential linked chains and IO_LINK support

[Bhatia, Sumeet]

Hello everyone,

I've been exploring iouring to submit disk operations. My application generates disk operations based on some events and operations are unknown until those events occur.  Some of these disk operations are interdependent others are not. 

Example: Following operations are generated and submitted before any of them are complete
operation_0 (independent operation)
operation_1 (independent operation),​
operation_2 (to be issued only if operation_0 was successful),
operation_3 (independent operation),
operation_4 (to be issued only if operation_1 was successful)

In my example I have two independent link chains, (operation_0, operation_2) and (operation_1, operation_4).  iouring documentation suggests IOSQE_IO_LINK expects link chains to be sequential and will not support my use case. 

I explored creating new iouring context for each of these linked chains. But it turns out depending on disk size there can be somewhere between 500-1000 such chains. I'm not sure whether it is prudent to create that many iouring contexts.

I am reaching out to check whether there would be a generic need to support nonsequential linked chains on a single iouring context. Would love to hear all your thoughts.

Thanks,
Sumeet

Re: Non sequential linked chains and IO_LINK support

[Pavel Begunkov]

On 07/05/2020 00:46, Bhatia, Sumeet wrote:
> Hello everyone,
> 
> I've been exploring iouring to submit disk operations. My application generates disk operations based on some events and operations are unknown until those events occur.  Some of these disk operations are interdependent others are not. 
> 
> Example: Following operations are generated and submitted before any of them are complete
> operation_0 (independent operation)
> operation_1 (independent operation),​
> operation_2 (to be issued only if operation_0 was successful),
> operation_3 (independent operation),
> operation_4 (to be issued only if operation_1 was successful)
> 
> In my example I have two independent link chains, (operation_0, operation_2) and (operation_1, operation_4).  iouring documentation suggests IOSQE_IO_LINK expects link chains to be sequential and will not support my use case. 

First of all, there shouldn't be a submission (i.e. io_uring_enter(to_submit>0))
between adding linked requests to a submission queue (SQ). It'd be racy otherwise.

E.g. you can't do:

add_sqe(op0)
submit(op0)
add_sqe(op2, linked)

Though the following is valid, as we don't submit op0:

add_sqe(opX)
add_sqe(op0)
submit(up until opX)
add_sqe(op2, linked)


And that means you can reorder them just before submitting, or filing them into
the SQ in a better order.

Is it helpful? Let's figure out how to cover your case.


> I explored creating new iouring context for each of these linked chains. But it turns out depending on disk size there can be somewhere between 500-1000 such chains. I'm not sure whether it is prudent to create that many iouring contexts.

Then you would need to wait on them (e.g. epoll or 1000 threads), and that would
defeat the whole idea. In any case even with sharing io-wq and having small CQ
and SQ, it'd be wasteful keeping many resources duplicated.

> 
> I am reaching out to check whether there would be a generic need to support nonsequential linked chains on a single iouring context. Would love to hear all your thoughts.
> 
> Thanks,
> Sumeet
> 

-- 
Pavel Begunkov

Re: Non sequential linked chains and IO_LINK support

[Bhatia, Sumeet]

Thank you for the response!

Unfortunately when the application submits operation_0 it has no way of determining if/when operation_2 would be generated. 

For now I plan to maintain a list of outstanding operations. If operation_2 gets generated while operation_0 is in flight the application will hold its submission until operation_0 is completed. 

I wanted to check whether this would be a generic use case and would warrant native support in iouring?

Thanks,
Sumeet
________________________________________
From: Pavel Begunkov <asml.silence@gmail.com>
Sent: Wednesday, May 6, 2020 6:11 PM
To: Bhatia, Sumeet; io-uring@vger.kernel.org
Cc: Hegde, Pramod; Jens Axboe
Subject: RE: [EXTERNAL] Non sequential linked chains and IO_LINK support

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



On 07/05/2020 00:46, Bhatia, Sumeet wrote:
> Hello everyone,
>
> I've been exploring iouring to submit disk operations. My application generates disk operations based on some events and operations are unknown until those events occur.  Some of these disk operations are interdependent others are not.
>
> Example: Following operations are generated and submitted before any of them are complete
> operation_0 (independent operation)
> operation_1 (independent operation),​
> operation_2 (to be issued only if operation_0 was successful),
> operation_3 (independent operation),
> operation_4 (to be issued only if operation_1 was successful)
>
> In my example I have two independent link chains, (operation_0, operation_2) and (operation_1, operation_4).  iouring documentation suggests IOSQE_IO_LINK expects link chains to be sequential and will not support my use case.

First of all, there shouldn't be a submission (i.e. io_uring_enter(to_submit>0))
between adding linked requests to a submission queue (SQ). It'd be racy otherwise.

E.g. you can't do:

add_sqe(op0)
submit(op0)
add_sqe(op2, linked)

Though the following is valid, as we don't submit op0:

add_sqe(opX)
add_sqe(op0)
submit(up until opX)
add_sqe(op2, linked)


And that means you can reorder them just before submitting, or filing them into
the SQ in a better order.

Is it helpful? Let's figure out how to cover your case.


> I explored creating new iouring context for each of these linked chains. But it turns out depending on disk size there can be somewhere between 500-1000 such chains. I'm not sure whether it is prudent to create that many iouring contexts.

Then you would need to wait on them (e.g. epoll or 1000 threads), and that would
defeat the whole idea. In any case even with sharing io-wq and having small CQ
and SQ, it'd be wasteful keeping many resources duplicated.

>
> I am reaching out to check whether there would be a generic need to support nonsequential linked chains on a single iouring context. Would love to hear all your thoughts.
>
> Thanks,
> Sumeet
>

--
Pavel Begunkov

Re: Non sequential linked chains and IO_LINK support

[Pavel Begunkov]

On 07/05/2020 04:04, Bhatia, Sumeet wrote:
> Thank you for the response!
> 
> Unfortunately when the application submits operation_0 it has no way of determining if/when operation_2 would be generated. 
> 
> For now I plan to maintain a list of outstanding operations. If operation_2 gets generated while operation_0 is in flight the application will hold its submission until operation_0 is completed. 

Yes, that's the general workflow. Links in current form are not very helpful for
your case. Those who don't care about latency are trying to throttle requests a
bit, to collect several of them and send at once.

> I wanted to check whether this would be a generic use case and would warrant native support in iouring?

Trying to link to a request that was already submitted, would just complicate
the kernel and userspace as well, I don't think it's viable to support that,
unless there will be huge benefit.

On the other hand, this is an interesting case for our BPF ideas.
E.g. you load a BPF program, which on request completion will poll your internal
queue for a specific disk and submit requests from there. Kind of custom
polling. We don't have BPF in io_uring yet, though.


> Thanks,
> Sumeet
> ________________________________________
> From: Pavel Begunkov <asml.silence@gmail.com>
> Sent: Wednesday, May 6, 2020 6:11 PM
> To: Bhatia, Sumeet; io-uring@vger.kernel.org
> Cc: Hegde, Pramod; Jens Axboe
> Subject: RE: [EXTERNAL] Non sequential linked chains and IO_LINK support
> 
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> 
> 
> 
> On 07/05/2020 00:46, Bhatia, Sumeet wrote:
>> Hello everyone,
>>
>> I've been exploring iouring to submit disk operations. My application generates disk operations based on some events and operations are unknown until those events occur.  Some of these disk operations are interdependent others are not.
>>
>> Example: Following operations are generated and submitted before any of them are complete
>> operation_0 (independent operation)
>> operation_1 (independent operation),​
>> operation_2 (to be issued only if operation_0 was successful),
>> operation_3 (independent operation),
>> operation_4 (to be issued only if operation_1 was successful)
>>
>> In my example I have two independent link chains, (operation_0, operation_2) and (operation_1, operation_4).  iouring documentation suggests IOSQE_IO_LINK expects link chains to be sequential and will not support my use case.
> 
> First of all, there shouldn't be a submission (i.e. io_uring_enter(to_submit>0))
> between adding linked requests to a submission queue (SQ). It'd be racy otherwise.
> 
> E.g. you can't do:
> 
> add_sqe(op0)
> submit(op0)
> add_sqe(op2, linked)
> 
> Though the following is valid, as we don't submit op0:
> 
> add_sqe(opX)
> add_sqe(op0)
> submit(up until opX)
> add_sqe(op2, linked)
> 
> 
> And that means you can reorder them just before submitting, or filing them into
> the SQ in a better order.
> 
> Is it helpful? Let's figure out how to cover your case.
> 
> 
>> I explored creating new iouring context for each of these linked chains. But it turns out depending on disk size there can be somewhere between 500-1000 such chains. I'm not sure whether it is prudent to create that many iouring contexts.
> 
> Then you would need to wait on them (e.g. epoll or 1000 threads), and that would
> defeat the whole idea. In any case even with sharing io-wq and having small CQ
> and SQ, it'd be wasteful keeping many resources duplicated.
> 
>>
>> I am reaching out to check whether there would be a generic need to support nonsequential linked chains on a single iouring context. Would love to hear all your thoughts.
>>
>> Thanks,
>> Sumeet
>>
> 
> --
> Pavel Begunkov
> 

-- 
Pavel Begunkov