Is it safe to submit and reap IOs in different threads?

Is it safe to submit and reap IOs in different threads?

[Bhatia, Sumeet]

Hello,

My application has a thread per disk (aka producer_thread) that generates disk operations. I have io-uring context per disk to ensure iouring submissions are from a single thread.

producer_thread executes only if new disk operations are to be submitted else it yields. It'll be significant code change to modify this behavior. For this reason I spin up a new thread (aka consumer thread) per io-uring context to reap IO completions.

My reading of fs/io_uring.c suggests it is safe to submit IOs from producer_thread and reap IOs from consumer_thread. My prototype based on liburing (Ref: https://pastebin.com/6u2FZB0D) works fine too. 

I would like to get your thoughts on whether this approach is indeed safe or am I overlooking any race condition?

Thanks,
Sumeet

Re: Is it safe to submit and reap IOs in different threads?

[Pavel Begunkov]

On 07/05/2020 02:02, Bhatia, Sumeet wrote:
> Hello,
> 
> My application has a thread per disk (aka producer_thread) that generates disk operations. I have io-uring context per disk to ensure iouring submissions are from a single thread.
> 
> producer_thread executes only if new disk operations are to be submitted else it yields. It'll be significant code change to modify this behavior. For this reason I spin up a new thread (aka consumer thread) per io-uring context to reap IO completions.
> 
> My reading of fs/io_uring.c suggests it is safe to submit IOs from producer_thread and reap IOs from consumer_thread. My prototype based on liburing (Ref: https://pastebin.com/6u2FZB0D) works fine too. 
> 
> I would like to get your thoughts on whether this approach is indeed safe or am I overlooking any race condition?

Shortly, if you don't mix up submissions and completions, it's safe. There is
only pitfall I found in liburing -- you shouldn't pass a timeout in wait
functions (e.g. io_uring_wait_cqes()), which would do submission.

Also take a look:
https://github.com/axboe/liburing/issues/108#issuecomment-619616721

-- 
Pavel Begunkov