[PATCH] io_uring: wakeup threads waiting for EPOLLOUT events

[PATCH] io_uring: wakeup threads waiting for EPOLLOUT events

[Stefano Garzarella]

io_uring_poll() sets EPOLLOUT flag if there is space in the
SQ ring, then we should wakeup threads waiting for EPOLLOUT
events when we expose the new SQ head to the userspace.

Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
---

Do you think is better to change the name of 'cq_wait' and 'cq_fasync'?

Thanks,
Stefano
---
 fs/io_uring.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/fs/io_uring.c b/fs/io_uring.c
index 38b54051facd..5c6ff5f9e741 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -3687,6 +3687,11 @@ static void io_commit_sqring(struct io_ring_ctx *ctx)
 		 * write new data to them.
 		 */
 		smp_store_release(&rings->sq.head, ctx->cached_sq_head);
+
+		if (wq_has_sleeper(&ctx->cq_wait)) {
+			wake_up_interruptible(&ctx->cq_wait);
+			kill_fasync(&ctx->cq_fasync, SIGIO, POLL_OUT);
+		}
 	}
 }
 
-- 
2.24.1

Re: [PATCH] io_uring: wakeup threads waiting for EPOLLOUT events

[Jens Axboe]

On 1/16/20 6:49 AM, Stefano Garzarella wrote:
> io_uring_poll() sets EPOLLOUT flag if there is space in the
> SQ ring, then we should wakeup threads waiting for EPOLLOUT
> events when we expose the new SQ head to the userspace.
> 
> Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
> ---
> 
> Do you think is better to change the name of 'cq_wait' and 'cq_fasync'?

I honestly think it'd be better to have separate waits for in/out poll,
the below patch will introduce some unfortunate cacheline traffic
between the submitter and completer side.

-- 
Jens Axboe

Re: [PATCH] io_uring: wakeup threads waiting for EPOLLOUT events

[Stefano Garzarella]

On Thu, Jan 16, 2020 at 08:29:07AM -0700, Jens Axboe wrote:
> On 1/16/20 6:49 AM, Stefano Garzarella wrote:
> > io_uring_poll() sets EPOLLOUT flag if there is space in the
> > SQ ring, then we should wakeup threads waiting for EPOLLOUT
> > events when we expose the new SQ head to the userspace.
> > 
> > Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
> > ---
> > 
> > Do you think is better to change the name of 'cq_wait' and 'cq_fasync'?
> 
> I honestly think it'd be better to have separate waits for in/out poll,
> the below patch will introduce some unfortunate cacheline traffic
> between the submitter and completer side.

Agree, make sense. I'll send a v2 with a new 'sq_wait'.

About fasync, do you think could be useful the POLL_OUT support?
In this case, maybe is not simple to have two separate fasync_struct,
do you have any advice?

Thanks,
Stefano

Re: [PATCH] io_uring: wakeup threads waiting for EPOLLOUT events

[Jens Axboe]

On 1/16/20 8:55 AM, Stefano Garzarella wrote:
> On Thu, Jan 16, 2020 at 08:29:07AM -0700, Jens Axboe wrote:
>> On 1/16/20 6:49 AM, Stefano Garzarella wrote:
>>> io_uring_poll() sets EPOLLOUT flag if there is space in the
>>> SQ ring, then we should wakeup threads waiting for EPOLLOUT
>>> events when we expose the new SQ head to the userspace.
>>>
>>> Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
>>> ---
>>>
>>> Do you think is better to change the name of 'cq_wait' and 'cq_fasync'?
>>
>> I honestly think it'd be better to have separate waits for in/out poll,
>> the below patch will introduce some unfortunate cacheline traffic
>> between the submitter and completer side.
> 
> Agree, make sense. I'll send a v2 with a new 'sq_wait'.
> 
> About fasync, do you think could be useful the POLL_OUT support?
> In this case, maybe is not simple to have two separate fasync_struct,
> do you have any advice?

The fasync should not matter, it's all in the checking of whether the sq
side has any sleepers. This is rarely going to be the case, so as long
as we can keep the check cheap, then I think we're fine.

Since the use case is mostly single submitter, unless you're doing
something funky or unusual, you're not going to be needing POLLOUT ever.
Hence I don't want to add any cost for it, I'd even advocate just doing
waitqueue_active() perhaps, if we can safely pull it off.

-- 
Jens Axboe

Re: [PATCH] io_uring: wakeup threads waiting for EPOLLOUT events

[Stefano Garzarella]

On Thu, Jan 16, 2020 at 09:00:24AM -0700, Jens Axboe wrote:
> On 1/16/20 8:55 AM, Stefano Garzarella wrote:
> > On Thu, Jan 16, 2020 at 08:29:07AM -0700, Jens Axboe wrote:
> >> On 1/16/20 6:49 AM, Stefano Garzarella wrote:
> >>> io_uring_poll() sets EPOLLOUT flag if there is space in the
> >>> SQ ring, then we should wakeup threads waiting for EPOLLOUT
> >>> events when we expose the new SQ head to the userspace.
> >>>
> >>> Signed-off-by: Stefano Garzarella <sgarzare@redhat.com>
> >>> ---
> >>>
> >>> Do you think is better to change the name of 'cq_wait' and 'cq_fasync'?
> >>
> >> I honestly think it'd be better to have separate waits for in/out poll,
> >> the below patch will introduce some unfortunate cacheline traffic
> >> between the submitter and completer side.
> > 
> > Agree, make sense. I'll send a v2 with a new 'sq_wait'.
> > 
> > About fasync, do you think could be useful the POLL_OUT support?
> > In this case, maybe is not simple to have two separate fasync_struct,
> > do you have any advice?
> 
> The fasync should not matter, it's all in the checking of whether the sq
> side has any sleepers. This is rarely going to be the case, so as long
> as we can keep the check cheap, then I think we're fine.

Right.

> 
> Since the use case is mostly single submitter, unless you're doing
> something funky or unusual, you're not going to be needing POLLOUT ever.

The case that I had in mind was with kernel side polling enabled and
a single submitter that can use epoll() to wait free slots in the SQ
ring. (I don't have a test, maybe I can write one...)

> Hence I don't want to add any cost for it, I'd even advocate just doing
> waitqueue_active() perhaps, if we can safely pull it off.

I'll try!

Thanks,
Stefano

Re: [PATCH] io_uring: wakeup threads waiting for EPOLLOUT events

[Jens Axboe]

On 1/16/20 9:26 AM, Stefano Garzarella wrote:
>> Since the use case is mostly single submitter, unless you're doing
>> something funky or unusual, you're not going to be needing POLLOUT ever.
> 
> The case that I had in mind was with kernel side polling enabled and
> a single submitter that can use epoll() to wait free slots in the SQ
> ring. (I don't have a test, maybe I can write one...)

Right, I think that's the only use case where it makes sense, because
you have someone else draining the sq side for you. A test case would
indeed be nice, liburing has a good arsenal of test cases and this would
be a good addition!

-- 
Jens Axboe

Re: [PATCH] io_uring: wakeup threads waiting for EPOLLOUT events

[Stefano Garzarella]

On Thu, Jan 16, 2020 at 09:30:12AM -0700, Jens Axboe wrote:
> On 1/16/20 9:26 AM, Stefano Garzarella wrote:
> >> Since the use case is mostly single submitter, unless you're doing
> >> something funky or unusual, you're not going to be needing POLLOUT ever.
> > 
> > The case that I had in mind was with kernel side polling enabled and
> > a single submitter that can use epoll() to wait free slots in the SQ
> > ring. (I don't have a test, maybe I can write one...)
> 
> Right, I think that's the only use case where it makes sense, because
> you have someone else draining the sq side for you. A test case would
> indeed be nice, liburing has a good arsenal of test cases and this would
> be a good addition!

Sure, I'll send a test to liburing for this case!

Thanks,
Stefano

Re: [PATCH] io_uring: wakeup threads waiting for EPOLLOUT events

[Jens Axboe]

On 1/16/20 10:03 AM, Stefano Garzarella wrote:
> On Thu, Jan 16, 2020 at 09:30:12AM -0700, Jens Axboe wrote:
>> On 1/16/20 9:26 AM, Stefano Garzarella wrote:
>>>> Since the use case is mostly single submitter, unless you're doing
>>>> something funky or unusual, you're not going to be needing POLLOUT ever.
>>>
>>> The case that I had in mind was with kernel side polling enabled and
>>> a single submitter that can use epoll() to wait free slots in the SQ
>>> ring. (I don't have a test, maybe I can write one...)
>>
>> Right, I think that's the only use case where it makes sense, because
>> you have someone else draining the sq side for you. A test case would
>> indeed be nice, liburing has a good arsenal of test cases and this would
>> be a good addition!
> 
> Sure, I'll send a test to liburing for this case!

Gentle ping on the test case :-)

-- 
Jens Axboe

Re: [PATCH] io_uring: wakeup threads waiting for EPOLLOUT events

[Stefano Garzarella]

On Thu, Jan 23, 2020 at 12:13:57PM -0700, Jens Axboe wrote:
> On 1/16/20 10:03 AM, Stefano Garzarella wrote:
> > On Thu, Jan 16, 2020 at 09:30:12AM -0700, Jens Axboe wrote:
> >> On 1/16/20 9:26 AM, Stefano Garzarella wrote:
> >>>> Since the use case is mostly single submitter, unless you're doing
> >>>> something funky or unusual, you're not going to be needing POLLOUT ever.
> >>>
> >>> The case that I had in mind was with kernel side polling enabled and
> >>> a single submitter that can use epoll() to wait free slots in the SQ
> >>> ring. (I don't have a test, maybe I can write one...)
> >>
> >> Right, I think that's the only use case where it makes sense, because
> >> you have someone else draining the sq side for you. A test case would
> >> indeed be nice, liburing has a good arsenal of test cases and this would
> >> be a good addition!
> > 
> > Sure, I'll send a test to liburing for this case!
> 
> Gentle ping on the test case :-)
> 

Yes, you are right :-)

I was a little busy this week to finish some works before DevConf.
I hope to work on the test case these days, so by Monday I hope I have it ;-)

Cheers,
Stefano

Re: [PATCH] io_uring: wakeup threads waiting for EPOLLOUT events

[Jens Axboe]

On 1/23/20 2:45 PM, Stefano Garzarella wrote:
> On Thu, Jan 23, 2020 at 12:13:57PM -0700, Jens Axboe wrote:
>> On 1/16/20 10:03 AM, Stefano Garzarella wrote:
>>> On Thu, Jan 16, 2020 at 09:30:12AM -0700, Jens Axboe wrote:
>>>> On 1/16/20 9:26 AM, Stefano Garzarella wrote:
>>>>>> Since the use case is mostly single submitter, unless you're doing
>>>>>> something funky or unusual, you're not going to be needing POLLOUT ever.
>>>>>
>>>>> The case that I had in mind was with kernel side polling enabled and
>>>>> a single submitter that can use epoll() to wait free slots in the SQ
>>>>> ring. (I don't have a test, maybe I can write one...)
>>>>
>>>> Right, I think that's the only use case where it makes sense, because
>>>> you have someone else draining the sq side for you. A test case would
>>>> indeed be nice, liburing has a good arsenal of test cases and this would
>>>> be a good addition!
>>>
>>> Sure, I'll send a test to liburing for this case!
>>
>> Gentle ping on the test case :-)
>>
> 
> Yes, you are right :-)
> 
> I was a little busy this week to finish some works before DevConf.  I
> hope to work on the test case these days, so by Monday I hope I have
> it ;-)

Thanks, all good, just a gentle nudge ;-)

-- 
Jens Axboe