Re: [PATCH 5/6] io_uring: add support for futex wake and wait

[Jens Axboe <axboe@kernel.dk>]

On 6/12/23 10:06?AM, Gabriel Krisman Bertazi wrote:
> Jens Axboe <axboe@kernel.dk> writes:
> 
>> Add support for FUTEX_WAKE/WAIT primitives.
> 
> This is great.  I was so sure io_uring had this support already for some
> reason.  I might have dreamed it.

I think you did :-)

> The semantics are tricky, though. You might want to CC peterZ and tglx
> directly.

For sure, I'll take it wider soon enough. Just wanted to iron out
io_uring details first.

>> IORING_OP_FUTEX_WAKE is mix of FUTEX_WAKE and FUTEX_WAKE_BITSET, as
>> it does support passing in a bitset.
> 
> As far as I know, the _BITSET variant are not commonly used in the
> current interface.  I haven't seen any code that really benefits from
> it.

Since FUTEX_WAKE is a strict subset of FUTEX_WAKE_BITSET, makes little
sense to not just support both imho.

>> Similary, IORING_OP_FUTEX_WAIT is a mix of FUTEX_WAIT and
>> FUTEX_WAIT_BITSET.
> 
> But it is definitely safe to have a single one, basically with the
> _BITSET semantics.

Yep I think so.

>> FUTEX_WAKE is straight forward, as we can always just do those inline.
>> FUTEX_WAIT will queue the futex with an appropriate callback, and
>> that callback will in turn post a CQE when it has triggered.
> 
> Even with an asynchronous model, it might make sense to halt execution
> of further queued operations until futex completes.  I think
> IOSQE_IO_DRAIN is a barrier only against the submission part, so it
> wouldn't hep.  Is there a way to ensure this ordering?

You'd use link for that - link whatever depends on the wake to the futex
wait. Or just queue it up once you reap the wait completion, when that
is posted because we got woken.

> I know, it goes against the asynchronous nature of io_uring, but I think
> it might be a valid use case. Say we extend FUTEX_WAIT with a way to
> acquire the futex in kernel space.  Then, when the CQE returns, we know
> the lock is acquired.  if we can queue dependencies on that (stronger
> than the link semantics), we could queue operations to be executed once
> the lock is taken. Makes sense?

It does, and acquiring it _may_ make sense indeed. But I'd rather punt
that to a later thing, and focus on getting the standard (and smaller)
primitives done first.

>> Cancelations are supported, both from the application point-of-view,
>> but also to be able to cancel pending waits if the ring exits before
>> all events have occurred.
>>
>> This is just the barebones wait/wake support. Features to be added
>> later:
> 
> One item high on my wishlist would be the futexv semantics (wait on any
> of a set of futexes).  It cannot be implemented by issuing several
> FUTEX_WAIT.

Yep, I do think that one is interesting enough to consider upfront.
Unfortunately the internal implementation of that does not look that
great, though I'm sure we can make that work. But would likely require
some futexv refactoring to make it work. I can take a look at it.

You could obviously do futexv with this patchset, just posting N futex
waits and canceling N-1 when you get woken by one. Though that's of
course not very pretty or nice to use, but design wise it would totally
work as you don't actually block on these with io_uring.

-- 
Jens Axboe