liburing: expose syscalls?

liburing: expose syscalls?

[Andres Freund]

Hi,

As long as the syscalls aren't exposed by glibc it'd be useful - at
least for me - to have liburing expose the syscalls without really going
through liburing facilities...

Right now I'm e.g. using a "raw" io_uring_enter(IORING_ENTER_GETEVENTS)
to be able to have multiple processes safely wait for events on the same
uring, without needing to hold the lock [1] protecting the ring [2].  It's
probably a good idea to add a liburing function to be able to do so, but
I'd guess there are going to continue to be cases like that. In a bit
of time it seems likely that at least open source users of uring that
are included in databases, have to work against multiple versions of
liburing (as usually embedding libs is not allowed), and sometimes that
is easier if one can backfill a function or two if necessary.

That syscall should probably be under a name that won't conflict with
eventual glibc implementation of the syscall.

Obviously I can just do the syscall() etc myself, but it seems
unnecessary to have a separate copy of the ifdefs for syscall numbers
etc.

What do you think?


[1] It's more efficient to have the kernel wake up the waiting processes
directly, rather than waking up one process, which then wakes up the
other processes by releasing a lock.

[2] The reason one can't just use io_uring_submit_and_wait or such, is
because it's not safe to call __io_uring_flush_sq(ring) while somebody
else might access the ring.

Greetings,

Andres Freund

Re: liburing: expose syscalls?

[Jens Axboe]

On 2/1/20 5:53 AM, Andres Freund wrote:
> Hi,
> 
> As long as the syscalls aren't exposed by glibc it'd be useful - at
> least for me - to have liburing expose the syscalls without really going
> through liburing facilities...
> 
> Right now I'm e.g. using a "raw" io_uring_enter(IORING_ENTER_GETEVENTS)
> to be able to have multiple processes safely wait for events on the same
> uring, without needing to hold the lock [1] protecting the ring [2].  It's
> probably a good idea to add a liburing function to be able to do so, but
> I'd guess there are going to continue to be cases like that. In a bit
> of time it seems likely that at least open source users of uring that
> are included in databases, have to work against multiple versions of
> liburing (as usually embedding libs is not allowed), and sometimes that
> is easier if one can backfill a function or two if necessary.
> 
> That syscall should probably be under a name that won't conflict with
> eventual glibc implementation of the syscall.
> 
> Obviously I can just do the syscall() etc myself, but it seems
> unnecessary to have a separate copy of the ifdefs for syscall numbers
> etc.
> 
> What do you think?

Not sure what I'm missing here, but liburing already has
__sys_io_uring_enter() for this purpose, and ditto for the register
and setup functions?

-- 
Jens Axboe

Re: liburing: expose syscalls?

[Andres Freund]

Hi, 

On February 1, 2020 6:39:41 PM GMT+01:00, Jens Axboe <axboe@kernel.dk> wrote:
>On 2/1/20 5:53 AM, Andres Freund wrote:
>> Hi,
>> 
>> As long as the syscalls aren't exposed by glibc it'd be useful - at
>> least for me - to have liburing expose the syscalls without really
>going
>> through liburing facilities...
>> 
>> Right now I'm e.g. using a "raw"
>io_uring_enter(IORING_ENTER_GETEVENTS)
>> to be able to have multiple processes safely wait for events on the
>same
>> uring, without needing to hold the lock [1] protecting the ring [2]. 
>It's
>> probably a good idea to add a liburing function to be able to do so,
>but
>> I'd guess there are going to continue to be cases like that. In a bit
>> of time it seems likely that at least open source users of uring that
>> are included in databases, have to work against multiple versions of
>> liburing (as usually embedding libs is not allowed), and sometimes
>that
>> is easier if one can backfill a function or two if necessary.
>> 
>> That syscall should probably be under a name that won't conflict with
>> eventual glibc implementation of the syscall.
>> 
>> Obviously I can just do the syscall() etc myself, but it seems
>> unnecessary to have a separate copy of the ifdefs for syscall numbers
>> etc.
>> 
>> What do you think?
>
>Not sure what I'm missing here, but liburing already has
>__sys_io_uring_enter() for this purpose, and ditto for the register
>and setup functions?

Aren't they hidden to the outside by the symbol versioning script?

Andres
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: liburing: expose syscalls?

[Jens Axboe]

On 2/1/20 10:49 AM, Andres Freund wrote:
> Hi, 
> 
> On February 1, 2020 6:39:41 PM GMT+01:00, Jens Axboe <axboe@kernel.dk> wrote:
>> On 2/1/20 5:53 AM, Andres Freund wrote:
>>> Hi,
>>>
>>> As long as the syscalls aren't exposed by glibc it'd be useful - at
>>> least for me - to have liburing expose the syscalls without really
>> going
>>> through liburing facilities...
>>>
>>> Right now I'm e.g. using a "raw"
>> io_uring_enter(IORING_ENTER_GETEVENTS)
>>> to be able to have multiple processes safely wait for events on the
>> same
>>> uring, without needing to hold the lock [1] protecting the ring [2]. 
>> It's
>>> probably a good idea to add a liburing function to be able to do so,
>> but
>>> I'd guess there are going to continue to be cases like that. In a bit
>>> of time it seems likely that at least open source users of uring that
>>> are included in databases, have to work against multiple versions of
>>> liburing (as usually embedding libs is not allowed), and sometimes
>> that
>>> is easier if one can backfill a function or two if necessary.
>>>
>>> That syscall should probably be under a name that won't conflict with
>>> eventual glibc implementation of the syscall.
>>>
>>> Obviously I can just do the syscall() etc myself, but it seems
>>> unnecessary to have a separate copy of the ifdefs for syscall numbers
>>> etc.
>>>
>>> What do you think?
>>
>> Not sure what I'm missing here, but liburing already has
>> __sys_io_uring_enter() for this purpose, and ditto for the register
>> and setup functions?
> 
> Aren't they hidden to the outside by the symbol versioning script?

So you just want to have them exposed? I'd be fine with that. I'll
take a patch :-)

-- 
Jens Axboe

Re: liburing: expose syscalls?

[Pavel Begunkov]

[-- Attachment #1.1: Type: text/plain, Size: 2081 bytes --]

On 01/02/2020 20:52, Jens Axboe wrote:
> On 2/1/20 10:49 AM, Andres Freund wrote:
>> Hi, 
>>
>> On February 1, 2020 6:39:41 PM GMT+01:00, Jens Axboe <axboe@kernel.dk> wrote:
>>> On 2/1/20 5:53 AM, Andres Freund wrote:
>>>> Hi,
>>>>
>>>> As long as the syscalls aren't exposed by glibc it'd be useful - at
>>>> least for me - to have liburing expose the syscalls without really
>>> going
>>>> through liburing facilities...
>>>>
>>>> Right now I'm e.g. using a "raw"
>>> io_uring_enter(IORING_ENTER_GETEVENTS)
>>>> to be able to have multiple processes safely wait for events on the
>>> same
>>>> uring, without needing to hold the lock [1] protecting the ring [2]. 
>>> It's
>>>> probably a good idea to add a liburing function to be able to do so,
>>> but
>>>> I'd guess there are going to continue to be cases like that. In a bit
>>>> of time it seems likely that at least open source users of uring that
>>>> are included in databases, have to work against multiple versions of
>>>> liburing (as usually embedding libs is not allowed), and sometimes
>>> that
>>>> is easier if one can backfill a function or two if necessary.
>>>>
>>>> That syscall should probably be under a name that won't conflict with
>>>> eventual glibc implementation of the syscall.
>>>>
>>>> Obviously I can just do the syscall() etc myself, but it seems
>>>> unnecessary to have a separate copy of the ifdefs for syscall numbers
>>>> etc.
>>>>
>>>> What do you think?
>>>
>>> Not sure what I'm missing here, but liburing already has
>>> __sys_io_uring_enter() for this purpose, and ditto for the register
>>> and setup functions?
>>
>> Aren't they hidden to the outside by the symbol versioning script?
> 
> So you just want to have them exposed? I'd be fine with that. I'll
> take a patch :-)
> 

Depends on how it's used, but I'd strive to inline __sys_io_uring_enter()
to remove the extra indirect call into the shared lib. Though, not sure about
packaging and all this stuff. May be useful to do that for liburing as well.

-- 
Pavel Begunkov


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: liburing: expose syscalls?

[Jens Axboe]

On 2/1/20 2:16 PM, Pavel Begunkov wrote:
> On 01/02/2020 20:52, Jens Axboe wrote:
>> On 2/1/20 10:49 AM, Andres Freund wrote:
>>> Hi, 
>>>
>>> On February 1, 2020 6:39:41 PM GMT+01:00, Jens Axboe <axboe@kernel.dk> wrote:
>>>> On 2/1/20 5:53 AM, Andres Freund wrote:
>>>>> Hi,
>>>>>
>>>>> As long as the syscalls aren't exposed by glibc it'd be useful - at
>>>>> least for me - to have liburing expose the syscalls without really
>>>> going
>>>>> through liburing facilities...
>>>>>
>>>>> Right now I'm e.g. using a "raw"
>>>> io_uring_enter(IORING_ENTER_GETEVENTS)
>>>>> to be able to have multiple processes safely wait for events on the
>>>> same
>>>>> uring, without needing to hold the lock [1] protecting the ring [2]. 
>>>> It's
>>>>> probably a good idea to add a liburing function to be able to do so,
>>>> but
>>>>> I'd guess there are going to continue to be cases like that. In a bit
>>>>> of time it seems likely that at least open source users of uring that
>>>>> are included in databases, have to work against multiple versions of
>>>>> liburing (as usually embedding libs is not allowed), and sometimes
>>>> that
>>>>> is easier if one can backfill a function or two if necessary.
>>>>>
>>>>> That syscall should probably be under a name that won't conflict with
>>>>> eventual glibc implementation of the syscall.
>>>>>
>>>>> Obviously I can just do the syscall() etc myself, but it seems
>>>>> unnecessary to have a separate copy of the ifdefs for syscall numbers
>>>>> etc.
>>>>>
>>>>> What do you think?
>>>>
>>>> Not sure what I'm missing here, but liburing already has
>>>> __sys_io_uring_enter() for this purpose, and ditto for the register
>>>> and setup functions?
>>>
>>> Aren't they hidden to the outside by the symbol versioning script?
>>
>> So you just want to have them exposed? I'd be fine with that. I'll
>> take a patch :-)
>>
> 
> Depends on how it's used, but I'd strive to inline
> __sys_io_uring_enter() to remove the extra indirect call into the
> shared lib. Though, not sure about packaging and all this stuff. May
> be useful to do that for liburing as well.

Not sure that actually matters when you're doing a syscall anyway, that
should be the long pole for the operation.

-- 
Jens Axboe

Re: liburing: expose syscalls?

[Pavel Begunkov]

[-- Attachment #1.1: Type: text/plain, Size: 661 bytes --]

On 02/02/2020 01:51, Jens Axboe wrote:
>>> So you just want to have them exposed? I'd be fine with that. I'll
>>> take a patch :-)
>>>
>>
>> Depends on how it's used, but I'd strive to inline
>> __sys_io_uring_enter() to remove the extra indirect call into the
>> shared lib. Though, not sure about packaging and all this stuff. May
>> be useful to do that for liburing as well.
> 
> Not sure that actually matters when you're doing a syscall anyway, that
> should be the long pole for the operation.
> 

Yeah, but they are starting to stack up (+syscall() from glibc) and can became
even more layered on the user side.

-- 
Pavel Begunkov


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: liburing: expose syscalls?

[Jens Axboe]

On 2/1/20 4:30 PM, Pavel Begunkov wrote:
> On 02/02/2020 01:51, Jens Axboe wrote:
>>>> So you just want to have them exposed? I'd be fine with that. I'll
>>>> take a patch :-)
>>>>
>>>
>>> Depends on how it's used, but I'd strive to inline
>>> __sys_io_uring_enter() to remove the extra indirect call into the
>>> shared lib. Though, not sure about packaging and all this stuff. May
>>> be useful to do that for liburing as well.
>>
>> Not sure that actually matters when you're doing a syscall anyway, that
>> should be the long pole for the operation.
>>
> 
> Yeah, but they are starting to stack up (+syscall() from glibc) and can became
> even more layered on the user side.

Not saying it's free, just that I expect the actual system call to dominate.
But I hear what you are saying, this is exactly why the liburing hot path
resides in static inline code, and doesn't require library calls. I did
draw the line on anything that needs a system call, figuring that wasn't
worth over-optimizing.

I could be full of shit, numbers talk!

-- 
Jens Axboe

Re: liburing: expose syscalls?

[Andres Freund]

Hi,

On 2020-02-01 20:29:39 -0700, Jens Axboe wrote:
> On 2/1/20 4:30 PM, Pavel Begunkov wrote:
> > On 02/02/2020 01:51, Jens Axboe wrote:
> >>>> So you just want to have them exposed? I'd be fine with that. I'll
> >>>> take a patch :-)
> >>>>
> >>>
> >>> Depends on how it's used, but I'd strive to inline
> >>> __sys_io_uring_enter() to remove the extra indirect call into the
> >>> shared lib. Though, not sure about packaging and all this stuff. May
> >>> be useful to do that for liburing as well.
> >>
> >> Not sure that actually matters when you're doing a syscall anyway, that
> >> should be the long pole for the operation.
> >>
> > 
> > Yeah, but they are starting to stack up (+syscall() from glibc) and can became
> > even more layered on the user side.
> 
> Not saying it's free, just that I expect the actual system call to dominate.
> But I hear what you are saying, this is exactly why the liburing hot path
> resides in static inline code, and doesn't require library calls. I did
> draw the line on anything that needs a system call, figuring that wasn't
> worth over-optimizing.

That is my intuition too, especially in my case where I'm "intending" to
block in the kernel. I just tried locally, and at least with storage (a
Samsung 970 pro NVMe, in my laptop) in the path, runtime differences are
below run to run noise, and the profile doesn't show any meaningful time
spent in the indirection.

I don't really see a reason not to move syscall.c functions into a
header however. Would allow the to remove the syscall() indirection
transparently once glibc gets around supporting the syscall too. Given
that it's essentially just kernel interface definitions, how about just
putting it into liburing/io_uring.h?

I'll open a PR.  If somebody wants to bikeshed on using a name other
than __sys_*...

Greetings,

Andres Freund