io-wrk threads on socket vs non-socket

io-wrk threads on socket vs non-socket

[Marek Majkowski]

Hi!

I'm playing with io-uring, and I found the io-wrk thread situation confusing.

(A) In one case, I have a SOCK_DGRAM socket (blocking), over which I
do IORING_OP_RECVMSG. This works well, and unless I mark the sqe as
IOSQE_ASYNC, it doesn't seem to start an io-wrk kernel thread.

(B) However, the same can't be said of another situation. In the
second case I have a tap file descriptor (blocking), which doesn't
support "Socket operations on non-socket", so I must do
IORING_OP_READV. This however seems to start a new io-wrk for each
readv request:

$ pstree -pt `pidof tapuring`
tapuring(44932)─┬─{iou-wrk-44932}(44937)
                ├─{iou-wrk-44932}(44938)
                ├─{iou-wrk-44932}(44939)
                ├─{iou-wrk-44932}(44940)
                ├─{iou-wrk-44932}(44941)
                ├─{iou-wrk-44932}(44942)

I would expect both situations to behave the same way.

The manpage for IOSQE_ASYNC:

       IOSQE_ASYNC
              Normal operation for io_uring is to try and issue an sqe
              as non-blocking first, and if that fails, execute it in an
              async manner. To support more efficient overlapped
              operation of requests that the application knows/assumes
              will always (or most of the time) block, the application
              can ask for an sqe to be issued async from the start. Note
              that this flag immediately causes the SQE to be offloaded
              to an async helper thread with no initial non-blocking
              attempt.  This may be less efficient and should not be
              used liberally or without understanding the performance
              and efficiency tradeoffs.

This seems to cover the tap file descriptor case. It tries to readv
and when that fails a new io-wrk is spawned. Fine. However, as I
described it seems this is not true for sockets, as without
IOSQE_ASYNC the io-wrk thread is _not_ spawned there?

Is the behaviour different due to socket vs non-socket or readv vs recvmsg?

Please advise.

Marek

Re: io-wrk threads on socket vs non-socket

[Jens Axboe]

On 6/14/23 8:09?AM, Marek Majkowski wrote:
> Hi!
> 
> I'm playing with io-uring, and I found the io-wrk thread situation confusing.
> 
> (A) In one case, I have a SOCK_DGRAM socket (blocking), over which I
> do IORING_OP_RECVMSG. This works well, and unless I mark the sqe as
> IOSQE_ASYNC, it doesn't seem to start an io-wrk kernel thread.
> 
> (B) However, the same can't be said of another situation. In the
> second case I have a tap file descriptor (blocking), which doesn't
> support "Socket operations on non-socket", so I must do
> IORING_OP_READV. This however seems to start a new io-wrk for each
> readv request:
> 
> $ pstree -pt `pidof tapuring`
> tapuring(44932)???{iou-wrk-44932}(44937)
>                 ??{iou-wrk-44932}(44938)
>                 ??{iou-wrk-44932}(44939)
>                 ??{iou-wrk-44932}(44940)
>                 ??{iou-wrk-44932}(44941)
>                 ??{iou-wrk-44932}(44942)
> 
> I would expect both situations to behave the same way.
> 
> The manpage for IOSQE_ASYNC:
> 
>        IOSQE_ASYNC
>               Normal operation for io_uring is to try and issue an sqe
>               as non-blocking first, and if that fails, execute it in an
>               async manner. To support more efficient overlapped
>               operation of requests that the application knows/assumes
>               will always (or most of the time) block, the application
>               can ask for an sqe to be issued async from the start. Note
>               that this flag immediately causes the SQE to be offloaded
>               to an async helper thread with no initial non-blocking
>               attempt.  This may be less efficient and should not be
>               used liberally or without understanding the performance
>               and efficiency tradeoffs.
> 
> This seems to cover the tap file descriptor case. It tries to readv
> and when that fails a new io-wrk is spawned. Fine. However, as I
> described it seems this is not true for sockets, as without
> IOSQE_ASYNC the io-wrk thread is _not_ spawned there?
> 
> Is the behaviour different due to socket vs non-socket or readv vs
> recvmsg?

What kernel are you using? tap just recently got FMODE_NOWAIT support,
which should trigger poll instead of needing to spawn an io worker.

Also, as usual, a test case would be appreciated. Particularly if this
is on a current kernel where we would not expect to see io-wq activity
for a read of tap.

-- 
Jens Axboe

Re: io-wrk threads on socket vs non-socket

[Marek Majkowski]

On Wed, Jun 14, 2023 at 6:03 PM Jens Axboe <axboe@kernel.dk> wrote:
>
> On 6/14/23 8:09?AM, Marek Majkowski wrote:
> > Hi!
> >
> > I'm playing with io-uring, and I found the io-wrk thread situation confusing.
> >
> > (A) In one case, I have a SOCK_DGRAM socket (blocking), over which I
> > do IORING_OP_RECVMSG. This works well, and unless I mark the sqe as
> > IOSQE_ASYNC, it doesn't seem to start an io-wrk kernel thread.
> >
> > (B) However, the same can't be said of another situation. In the
> > second case I have a tap file descriptor (blocking), which doesn't
> > support "Socket operations on non-socket", so I must do
> > IORING_OP_READV. This however seems to start a new io-wrk for each
> > readv request:
> >
> > $ pstree -pt `pidof tapuring`
> > tapuring(44932)???{iou-wrk-44932}(44937)
> >                 ??{iou-wrk-44932}(44938)
> >                 ??{iou-wrk-44932}(44939)
> >                 ??{iou-wrk-44932}(44940)
> >                 ??{iou-wrk-44932}(44941)
> >                 ??{iou-wrk-44932}(44942)
> >
> > I would expect both situations to behave the same way.
> >
> > The manpage for IOSQE_ASYNC:
> >
> >        IOSQE_ASYNC
> >               Normal operation for io_uring is to try and issue an sqe
> >               as non-blocking first, and if that fails, execute it in an
> >               async manner. To support more efficient overlapped
> >               operation of requests that the application knows/assumes
> >               will always (or most of the time) block, the application
> >               can ask for an sqe to be issued async from the start. Note
> >               that this flag immediately causes the SQE to be offloaded
> >               to an async helper thread with no initial non-blocking
> >               attempt.  This may be less efficient and should not be
> >               used liberally or without understanding the performance
> >               and efficiency tradeoffs.
> >
> > This seems to cover the tap file descriptor case. It tries to readv
> > and when that fails a new io-wrk is spawned. Fine. However, as I
> > described it seems this is not true for sockets, as without
> > IOSQE_ASYNC the io-wrk thread is _not_ spawned there?
> >
> > Is the behaviour different due to socket vs non-socket or readv vs
> > recvmsg?
>
> What kernel are you using? tap just recently got FMODE_NOWAIT support,
> which should trigger poll instead of needing to spawn an io worker.
>
> Also, as usual, a test case would be appreciated. Particularly if this
> is on a current kernel where we would not expect to see io-wq activity
> for a read of tap.


After two days I think I finally have some repro. Let's track this
particular tap io-wrk issue under
https://github.com/axboe/liburing/issues/886

I can confirm that indeed (apart form the mentioned bug) tap is going
into poll mode, and doesn't launch io-wrk.

However, I still miss a piece of the puzzle about io-wrk polling.

Let's assume a different situation, let's say I have a dozen of
sockets, and do a recvmsg on each of them, with IOSQE_ASYNC flag set.

I would expect to see a dozen of io-wrk threads, however I'm not fully
understanding what will they do. Will each do a full poll() over all
the sockets? If a packet arrives to one socket, will all the io-wrk
threads wakeup? What if there is a limit of 2 io-wrk threads and there
are more sockets?

I think I'm asking about the relationship between SQE's / sockets and
io-wrk poller.

Marek