Re: io-wrk threads on socket vs non-socket

[Marek Majkowski <marek@cloudflare.com>]

On Wed, Jun 14, 2023 at 6:03 PM Jens Axboe <axboe@kernel.dk> wrote:
>
> On 6/14/23 8:09?AM, Marek Majkowski wrote:
> > Hi!
> >
> > I'm playing with io-uring, and I found the io-wrk thread situation confusing.
> >
> > (A) In one case, I have a SOCK_DGRAM socket (blocking), over which I
> > do IORING_OP_RECVMSG. This works well, and unless I mark the sqe as
> > IOSQE_ASYNC, it doesn't seem to start an io-wrk kernel thread.
> >
> > (B) However, the same can't be said of another situation. In the
> > second case I have a tap file descriptor (blocking), which doesn't
> > support "Socket operations on non-socket", so I must do
> > IORING_OP_READV. This however seems to start a new io-wrk for each
> > readv request:
> >
> > $ pstree -pt `pidof tapuring`
> > tapuring(44932)???{iou-wrk-44932}(44937)
> >                 ??{iou-wrk-44932}(44938)
> >                 ??{iou-wrk-44932}(44939)
> >                 ??{iou-wrk-44932}(44940)
> >                 ??{iou-wrk-44932}(44941)
> >                 ??{iou-wrk-44932}(44942)
> >
> > I would expect both situations to behave the same way.
> >
> > The manpage for IOSQE_ASYNC:
> >
> >        IOSQE_ASYNC
> >               Normal operation for io_uring is to try and issue an sqe
> >               as non-blocking first, and if that fails, execute it in an
> >               async manner. To support more efficient overlapped
> >               operation of requests that the application knows/assumes
> >               will always (or most of the time) block, the application
> >               can ask for an sqe to be issued async from the start. Note
> >               that this flag immediately causes the SQE to be offloaded
> >               to an async helper thread with no initial non-blocking
> >               attempt.  This may be less efficient and should not be
> >               used liberally or without understanding the performance
> >               and efficiency tradeoffs.
> >
> > This seems to cover the tap file descriptor case. It tries to readv
> > and when that fails a new io-wrk is spawned. Fine. However, as I
> > described it seems this is not true for sockets, as without
> > IOSQE_ASYNC the io-wrk thread is _not_ spawned there?
> >
> > Is the behaviour different due to socket vs non-socket or readv vs
> > recvmsg?
>
> What kernel are you using? tap just recently got FMODE_NOWAIT support,
> which should trigger poll instead of needing to spawn an io worker.
>
> Also, as usual, a test case would be appreciated. Particularly if this
> is on a current kernel where we would not expect to see io-wq activity
> for a read of tap.


After two days I think I finally have some repro. Let's track this
particular tap io-wrk issue under
https://github.com/axboe/liburing/issues/886

I can confirm that indeed (apart form the mentioned bug) tap is going
into poll mode, and doesn't launch io-wrk.

However, I still miss a piece of the puzzle about io-wrk polling.

Let's assume a different situation, let's say I have a dozen of
sockets, and do a recvmsg on each of them, with IOSQE_ASYNC flag set.

I would expect to see a dozen of io-wrk threads, however I'm not fully
understanding what will they do. Will each do a full poll() over all
the sockets? If a packet arrives to one socket, will all the io-wrk
threads wakeup? What if there is a limit of 2 io-wrk threads and there
are more sockets?

I think I'm asking about the relationship between SQE's / sockets and
io-wrk poller.

Marek