io-wrk threads on socket vs non-socket

[Marek Majkowski <marek@cloudflare.com>]

Hi!

I'm playing with io-uring, and I found the io-wrk thread situation confusing.

(A) In one case, I have a SOCK_DGRAM socket (blocking), over which I
do IORING_OP_RECVMSG. This works well, and unless I mark the sqe as
IOSQE_ASYNC, it doesn't seem to start an io-wrk kernel thread.

(B) However, the same can't be said of another situation. In the
second case I have a tap file descriptor (blocking), which doesn't
support "Socket operations on non-socket", so I must do
IORING_OP_READV. This however seems to start a new io-wrk for each
readv request:

$ pstree -pt `pidof tapuring`
tapuring(44932)─┬─{iou-wrk-44932}(44937)
                ├─{iou-wrk-44932}(44938)
                ├─{iou-wrk-44932}(44939)
                ├─{iou-wrk-44932}(44940)
                ├─{iou-wrk-44932}(44941)
                ├─{iou-wrk-44932}(44942)

I would expect both situations to behave the same way.

The manpage for IOSQE_ASYNC:

       IOSQE_ASYNC
              Normal operation for io_uring is to try and issue an sqe
              as non-blocking first, and if that fails, execute it in an
              async manner. To support more efficient overlapped
              operation of requests that the application knows/assumes
              will always (or most of the time) block, the application
              can ask for an sqe to be issued async from the start. Note
              that this flag immediately causes the SQE to be offloaded
              to an async helper thread with no initial non-blocking
              attempt.  This may be less efficient and should not be
              used liberally or without understanding the performance
              and efficiency tradeoffs.

This seems to cover the tap file descriptor case. It tries to readv
and when that fails a new io-wrk is spawned. Fine. However, as I
described it seems this is not true for sockets, as without
IOSQE_ASYNC the io-wrk thread is _not_ spawned there?

Is the behaviour different due to socket vs non-socket or readv vs recvmsg?

Please advise.

Marek