[PATCHSET v2] Add lockless MPSC FIFO queue for task work

[Jens Axboe <axboe@kernel.dk>]

Hi,

For v1, see here:

https://lore.kernel.org/io-uring/20260611160553.1486640-1-axboe@kernel.dk/

v2 moves the regular task_work over to mpscq as well, and adds a few
optimizations on top of v1. See the changes section for more details.

Can also be found in a git tree here:

https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git/log/?h=io_uring-tw-mpscq

Changes since v1:
- Add patches moving !defer task_work to mpscq as well.
- Get rid of ->cq_wait_added and the associated atomics on that. Have
  waiter set ->cq_wait_nr and just count down from that. Removes another
  atomic in local work additions, and eliminates the need for the
  atomic_try_cmpxchg() for the faster lazy wakes.
- Get rid of RCU read lock for local work additions (Caleb)
- Cleanup up mpscq API a bit (Caleb)
- Correct mpscq comment (Caleb)

 include/linux/io_uring_types.h |  39 ++++-
 io_uring/cancel.c              |   2 -
 io_uring/io_uring.c            |   9 +-
 io_uring/mpscq.h               | 118 +++++++++++++
 io_uring/sqpoll.c              |  30 ++--
 io_uring/tctx.c                |   3 +-
 io_uring/tw.c                  | 307 +++++++++++++++------------------
 io_uring/tw.h                  |  11 +-
 io_uring/wait.c                |   2 +-
 io_uring/wait.h                |  10 +-
 10 files changed, 316 insertions(+), 215 deletions(-)

-- 
Jens Axboe