[PATCH v3 00/10] BPF controlled io_uring

[Pavel Begunkov <asml.silence@gmail.com>]

This patch set adds a hook into the io_uring waiting loop and
allows to attach a BPF program to it, which is implemented as
io_uring specific struct_ops. It allows event processing and
request submission as well as waiting tuning.

There is a bunch of cases collected over time it's designed to cover:

- Syscall avoidance. Instead of returning to the userspace for
  CQE processing, a part of the logic can be moved into BPF to
  avoid excessive number of syscalls.

- Smarter request ordering and linking. Request links are pretty
  limited and inflexible as they can't pass information from one
  request to another. With BPF we can peek at CQEs and memory and
  compile a subsequent request.

- Eventual deprecation of links. Linked request kernel logic is
  a large liability. It introduces a lot of complexity to core
  io_uring, and also leaking into other areas, e.g. affecting
  decisions on what and when is initialised. It'll be a big win
  if it can be moved into the new hook as a kernel function or
  even better BPF.

- Access to in-kernel io_uring resources. For example, there are
  registered buffers that can't be directly accessed by the userspace,
  however we can give BPF the ability to peek at them. It can be used
  to take a look at in-buffer app level headers to decide what to do
  with data next and issuing IO using it.

- Finer control over waiting algorithms. io_uring has min-wait support,
  however it's still limited by uapi, and elaborate parametrisation
  for more complex algorithms won't be feasible.

- Optimised waiting. On the same note, mixing requests of different
  types and combining submissions with waiting into a single syscall
  proved to be troublesome because of different ways requests are
  executed. BPF and CQE parsing will help with that.

- Smarter polling. Napi polling is performed only once per syscall
  and then it switches to waiting. We can do smarter and intermix
  polling with waiting using the hook.

It might need more specialised kfuncs in the future, but the core
functionality is implemented with just two simple functions. One
returns region memory, which gives BPF access to CQ/SQ/etc. And
the second is for submitting requests. It's also given a structure
as an argument, which is used to pass waiting information.

Previously, a test sequentially executing N nop request in BPF
showed a 50% performance edge over 2-nop links, and 80% with no
linking at all on a mitigated kernel.

since v2: https://lore.kernel.org/io-uring/cover.1749214572.git.asml.silence@gmail.com/
  - Removed most of utility kfuncs and replaced it with a single
    helper returning the ring memory.
  - Added KF_TRUSTED_ARGS to kfuncs
  - Fix ifdef guarding
  - Added a selftest
  - Adjusted the waiting loop
  - Reused the bpf lock section for task_work execution

Pavel Begunkov (10):
  io_uring: rename the wait queue entry field
  io_uring: simplify io_cqring_wait_schedule results
  io_uring: export __io_run_local_work
  io_uring: extract waiting parameters into a struct
  io_uring/bpf: add stubs for bpf struct_ops
  io_uring/bpf: add handle events callback
  io_uring/bpf: implement struct_ops registration
  io_uring/bpf: add basic kfunc helpers
  selftests/io_uring: update mini liburing
  selftests/io_uring: add bpf io_uring selftests

 include/linux/io_uring_types.h               |   6 +
 io_uring/Kconfig                             |   5 +
 io_uring/Makefile                            |   1 +
 io_uring/bpf.c                               | 277 +++++++++++++++++++
 io_uring/bpf.h                               |  47 ++++
 io_uring/io_uring.c                          |  63 +++--
 io_uring/io_uring.h                          |  18 +-
 io_uring/napi.c                              |   4 +-
 tools/include/io_uring/mini_liburing.h       |  57 +++-
 tools/testing/selftests/Makefile             |   3 +-
 tools/testing/selftests/io_uring/Makefile    | 164 +++++++++++
 tools/testing/selftests/io_uring/basic.bpf.c |  81 ++++++
 tools/testing/selftests/io_uring/common.h    |   2 +
 tools/testing/selftests/io_uring/runner.c    |  80 ++++++
 tools/testing/selftests/io_uring/types.bpf.h | 136 +++++++++
 15 files changed, 900 insertions(+), 44 deletions(-)
 create mode 100644 io_uring/bpf.c
 create mode 100644 io_uring/bpf.h
 create mode 100644 tools/testing/selftests/io_uring/Makefile
 create mode 100644 tools/testing/selftests/io_uring/basic.bpf.c
 create mode 100644 tools/testing/selftests/io_uring/common.h
 create mode 100644 tools/testing/selftests/io_uring/runner.c
 create mode 100644 tools/testing/selftests/io_uring/types.bpf.h

-- 
2.49.0