[GIT PULL] io_uring core updates for 5.19-rc1

[Jens Axboe <axboe@kernel.dk>]

Hi Linus,

Here are the main io_uring changes for the 5.19 merge window. This pull
request contains:

- Fixes for sparse type warnings (Christoph, Vasily)

- Support for multi-shot accept (Hao)

- Support for io_uring managed fixed files, rather than always needing
  the applicationt o manage the indices (me)

- Fix for a spurious poll wakeup (Dylan)

- CQE overflow fixes (Dylan)

- Support more types of cancelations (me)

- Support for co-operative task_work signaling, rather than always
  forcing an IPI (me)

- Support for doing poll first when appropriate, rather than always
  attempting a transfer first (me)

- Provided buffer cleanups and support for mapped buffers (me)

- Improve how io_uring handles inflight SCM files (Pavel)

- Speedups for registered files (Pavel, me)

- Organize the completion data in a struct in io_kiocb rather than keep
  it in separate spots (Pavel)

- task_work improvements (Pavel)

- Cleanup and optimize the submission path, in general and for handling
  links (Pavel)

- Speedups for registered resource handling (Pavel)

- Support sparse buffers and file maps (Pavel, me)

- Various fixes and cleanups (Almog, Pavel, me)

This will merge cleanly, but due to the completion data change mentioned
above, a late fix in 5.18 will cause it to fail compile. Fix is to do:

diff --git a/fs/io_uring.c b/fs/io_uring.c
index 59e65d70dfda..0c6a14276186 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -7413,7 +7413,7 @@ static int io_req_prep_async(struct io_kiocb *req)
 
 	/* assign early for deferred execution for non-fixed file */
 	if (def->needs_file && !(req->flags & REQ_F_FIXED_FILE))
-		req->file = io_file_get_normal(req, req->fd);
+		req->file = io_file_get_normal(req, req->cqe.fd);
 	if (!def->needs_async_setup)
 		return 0;
 	if (WARN_ON_ONCE(req_has_async_data(req)))

and then it'll be fine.

Please pull!


The following changes since commit af2d861d4cd2a4da5137f795ee3509e6f944a25b:

  Linux 5.18-rc4 (2022-04-24 14:51:22 -0700)

are available in the Git repository at:

  git://git.kernel.dk/linux-block.git tags/for-5.19/io_uring-2022-05-22

for you to fetch changes up to 0e7579ca732a39cc377e17509dda9bfc4f6ba78e:

  io_uring: fix incorrect __kernel_rwf_t cast (2022-05-19 12:27:59 -0600)

----------------------------------------------------------------
for-5.19/io_uring-2022-05-22

----------------------------------------------------------------
Almog Khaikin (1):
      io_uring: replace smp_mb() with smp_mb__after_atomic() in io_sq_thread()

Christoph Hellwig (6):
      io_uring: use a rwf_t for io_rw.flags
      io_uring: don't use ERR_PTR for user pointers
      io_uring: drop a spurious inline on a forward declaration
      io_uring: make apoll_events a __poll_t
      io_uring: consistently use the EPOLL* defines
      io_uring: use rcu_dereference in io_close

Dylan Yudaken (6):
      io_uring: add trace support for CQE overflow
      io_uring: trace cqe overflows
      io_uring: rework io_uring_enter to simplify return value
      io_uring: use constants for cq_overflow bitfield
      io_uring: return an error when cqe is dropped
      io_uring: only wake when the correct events are set

Hao Xu (4):
      io_uring: add IORING_ACCEPT_MULTISHOT for accept
      io_uring: add REQ_F_APOLL_MULTISHOT for requests
      io_uring: let fast poll support multishot
      io_uring: implement multishot mode for accept

Jens Axboe (43):
      io_uring: move finish_wait() outside of loop in cqring_wait()
      io_uring: store SCM state in io_fixed_file->file_ptr
      io_uring: remove dead 'poll_only' argument to io_poll_cancel()
      io_uring: pass in struct io_cancel_data consistently
      io_uring: add support for IORING_ASYNC_CANCEL_ALL
      io_uring: allow IORING_OP_ASYNC_CANCEL with 'fd' key
      io_uring: add support for IORING_ASYNC_CANCEL_ANY
      io_uring: support MSG_WAITALL for IORING_OP_SEND(MSG)
      io_uring: allow re-poll if we made progress
      io_uring: fix compile warning for 32-bit builds
      task_work: allow TWA_SIGNAL without a rescheduling IPI
      io_uring: serialize ctx->rings->sq_flags with atomic_or/and
      io-wq: use __set_notify_signal() to wake workers
      io_uring: set task_work notify method at init time
      io_uring: use TWA_SIGNAL_NO_IPI if IORING_SETUP_COOP_TASKRUN is used
      io_uring: add IORING_SETUP_TASKRUN_FLAG
      io_uring: check IOPOLL/ioprio support upfront
      io_uring: add POLL_FIRST support for send/sendmsg and recv/recvmsg
      io_uring: use 'sr' vs 'req->sr_msg' consistently
      io_uring: kill io_recv_buffer_select() wrapper
      io_uring: make io_buffer_select() return the user address directly
      io_uring: kill io_rw_buffer_select() wrapper
      io_uring: ignore ->buf_index if REQ_F_BUFFER_SELECT isn't set
      io_uring: always use req->buf_index for the provided buffer group
      io_uring: get rid of hashed provided buffer groups
      io_uring: never call io_buffer_select() for a buffer re-select
      io_uring: abstract out provided buffer list selection
      io_uring: move provided and fixed buffers into the same io_kiocb area
      io_uring: move provided buffer state closer to submit state
      io_uring: eliminate the need to track provided buffer ID separately
      io_uring: don't clear req->kbuf when buffer selection is done
      io_uring: track fixed files with a bitmap
      io_uring: add basic fixed file allocator
      io_uring: allow allocated fixed files for openat/openat2
      io_uring: allow allocated fixed files for accept
      io_uring: bump max direct descriptor count to 1M
      io_uring: add flag for allocating a fully sparse direct descriptor space
      io_uring: fix locking state for empty buffer group
      io_uring: add buffer selection support to IORING_OP_NOP
      io_uring: add io_pin_pages() helper
      io_uring: add support for ring mapped supplied buffers
      io_uring: initialize io_buffer_list head when shared ring is unregistered
      io_uring: disallow mixed provided buffer group registrations

Pavel Begunkov (50):
      io_uring: small optimisation of tctx_task_work
      io_uring: remove extra ifs around io_commit_cqring
      io_uring: refactor io_req_find_next
      io_uring: optimise io_free_batch_list
      io_uring: move poll recycling later in compl flushing
      io_uring: clean up io_queue_next()
      io_uring: split off IOPOLL argument verifiction
      io_uring: pre-calculate syscall iopolling decision
      io_uring: optimise mutex locking for submit+iopoll
      io_uring: cleanup conditional submit locking
      io_uring: partially uninline io_put_task()
      io_uring: silence io_for_each_link() warning
      io_uring: refactor io_req_add_compl_list()
      io_uring: don't scm-account for non af_unix sockets
      io_uring: uniform SCM accounting
      io_uring: refactor __io_sqe_files_scm
      io_uring: don't pass around fixed index for scm
      io_uring: deduplicate SCM accounting
      io_uring: rename io_sqe_file_register
      io_uring: explicitly keep a CQE in io_kiocb
      io_uring: memcpy CQE from req
      io_uring: shrink final link flush
      io_uring: inline io_flush_cached_reqs
      io_uring: helper for empty req cache checks
      io_uring: add helper to return req to cache list
      io_uring: optimise submission loop invariant
      io_uring: optimise submission left counting
      io_uring: optimise io_get_cqe()
      io_uring: clean poll tw PF_EXITING handling
      io_uring: minor refactoring for some tw handlers
      io_uring: kill io_put_req_deferred()
      io_uring: inline io_free_req()
      io_uring: helper for prep+queuing linked timeouts
      io_uring: inline io_queue_sqe()
      io_uring: rename io_queue_async_work()
      io_uring: refactor io_queue_sqe()
      io_uring: introduce IO_REQ_LINK_FLAGS
      io_uring: refactor lazy link fail
      io_uring: refactor io_submit_sqe()
      io_uring: inline io_req_complete_fail_submit()
      io_uring: add data_race annotations
      io_uring: use right helpers for file assign locking
      io_uring: refactor io_assign_file error path
      io_uring: store rsrc node in req instead of refs
      io_uring: add a helper for putting rsrc nodes
      io_uring: kill ctx arg from io_req_put_rsrc
      io_uring: move timeout locking in io_timeout_cancel()
      io_uring: refactor io_disarm_next() locking
      io_uring: avoid io-wq -EAGAIN looping for !IOPOLL
      io_uring: add fully sparse buffer registration

Vasily Averin (1):
      io_uring: fix incorrect __kernel_rwf_t cast

 fs/io-wq.c                      |    4 +-
 fs/io-wq.h                      |    1 +
 fs/io_uring.c                   | 2659 ++++++++++++++++++++++++---------------
 include/linux/sched/signal.h    |   13 +-
 include/linux/task_work.h       |    1 +
 include/trace/events/io_uring.h |   44 +-
 include/uapi/linux/io_uring.h   |   95 +-
 kernel/task_work.c              |   25 +-
 8 files changed, 1812 insertions(+), 1030 deletions(-)

-- 
Jens Axboe