[GIT PULL] io_uring fixes for 5.12-rc2

[Jens Axboe <axboe@kernel.dk>]

Hi Linus,

A bit of a mix between fallout from the worker change,
cleanups/reductions now possible from that change, and fixes in general.
In detail:

- Fully serialize manager and worker creation, fixing races due to that.

- Clean up some naming that had gone stale.

- SQPOLL fixes.

- Fix race condition around task_work rework that went into this merge
  window.

- Implement unshare. Used for when the original task does unshare(2) or
  setuid/seteuid and friends, drops the original workers and forks new
  ones.

- Drop the only remaining piece of state shuffling we had left, which
  was cred. Move it into issue instead, and we can drop all of that code
  too.

- Kill f_op->flush() usage. That was such a nasty hack that we had out
  of necessity, we no longer need it.

- Following from ->flush() removal, we can also drop various bits of ctx
  state related to SQPOLL and cancelations.

- Fix an issue with IOPOLL retry, which originally was fallout from a
  filemap change (removing iov_iter_revert()), but uncovered an issue
  with iovec re-import too late.

- Fix an issue with system suspend.

- Use xchg() for fallback work, instead of cmpxchg().

- Properly destroy io-wq on exec.

- Add create_io_thread() core helper, and use that in io-wq and
  io_uring. This allows us to remove various silly completion events
  related to thread setup.

- A few error handling fixes.

This should be the grunt of fixes necessary for the new workers, next
week should be quieter. We've got a pending series from Pavel on
cancelations, and how tasks and rings are indexed. Outside of that,
should just be minor fixes. Even with these fixes, we're still killing a
net ~80 lines.

Please pull!


The following changes since commit fe07bfda2fb9cdef8a4d4008a409bb02f35f1bd8:

  Linux 5.12-rc1 (2021-02-28 16:05:19 -0800)

are available in the Git repository at:

  git://git.kernel.dk/linux-block.git tags/io_uring-5.12-2021-03-05

for you to fetch changes up to e45cff58858883290c98f65d409839a7295c95f3:

  io_uring: don't restrict issue_flags for io_openat (2021-03-05 09:52:29 -0700)

----------------------------------------------------------------
io_uring-5.12-2021-03-05

----------------------------------------------------------------
Jens Axboe (27):
      io-wq: wait for worker startup when forking a new one
      io-wq: have manager wait for all workers to exit
      io-wq: don't ask for a new worker if we're exiting
      io-wq: rename wq->done completion to wq->started
      io-wq: wait for manager exit on wq destroy
      io-wq: fix double put of 'wq' in error path
      io_uring: SQPOLL stop error handling fixes
      io_uring: don't use complete_all() on SQPOLL thread exit
      io-wq: provide an io_wq_put_and_exit() helper
      io_uring: fix race condition in task_work add and clear
      io_uring: remove unused argument 'tsk' from io_req_caches_free()
      io_uring: kill unnecessary REQ_F_WORK_INITIALIZED checks
      io_uring: move cred assignment into io_issue_sqe()
      io_uring: kill unnecessary io_run_ctx_fallback() in io_ring_exit_work()
      io_uring: kill io_uring_flush()
      io_uring: ensure that SQPOLL thread is started for exit
      io_uring: ignore double poll add on the same waitqueue head
      io-wq: fix error path leak of buffered write hash map
      io_uring: fix -EAGAIN retry with IOPOLL
      io_uring: ensure that threads freeze on suspend
      io-wq: ensure all pending work is canceled on exit
      kernel: provide create_io_thread() helper
      io_uring: move to using create_io_thread()
      io_uring: don't keep looping for more events if we can't flush overflow
      io_uring: clear IOCB_WAITQ for non -EIOCBQUEUED return
      io-wq: kill hashed waitqueue before manager exits
      io_uring: make SQPOLL thread parking saner

Pavel Begunkov (14):
      io_uring: run fallback on cancellation
      io_uring: warn on not destroyed io-wq
      io_uring: destroy io-wq on exec
      io_uring: fix __tctx_task_work() ctx race
      io_uring: replace cmpxchg in fallback with xchg
      io_uring: kill sqo_dead and sqo submission halting
      io_uring: remove sqo_task
      io_uring: choose right tctx->io_wq for try cancel
      io_uring: inline io_req_clean_work()
      io_uring: inline __io_queue_async_work()
      io_uring: remove extra in_idle wake up
      io_uring: cancel-match based on flags
      io_uring: reliably cancel linked timeouts
      io_uring: don't restrict issue_flags for io_openat

 fs/io-wq.c                 | 261 +++++++++++------------
 fs/io-wq.h                 |   5 +-
 fs/io_uring.c              | 500 +++++++++++++++++++--------------------------
 include/linux/io_uring.h   |   2 +-
 include/linux/sched/task.h |   2 +
 kernel/fork.c              |  30 +++
 6 files changed, 361 insertions(+), 439 deletions(-)

-- 
Jens Axboe