[GIT PULL] io_uring changes for 5.5-rc

[Jens Axboe <axboe@kernel.dk>]

Hi Linus,

Here are the changes for io_uring for 5.5. A lot of stuff has been going
on this cycle, with improving the support for networked IO (and hence
unbounded request completion times) being one of the major themes.
There's been a set of fixes done this week, I'll send those out as well
once we're certain we're fully happy with them. This pull request
contains:

- Unification of the "normal" submit path and the SQPOLL path (Pavel)

- Support for sparse (and bigger) file sets, and updating of those file
  sets without needing to unregister/register again.

- Independently sized CQ ring, instead of just making it always 2x the
  SQ ring size. This makes it more flexible for networked applications.

- Support for overflowed CQ ring, never dropping events but providing
  backpressure on submits.

- Add support for absolute timeouts, not just relative ones.

- Support for generic cancellations. This divorces io_uring from
  workqueues as well, which additionally gets us one step closer to
  generic async system call support.

- With cancellations, we can support grabbing the process file table as
  well, just like we do mm context. This allows support for system calls
  that create file descriptors, like accept4() support that's built on
  top of that.

- Support for io_uring tracing (Dmitrii)

- Support for linked timeouts. These abort an operation if it isn't
  completed by the time noted in the linke timeout.

- Speedup tracking of poll requests

- Various cleanups making the coder easier to follow (Jackie, Pavel,
  Bob, YueHaibing, me)

- Update MAINTAINERS with new io_uring list

This will cause a merge conflict with some of the late fixes that
went into mainline. I'm attaching my merge commit of pulling master
into for-next the other day to resolve it. There are three main
conflicts, two first are trivial, the last one not that bad. But figured
I'd attach it for reference.

Please pull!


  git://git.kernel.dk/linux-block.git tags/for-5.5/io_uring-20191121


----------------------------------------------------------------
Bob Liu (2):
      io_uring: clean up io_uring_cancel_files()
      io_uring: introduce req_need_defer()

Dmitrii Dolgov (1):
      io_uring: add set of tracing events

Jackie Liu (5):
      io_uring: replace s->needs_lock with s->in_async
      io_uring: set -EINTR directly when a signal wakes up in io_cqring_wait
      io_uring: remove passed in 'ctx' function parameter ctx if possible
      io_uring: keep io_put_req only responsible for release and put req
      io_uring: separate the io_free_req and io_free_req_find_next interface

Jens Axboe (47):
      io_uring: run dependent links inline if possible
      io_uring: allow sparse fixed file sets
      io_uring: add support for IORING_REGISTER_FILES_UPDATE
      io_uring: allow application controlled CQ ring size
      io_uring: add support for absolute timeouts
      io_uring: add support for canceling timeout requests
      io-wq: small threadpool implementation for io_uring
      io_uring: replace workqueue usage with io-wq
      io_uring: io_uring: add support for async work inheriting files
      net: add __sys_accept4_file() helper
      io_uring: add support for IORING_OP_ACCEPT
      io_uring: protect fixed file indexing with array_index_nospec()
      io_uring: support for larger fixed file sets
      io_uring: fix race with canceling timeouts
      io_uring: io_wq_create() returns an error pointer, not NULL
      io_uring: support for generic async request cancel
      io_uring: remove io_uring_add_to_prev() trace event
      io_uring: add completion trace event
      MAINTAINERS: update io_uring entry
      io-wq: use proper nesting IRQ disabling spinlocks for cancel
      io_uring: enable optimized link handling for IORING_OP_POLL_ADD
      io_uring: fixup a few spots where link failure isn't flagged
      io_uring: kill dead REQ_F_LINK_DONE flag
      io_uring: abstract out io_async_cancel_one() helper
      io_uring: add support for linked SQE timeouts
      io_uring: make io_cqring_events() take 'ctx' as argument
      io_uring: pass in io_kiocb to fill/add CQ handlers
      io_uring: add support for backlogged CQ ring
      io-wq: io_wqe_run_queue() doesn't need to use list_empty_careful()
      io-wq: add support for bounded vs unbunded work
      io_uring: properly mark async work as bounded vs unbounded
      io_uring: reduce/pack size of io_ring_ctx
      io_uring: fix error clear of ->file_table in io_sqe_files_register()
      io_uring: convert accept4() -ERESTARTSYS into -EINTR
      io_uring: provide fallback request for OOM situations
      io_uring: make ASYNC_CANCEL work with poll and timeout
      io_uring: flag SQPOLL busy condition to userspace
      io_uring: don't do flush cancel under inflight_lock
      io_uring: fix -ENOENT issue with linked timer with short timeout
      io_uring: use correct "is IO worker" helper
      io_uring: fix potential deadlock in io_poll_wake()
      io_uring: check for validity of ->rings in teardown
      io_wq: add get/put_work handlers to io_wq_create()
      io-wq: ensure we have a stable view of ->cur_work for cancellations
      io-wq: ensure free/busy list browsing see all items
      io-wq: remove now redundant struct io_wq_nulls_list
      io_uring: make POLL_ADD/POLL_REMOVE scale better

Pavel Begunkov (8):
      io_uring: remove index from sqe_submit
      io_uring: Fix mm_fault with READ/WRITE_FIXED
      io_uring: Merge io_submit_sqes and io_ring_submit
      io_uring: io_queue_link*() right after submit
      io_uring: allocate io_kiocb upfront
      io_uring: Use submit info inlined into req
      io_uring: use inlined struct sqe_submit
      io_uring: Fix getting file for non-fd opcodes

YueHaibing (1):
      io-wq: use kfree_rcu() to simplify the code

 MAINTAINERS                     |    5 +-
 fs/Kconfig                      |    3 +
 fs/Makefile                     |    1 +
 fs/io-wq.c                      | 1065 +++++++++++++++++++
 fs/io-wq.h                      |   74 ++
 fs/io_uring.c                   | 2213 ++++++++++++++++++++++++++-------------
 include/Kbuild                  |    1 +
 include/linux/sched.h           |    1 +
 include/linux/socket.h          |    3 +
 include/trace/events/io_uring.h |  358 +++++++
 include/uapi/linux/io_uring.h   |   24 +-
 init/Kconfig                    |    1 +
 kernel/sched/core.c             |   16 +-
 net/socket.c                    |   65 +-
 14 files changed, 3098 insertions(+), 732 deletions(-)
 create mode 100644 fs/io-wq.c
 create mode 100644 fs/io-wq.h
 create mode 100644 include/trace/events/io_uring.h

commit 22ffc78881bce32ae83dbd315fb15cd7ef8a6e4a
Merge: a3085d8079be b226c9e1f4cb
Author: Jens Axboe <axboe@kernel.dk>
Date:   Fri Nov 15 18:03:21 2019 -0700

    Merge branch 'master' into for-next
    
    * master: (98 commits)
      afs: Fix race in commit bulk status fetch
      KVM: Add a comment describing the /dev/kvm no_compat handling
      drm/amdgpu: fix null pointer deref in firmware header printing
      rsxx: add missed destroy_workqueue calls in remove
      iocost: check active_list of all the ancestors in iocg_activate()
      rbd: silence bogus uninitialized warning in rbd_object_map_update_finish()
      ceph: increment/decrement dio counter on async requests
      ceph: take the inode lock before acquiring cap refs
      ALSA: usb-audio: Fix incorrect size check for processing/extension units
      KVM: x86/mmu: Take slots_lock when using kvm_mmu_zap_all_fast()
      kbuild: tell sparse about the $ARCH
      sparc: vdso: fix build error of vdso32
      block, bfq: deschedule empty bfq_queues not referred by any process
      mmc: sdhci-of-at91: fix quirk2 overwrite
      ALSA: usb-audio: Fix incorrect NULL check in create_yamaha_midi_quirk()
      io_uring: ensure registered buffer import returns the IO length
      io_uring: Fix getting file for timeout
      drm/i915/tgl: MOCS table update
      Revert "drm/i915/ehl: Update MOCS table for EHL"
      KVM: Forbid /dev/kvm being opened by a compat task when CONFIG_KVM_COMPAT=n
      ...
    
    Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --cc fs/io_uring.c
index 25f0e8fd935b,2c819c3c855d..187e1c5021ac
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@@ -340,8 -326,7 +340,9 @@@ struct io_kiocb 
  #define REQ_F_TIMEOUT		1024	/* timeout request */
  #define REQ_F_ISREG		2048	/* regular file */
  #define REQ_F_MUST_PUNT		4096	/* must be punted even for NONBLOCK */
- #define REQ_F_INFLIGHT		8192	/* on inflight list */
- #define REQ_F_COMP_LOCKED	16384	/* completion under lock */
+ #define REQ_F_TIMEOUT_NOSEQ	8192	/* no timeout sequence */
++#define REQ_F_INFLIGHT		16384	/* on inflight list */
++#define REQ_F_COMP_LOCKED	32768	/* completion under lock */
  	u64			user_data;
  	u32			result;
  	u32			sequence;
@@@ -482,9 -454,13 +483,13 @@@ static struct io_kiocb *io_get_timeout_
  	struct io_kiocb *req;
  
  	req = list_first_entry_or_null(&ctx->timeout_list, struct io_kiocb, list);
- 	if (req && !__req_need_defer(req)) {
- 		list_del_init(&req->list);
- 		return req;
+ 	if (req) {
+ 		if (req->flags & REQ_F_TIMEOUT_NOSEQ)
+ 			return NULL;
 -		if (!__io_sequence_defer(ctx, req)) {
++		if (!__req_need_defer(req)) {
+ 			list_del_init(&req->list);
+ 			return req;
+ 		}
  	}
  
  	return NULL;
@@@ -2296,12 -1946,7 +2301,13 @@@ static int io_timeout(struct io_kiocb *
  	if (get_timespec64(&ts, u64_to_user_ptr(sqe->addr)))
  		return -EFAULT;
  
 +	if (flags & IORING_TIMEOUT_ABS)
 +		mode = HRTIMER_MODE_ABS;
 +	else
 +		mode = HRTIMER_MODE_REL;
 +
 +	hrtimer_init(&req->timeout.timer, CLOCK_MONOTONIC, mode);
+ 	req->flags |= REQ_F_TIMEOUT;
  
  	/*
  	 * sqe->off holds how many events that need to occur for this
@@@ -2352,82 -2004,14 +2365,83 @@@
  		nxt->sequence++;
  	}
  	req->sequence -= span;
+ add:
  	list_add(&req->list, entry);
 +	req->timeout.timer.function = io_timeout_fn;
 +	hrtimer_start(&req->timeout.timer, timespec64_to_ktime(ts), mode);
  	spin_unlock_irq(&ctx->completion_lock);
 +	return 0;
 +}
  
 -	hrtimer_init(&req->timeout.timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
 -	req->timeout.timer.function = io_timeout_fn;
 -	hrtimer_start(&req->timeout.timer, timespec64_to_ktime(ts),
 -			HRTIMER_MODE_REL);
 +static bool io_cancel_cb(struct io_wq_work *work, void *data)
 +{
 +	struct io_kiocb *req = container_of(work, struct io_kiocb, work);
 +
 +	return req->user_data == (unsigned long) data;
 +}
 +
 +static int io_async_cancel_one(struct io_ring_ctx *ctx, void *sqe_addr)
 +{
 +	enum io_wq_cancel cancel_ret;
 +	int ret = 0;
 +
 +	cancel_ret = io_wq_cancel_cb(ctx->io_wq, io_cancel_cb, sqe_addr);
 +	switch (cancel_ret) {
 +	case IO_WQ_CANCEL_OK:
 +		ret = 0;
 +		break;
 +	case IO_WQ_CANCEL_RUNNING:
 +		ret = -EALREADY;
 +		break;
 +	case IO_WQ_CANCEL_NOTFOUND:
 +		ret = -ENOENT;
 +		break;
 +	}
 +
 +	return ret;
 +}
 +
 +static void io_async_find_and_cancel(struct io_ring_ctx *ctx,
 +				     struct io_kiocb *req, __u64 sqe_addr,
 +				     struct io_kiocb **nxt)
 +{
 +	unsigned long flags;
 +	int ret;
 +
 +	ret = io_async_cancel_one(ctx, (void *) (unsigned long) sqe_addr);
 +	if (ret != -ENOENT) {
 +		spin_lock_irqsave(&ctx->completion_lock, flags);
 +		goto done;
 +	}
 +
 +	spin_lock_irqsave(&ctx->completion_lock, flags);
 +	ret = io_timeout_cancel(ctx, sqe_addr);
 +	if (ret != -ENOENT)
 +		goto done;
 +	ret = io_poll_cancel(ctx, sqe_addr);
 +done:
 +	io_cqring_fill_event(req, ret);
 +	io_commit_cqring(ctx);
 +	spin_unlock_irqrestore(&ctx->completion_lock, flags);
 +	io_cqring_ev_posted(ctx);
 +
 +	if (ret < 0 && (req->flags & REQ_F_LINK))
 +		req->flags |= REQ_F_FAIL_LINK;
 +	io_put_req_find_next(req, nxt);
 +}
 +
 +static int io_async_cancel(struct io_kiocb *req, const struct io_uring_sqe *sqe,
 +			   struct io_kiocb **nxt)
 +{
 +	struct io_ring_ctx *ctx = req->ctx;
 +
 +	if (unlikely(ctx->flags & IORING_SETUP_IOPOLL))
 +		return -EINVAL;
 +	if (sqe->flags || sqe->ioprio || sqe->off || sqe->len ||
 +	    sqe->cancel_flags)
 +		return -EINVAL;
 +
 +	io_async_find_and_cancel(ctx, req, READ_ONCE(sqe->addr), NULL);
  	return 0;
  }
  

-- 
Jens Axboe