[GIT PULL] io_uring changes for 6.13-rc1

[GIT PULL] io_uring changes for 6.13-rc1

[Jens Axboe]

Hi Linus,

Here are the io_uring changes slated for the 6.13 kernel release. This
pull request contains:

- Cleanups of the eventfd handling code, making it fully private.

- Support for sending a sync message to another ring, without having a
  ring available to send a normal async message.

- Get rid of the separate unlocked hash table, unify everything around
  the single locked one.

- Add support for ring resizing. It can be hard to appropriately size
  the CQ ring upfront, if the application doesn't know how busy it will
  be. This results in applications sizing rings for the most busy case,
  which can be wasteful. With ring resizing, they can start small and
  grow the ring, if needed.

- Add support for fixed wait regions, rather than needing to copy the
  same wait data tons of times for each wait operation.

- Rewrite the resource node handling, which before was serialized per
  ring. This caused issues with particularly fixed files, where one file
  waiting on IO could hold up putting and freeing of other unrelated
  files. Now each node is handled separately. New code is much simpler
  too, and was a net 250 line reduction in code.

- Add support for just doing partial buffer clones, rather than always
  cloning the entire buffer table.

- Series adding static NAPI support, where a specific NAPI instance is
  used rather than having a list of them available that need lookup.

- Add support for mapped regions, and also convert the fixed wait
  support mentioned above to that concept. This avoids doing special
  mappings for various planned features, and folds the existing
  registered wait into that too.

- Add support for hybrid IO polling, which is a variant of strict IOPOLL
  but with an initial sleep delay to avoid spinning too early and
  wasting resources on devices that aren't necessarily in the < 5 usec
  category wrt latencies.

- Various cleanups and little fixes.

Please pull!


The following changes since commit 42f7652d3eb527d03665b09edac47f85fb600924:

  Linux 6.12-rc4 (2024-10-20 15:19:38 -0700)

are available in the Git repository at:

  git://git.kernel.dk/linux.git tags/for-6.13/io_uring-20241118

for you to fetch changes up to a652958888fb1ada3e4f6b548576c2d2c1b60d66:

  io_uring/region: fix error codes after failed vmap (2024-11-17 09:01:35 -0700)

----------------------------------------------------------------
for-6.13/io_uring-20241118

----------------------------------------------------------------
Al Viro (1):
      switch io_msg_ring() to CLASS(fd)

Jens Axboe (50):
      io_uring/eventfd: abstract out ev_fd put helper
      io_uring/eventfd: check for the need to async notifier earlier
      io_uring/eventfd: move actual signaling part into separate helper
      io_uring/eventfd: move trigger check into a helper
      io_uring/eventfd: abstract out ev_fd grab + release helpers
      io_uring/eventfd: move ctx->evfd_last_cq_tail into io_ev_fd
      io_uring/msg_ring: refactor a few helper functions
      io_uring/msg_ring: add support for sending a sync message
      io_uring/poll: remove 'ctx' argument from io_poll_req_delete()
      io_uring/poll: get rid of unlocked cancel hash
      io_uring/poll: get rid of io_poll_tw_hash_eject()
      io_uring/poll: get rid of per-hashtable bucket locks
      io_uring/cancel: get rid of init_hash_table() helper
      io_uring: move cancel hash tables to kvmalloc/kvfree
      io_uring/rsrc: don't assign bvec twice in io_import_fixed()
      io_uring/uring_cmd: get rid of using req->imu
      io_uring/rw: get rid of using req->imu
      io_uring: remove 'issue_flags' argument for io_req_set_rsrc_node()
      io_uring/net: move send zc fixed buffer import to issue path
      io_uring: kill 'imu' from struct io_kiocb
      io_uring: move max entry definition and ring sizing into header
      io_uring: abstract out a bit of the ring filling logic
      io_uring/memmap: explicitly return -EFAULT for mmap on NULL rings
      io_uring/register: add IORING_REGISTER_RESIZE_RINGS
      io_uring/sqpoll: wait on sqd->wait for thread parking
      io_uring: switch struct ext_arg from __kernel_timespec to timespec64
      io_uring: change io_get_ext_arg() to use uaccess begin + end
      io_uring: add support for fixed wait regions
      io_uring/nop: add support for testing registered files and buffers
      io_uring/rsrc: move struct io_fixed_file to rsrc.h header
      io_uring: specify freeptr usage for SLAB_TYPESAFE_BY_RCU io_kiocb cache
      io_uring/splice: open code 2nd direct file assignment
      io_uring/rsrc: kill io_charge_rsrc_node()
      io_uring/rsrc: get rid of per-ring io_rsrc_node list
      io_uring/rsrc: get rid of io_rsrc_node allocation cache
      io_uring/rsrc: add an empty io_rsrc_node for sparse buffer entries
      io_uring: only initialize io_kiocb rsrc_nodes when needed
      io_uring/rsrc: unify file and buffer resource tables
      io_uring/rsrc: add io_rsrc_node_lookup() helper
      io_uring/filetable: remove io_file_from_index() helper
      io_uring/filetable: kill io_reset_alloc_hint() helper
      io_uring/rsrc: add io_reset_rsrc_node() helper
      io_uring/rsrc: get rid of the empty node and dummy_ubuf
      io_uring/rsrc: allow cloning at an offset
      io_uring/rsrc: allow cloning with node replacements
      io_uring/rsrc: encode node type and ctx together
      io_uring/rsrc: split io_kiocb node type assignments
      io_uring: move cancelations to be io_uring_task based
      io_uring: remove task ref helpers
      io_uring: move struct io_kiocb from task_struct to io_uring_task

Ming Lei (4):
      io_uring/rsrc: pass 'struct io_ring_ctx' reference to rsrc helpers
      io_uring/rsrc: remove '->ctx_ptr' of 'struct io_rsrc_node'
      io_uring/rsrc: add & apply io_req_assign_buf_node()
      io_uring/uring_cmd: fix buffer index retrieval

Olivier Langlois (6):
      io_uring/napi: protect concurrent io_napi_entry timeout accesses
      io_uring/napi: fix io_napi_entry RCU accesses
      io_uring/napi: improve __io_napi_add
      io_uring/napi: Use lock guards
      io_uring/napi: clean up __io_napi_do_busy_loop
      io_uring/napi: add static napi tracking strategy

Pavel Begunkov (17):
      io_uring: kill io_llist_xchg
      io_uring: static_key for !IORING_SETUP_NO_SQARRAY
      io_uring: clean up cqe trace points
      io_uring/net: split send and sendmsg prep helpers
      io_uring/net: don't store send address ptr
      io_uring/net: don't alias send user pointer reads
      io_uring/net: clean up io_msg_copy_hdr
      io_uring: prevent speculating sq_array indexing
      io_uring: avoid normal tw intermediate fallback
      io_uring: fix invalid hybrid polling ctx leaks
      io_uring: fortify io_pin_pages with a warning
      io_uring: disable ENTER_EXT_ARG_REG for IOPOLL
      io_uring: temporarily disable registered waits
      io_uring: introduce concept of memory regions
      io_uring: add memory region registration
      io_uring: restore back registered wait arguments
      io_uring/region: fix error codes after failed vmap

hexue (1):
      io_uring: add support for hybrid IOPOLL

 include/linux/io_uring/cmd.h    |   2 +-
 include/linux/io_uring_types.h  |  88 ++++--
 include/trace/events/io_uring.h |  24 +-
 include/uapi/linux/io_uring.h   | 119 +++++++-
 io_uring/cancel.c               |  20 +-
 io_uring/cancel.h               |   1 -
 io_uring/eventfd.c              | 137 ++++++---
 io_uring/fdinfo.c               |  88 ++++--
 io_uring/filetable.c            |  71 ++---
 io_uring/filetable.h            |  35 +--
 io_uring/futex.c                |   4 +-
 io_uring/futex.h                |   4 +-
 io_uring/io_uring.c             | 433 ++++++++++++++------------
 io_uring/io_uring.h             |  30 +-
 io_uring/memmap.c               |  83 +++++
 io_uring/memmap.h               |  14 +
 io_uring/msg_ring.c             |  91 ++++--
 io_uring/msg_ring.h             |   1 +
 io_uring/napi.c                 | 184 +++++++----
 io_uring/napi.h                 |   8 +-
 io_uring/net.c                  | 112 ++++---
 io_uring/nop.c                  |  47 ++-
 io_uring/notif.c                |   7 +-
 io_uring/opdef.c                |   2 +
 io_uring/poll.c                 | 181 +++--------
 io_uring/poll.h                 |   2 +-
 io_uring/register.c             | 299 +++++++++++++++++-
 io_uring/rsrc.c                 | 657 +++++++++++++++-------------------------
 io_uring/rsrc.h                 |  97 +++---
 io_uring/rw.c                   | 105 +++++--
 io_uring/splice.c               |  42 ++-
 io_uring/splice.h               |   1 +
 io_uring/sqpoll.c               |   3 +-
 io_uring/tctx.c                 |   1 +
 io_uring/timeout.c              |  16 +-
 io_uring/timeout.h              |   2 +-
 io_uring/uring_cmd.c            |  27 +-
 io_uring/uring_cmd.h            |   2 +-
 io_uring/waitid.c               |   6 +-
 io_uring/waitid.h               |   2 +-
 40 files changed, 1825 insertions(+), 1223 deletions(-)

-- 
Jens Axboe

Re: [GIT PULL] io_uring changes for 6.13-rc1

[Sasha Levin]

Hi Jens, Thomas,

On Mon, Nov 18, 2024 at 07:22:59AM -0700, Jens Axboe wrote:
>hexue (1):
>      io_uring: add support for hybrid IOPOLL

After merging of this pull request into linus-next, I've started seeing
build errors:

/builds/linux/io_uring/rw.c: In function 'io_hybrid_iopoll_delay':
/builds/linux/io_uring/rw.c:1179:2: error: implicit declaration of function 'hrtimer_init_sleeper_on_stack'; did you mean 'hrtimer_setup_sleeper_on_stack'? [-Werror=implicit-function-declaration]
   hrtimer_init_sleeper_on_stack(&timer, CLOCK_MONOTONIC, mode);
   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   hrtimer_setup_sleeper_on_stack

This is because 01ee194d1aba ("io_uring: add support for hybrid IOPOLL")
adds a call to hrtimer_init_sleeper_on_stack() which was removed earlier
today in Thomas's PR[1], specifically in commit f3bef7aaa6c8
("hrtimers: Delete hrtimer_init_sleeper_on_stack()").


[1] https://lore.kernel.org/all/173195758632.1896928.11371209657780930206.tglx@xen13/

-- 
Thanks,
Sasha

Re: [GIT PULL] io_uring changes for 6.13-rc1

[Sasha Levin]

On Mon, Nov 18, 2024 at 05:06:46PM -0500, Sasha Levin wrote:
>Hi Jens, Thomas,
>
>On Mon, Nov 18, 2024 at 07:22:59AM -0700, Jens Axboe wrote:
>>hexue (1):
>>     io_uring: add support for hybrid IOPOLL
>
>After merging of this pull request into linus-next, I've started seeing
>build errors:
>
>/builds/linux/io_uring/rw.c: In function 'io_hybrid_iopoll_delay':
>/builds/linux/io_uring/rw.c:1179:2: error: implicit declaration of function 'hrtimer_init_sleeper_on_stack'; did you mean 'hrtimer_setup_sleeper_on_stack'? [-Werror=implicit-function-declaration]
>  hrtimer_init_sleeper_on_stack(&timer, CLOCK_MONOTONIC, mode);
>  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>  hrtimer_setup_sleeper_on_stack
>
>This is because 01ee194d1aba ("io_uring: add support for hybrid IOPOLL")
>adds a call to hrtimer_init_sleeper_on_stack() which was removed earlier
>today in Thomas's PR[1], specifically in commit f3bef7aaa6c8
>("hrtimers: Delete hrtimer_init_sleeper_on_stack()").

Linus,

Looks like this is a simple
s/hrtimer_init_sleeper_on_stack/hrtimer_setup_sleeper_on_stack , so this
issue could be addressed by replacing the new call with
hrtimer_setup_sleeper_on_stack() either in the io_uring or the
timers/core merge, whichever you pull last.

-- 
Thanks,
Sasha

Re: [GIT PULL] io_uring changes for 6.13-rc1

[Jens Axboe]

On 11/18/24 3:06 PM, Sasha Levin wrote:
> Hi Jens, Thomas,
> 
> On Mon, Nov 18, 2024 at 07:22:59AM -0700, Jens Axboe wrote:
>> hexue (1):
>>      io_uring: add support for hybrid IOPOLL
> 
> After merging of this pull request into linus-next, I've started seeing
> build errors:
> 
> /builds/linux/io_uring/rw.c: In function 'io_hybrid_iopoll_delay':
> /builds/linux/io_uring/rw.c:1179:2: error: implicit declaration of function 'hrtimer_init_sleeper_on_stack'; did you mean 'hrtimer_setup_sleeper_on_stack'? [-Werror=implicit-function-declaration]
>   hrtimer_init_sleeper_on_stack(&timer, CLOCK_MONOTONIC, mode);
>   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>   hrtimer_setup_sleeper_on_stack
> 
> This is because 01ee194d1aba ("io_uring: add support for hybrid IOPOLL")
> adds a call to hrtimer_init_sleeper_on_stack() which was removed earlier
> today in Thomas's PR[1], specifically in commit f3bef7aaa6c8
> ("hrtimers: Delete hrtimer_init_sleeper_on_stack()").

Right, forgot to mention that. linux-next has been carrying a fixup for
that which I was going to link, but it's not on a public list for some
reason. In any case, it's trivial to fixup, just change

hrtimer_init_sleeper_on_stack(&timer, CLOCK_MONOTONIC, mode);

to

hrtimer_setup_sleeper_on_stack(&timer, CLOCK_MONOTONIC, mode);

in io_uring/rw.c while merging either of the two trees that go in last.

FWIW, this kind of change should be done before -rc1 rather than be
queued up earlier. That's usually the best way to do these things and
avoid -next or merge window breakage. I'm guessing this tree isn't the
only one going to get hit by that.

-- 
Jens Axboe

Re: [GIT PULL] io_uring changes for 6.13-rc1

[Jens Axboe]

On 11/18/24 4:27 PM, Jens Axboe wrote:
> On 11/18/24 3:06 PM, Sasha Levin wrote:
>> Hi Jens, Thomas,
>>
>> On Mon, Nov 18, 2024 at 07:22:59AM -0700, Jens Axboe wrote:
>>> hexue (1):
>>>      io_uring: add support for hybrid IOPOLL
>>
>> After merging of this pull request into linus-next, I've started seeing
>> build errors:
>>
>> /builds/linux/io_uring/rw.c: In function 'io_hybrid_iopoll_delay':
>> /builds/linux/io_uring/rw.c:1179:2: error: implicit declaration of function 'hrtimer_init_sleeper_on_stack'; did you mean 'hrtimer_setup_sleeper_on_stack'? [-Werror=implicit-function-declaration]
>>   hrtimer_init_sleeper_on_stack(&timer, CLOCK_MONOTONIC, mode);
>>   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>   hrtimer_setup_sleeper_on_stack
>>
>> This is because 01ee194d1aba ("io_uring: add support for hybrid IOPOLL")
>> adds a call to hrtimer_init_sleeper_on_stack() which was removed earlier
>> today in Thomas's PR[1], specifically in commit f3bef7aaa6c8
>> ("hrtimers: Delete hrtimer_init_sleeper_on_stack()").
> 
> Right, forgot to mention that. linux-next has been carrying a fixup for
> that which I was going to link, but it's not on a public list for some
> reason.

My bad, it is on lkml:

https://lore.kernel.org/lkml/20241108143328.6d819fcb@canb.auug.org.au/

-- 
Jens Axboe

Re: [GIT PULL] io_uring changes for 6.13-rc1

[pr-tracker-bot]

The pull request you sent on Mon, 18 Nov 2024 07:22:59 -0700:

> git://git.kernel.dk/linux.git tags/for-6.13/io_uring-20241118

has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/8350142a4b4cedebfa76cd4cc6e5a7ba6a330629

Thank you!

-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/prtracker.html