public inbox for [email protected]
 help / color / mirror / Atom feed
From: Christoph Hellwig <[email protected]>
To: Jens Axboe <[email protected]>
Cc: Christian Brauner <[email protected]>,
	Keith Busch <[email protected]>, Sagi Grimberg <[email protected]>,
	Kanchan Joshi <[email protected]>,
	Hui Qi <[email protected]>,
	Nitesh Shetty <[email protected]>, Jan Kara <[email protected]>,
	Pavel Begunkov <[email protected]>,
	[email protected], [email protected],
	[email protected], [email protected],
	[email protected]
Subject: support block layer write streams and FDP
Date: Tue, 19 Nov 2024 13:16:14 +0100	[thread overview]
Message-ID: <[email protected]> (raw)

Hi all,

as a small interruptions to regularly scheduled culture wars this series
implements a properly layered approach to block layer write streams.

This is based on Keith "Subject: [PATCHv11 0/9] write hints with nvme fdp
and scsi streams", but doesn't bypass the file systems.

The rough idea is that block devices can expose a number of distinct
write streams, and bio submitter can pick on them.  All bios that do
not pick an explicit write stream get the default one.  On the driver
layer this is wird up to NVMe FDP, but it should also work for SCSI
and NVMe streams if someone cares enough.  On the upper layer the only
consuder right now are the block device node file operations, which
either support an explicit stream selection through io_uring, or
by mapping the old per-inode life time hints to streams.

The stream API is designed to also implementable by other files,
so a statx extension to expose the number of handles, and their
granularity is added as well.

This currently does not do the write hint mapping for file systems,
which needs to be done in the file system and under careful consideration
about how many of these streams the file system wants to grant to
the application - if any.  It also doesn't support querying how much
has been written to a "granularity unit" aka reclaim unit in NVMe,
which is essential if you want a WAF=1 but apparently not needed for
the current urgent users.

The last patch to support write streams on partitions works, but feels
like a not very nice interface to me, and might allow only to restricted
mappings for some.  It would be great if users that absolutely require
partition support to speak up and help improve it, otherwise I'd suggest
to skip it for the initial submission.

The series is based on Jens' for-next branch as of today, and also
available as git tree:

    git://git.infradead.org/users/hch/misc.git block-write-streams

Gitweb:

    http://git.infradead.org/?p=users/hch/misc.git;a=shortlog;h=refs/heads/block-write-streams

Diffstat:
 Documentation/ABI/stable/sysfs-block |   15 +++
 block/bdev.c                         |   15 +++
 block/bio.c                          |    2 
 block/blk-core.c                     |    2 
 block/blk-crypto-fallback.c          |    1 
 block/blk-merge.c                    |   39 ++-------
 block/blk-sysfs.c                    |    6 +
 block/bounce.c                       |    1 
 block/fops.c                         |   23 +++++
 block/genhd.c                        |   52 ++++++++++++
 block/partitions/core.c              |    6 -
 drivers/nvme/host/core.c             |  151 ++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/nvme.h             |   10 +-
 fs/stat.c                            |    2 
 include/linux/blk_types.h            |    8 +
 include/linux/blkdev.h               |   16 +++
 include/linux/fs.h                   |    1 
 include/linux/nvme.h                 |   77 +++++++++++++++++
 include/linux/stat.h                 |    2 
 include/uapi/linux/io_uring.h        |    4 
 include/uapi/linux/stat.h            |    7 +
 io_uring/io_uring.c                  |    2 
 io_uring/rw.c                        |    2 
 23 files changed, 405 insertions(+), 39 deletions(-)

             reply	other threads:[~2024-11-19 12:16 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-11-19 12:16 Christoph Hellwig [this message]
2024-11-19 12:16 ` [PATCH 01/15] fs: add write stream information to statx Christoph Hellwig
2024-11-19 12:16 ` [PATCH 02/15] fs: add a write stream field to the kiocb Christoph Hellwig
2024-11-19 12:16 ` [PATCH 03/15] io_uring: enable passing a per-io write stream Christoph Hellwig
2024-11-19 12:16 ` [PATCH 04/15] block: don't bother checking the data direction for merges Christoph Hellwig
2024-11-19 12:16 ` [PATCH 05/15] block: req->bio is always set in the merge code Christoph Hellwig
2024-11-19 12:16 ` [PATCH 06/15] block: add a bi_write_stream field Christoph Hellwig
2024-11-19 12:16 ` [PATCH 07/15] block: introduce max_write_streams queue limit Christoph Hellwig
2024-11-19 12:16 ` [PATCH 08/15] block: introduce a write_stream_granularity " Christoph Hellwig
2024-11-19 12:16 ` [PATCH 09/15] block: expose write streams for block device nodes Christoph Hellwig
2024-11-19 12:16 ` [PATCH 10/15] nvme: store the endurance group id in struct nvme_ns_head Christoph Hellwig
2024-11-19 12:16 ` [PATCH 11/15] nvme: pass a void pointer to nvme_get/set_features for the result Christoph Hellwig
2024-11-19 12:16 ` [PATCH 12/15] nvme: add a nvme_get_log_lsi helper Christoph Hellwig
2024-11-19 12:16 ` [PATCH 13/15] nvme.h: add FDP definitions Christoph Hellwig
2024-11-19 12:16 ` [PATCH 14/15] nvme: enable FDP support Christoph Hellwig
2024-11-19 18:17   ` Keith Busch
2024-11-19 18:24     ` Christoph Hellwig
2024-11-19 22:49       ` Keith Busch
2024-11-20  6:03         ` Christoph Hellwig
2024-11-19 12:16 ` [PATCH 15/15] RFC: block: allow write streams on partitions Christoph Hellwig

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox