public inbox for [email protected]
 help / color / mirror / Atom feed
From: Kanchan Joshi <[email protected]>
To: [email protected], [email protected],
	[email protected]
Cc: [email protected], [email protected], [email protected],
	[email protected], [email protected], [email protected],
	[email protected]
Subject: [RFC 00/13] uring-passthru for nvme
Date: Mon, 20 Dec 2021 19:47:21 +0530	[thread overview]
Message-ID: <[email protected]> (raw)
In-Reply-To: CGME20211220142227epcas5p280851b0a62baa78379979eb81af7a096@epcas5p2.samsung.com

Here is a revamped series on uring-passthru which is on top of Jens
"nvme-passthru-wip.2" branch.
https://git.kernel.dk/cgit/linux-block/commit/?h=nvme-passthru-wip.2

This scales much better than before with the addition of following:
- plugging
- passthru polling (sync and async; sync part comes from a patch that
  Keith did earlier)
- bio-cache (this is regardless of irq/polling since we submit/complete in
  task-contex anyway. Currently kicks in when fixed-buffer option is
also passed, but that's primarily to keep the plumbing simple)

Also the feedback from Christoph (previous fixed-buffer series) is in
which has streamlined the plumbing.

I look forward to further feedback/comments.

KIOPS(512b) on P5800x looked like this:

QD    uring    pt    uring-poll    pt-poll
8      538     589      831         902
64     967     1131     1351        1378
256    1043    1230     1376        1429

Here uring is operating on block-interface (nvme0n1) while 'pt' refers
to uring-passthru operating on char-interface (ng0n1).

Perf/testing is with this custom fio that turnes regular io into
passthru on supplying "uring_cmd=1" option.
https://github.com/joshkan/fio/tree/nvme-passthru-wip-polling
Example command-line:
fio -iodepth=256 -rw=randread -ioengine=io_uring -bs=512 -numjobs=1 -runtime=60 -group_reporting -iodepth_batch_submit=64 -iodepth_batch_complete_min=1 -iodepth_batch_complete_max=64 -fixedbufs=1 -hipri=1 -sqthread_poll=0 -filename=/dev/ng0n1 -name=io_uring_256 -uring_cmd=1

background/context:
https://linuxplumbersconf.org/event/11/contributions/989/attachments/747/1723/lpc-2021-building-a-fast-passthru.pdf

Changes from v5:
https://lore.kernel.org/linux-nvme/[email protected]/
1. Fixed-buffer passthru with same ioctl code + other feedback from hch
2. Plugging (from Jens)
3. Sync polling (from Keith)
3. Async polling via io_uring
4. Enable bio-cache for fixed-buffer passthru

Changes from v4:
https://lore.kernel.org/linux-nvme/[email protected]/
1. Moved to v5 branch of Jens, adapted to task-work changes in io_uring
2. Removed support for block-passthrough (over nvme0n1) for now
3. Added support for char-passthrough (over ng0n1)
4. Added fixed-buffer passthrough in io_uring and nvme plumbing


Anuj Gupta (3):
  io_uring: mark iopoll not supported for uring-cmd
  io_uring: modify unused field in io_uring_cmd to store flags
  io_uring: add support for uring_cmd with fixed-buffer

Jens Axboe (2):
  io_uring: plug for async bypass
  block: wire-up support for plugging

Kanchan Joshi (6):
  io_uring: add infra for uring_cmd completion in submitter-task
  nvme: wire-up support for async-passthru on char-device.
  io_uring: add flag and helper for fixed-buffer uring-cmd
  nvme: enable passthrough with fixed-buffer
  block: factor out helper for bio allocation from cache
  nvme: enable bio-cache for fixed-buffer passthru

Keith Busch (1):
  nvme: allow user passthrough commands to poll

Pankaj Raghav (1):
  nvme: Add async passthru polling support

 block/bio.c                     |  43 +++--
 block/blk-map.c                 |  46 ++++++
 block/blk-mq.c                  |  93 +++++------
 drivers/nvme/host/core.c        |  21 ++-
 drivers/nvme/host/ioctl.c       | 271 ++++++++++++++++++++++++++++----
 drivers/nvme/host/multipath.c   |   2 +
 drivers/nvme/host/nvme.h        |  13 +-
 drivers/nvme/host/pci.c         |   4 +-
 drivers/nvme/target/passthru.c  |   2 +-
 fs/io_uring.c                   | 113 +++++++++++--
 include/linux/bio.h             |   1 +
 include/linux/blk-mq.h          |   4 +
 include/linux/io_uring.h        |  26 ++-
 include/uapi/linux/io_uring.h   |   6 +-
 include/uapi/linux/nvme_ioctl.h |   4 +
 15 files changed, 542 insertions(+), 107 deletions(-)

-- 
2.25.1


       reply	other threads:[~2021-12-21  2:56 UTC|newest]

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <CGME20211220142227epcas5p280851b0a62baa78379979eb81af7a096@epcas5p2.samsung.com>
2021-12-20 14:17 ` Kanchan Joshi [this message]
     [not found]   ` <CGME20211220142228epcas5p2978d92d38f2015148d5f72913d6dbc3e@epcas5p2.samsung.com>
2021-12-20 14:17     ` [RFC 01/13] io_uring: add infra for uring_cmd completion in submitter-task Kanchan Joshi
2022-02-17  2:13       ` Luis Chamberlain
2022-02-17 15:39         ` Kanchan Joshi
2022-02-17 15:50           ` Jens Axboe
2022-02-17 17:56             ` Luis Chamberlain
2022-02-18 17:41               ` Kanchan Joshi
2022-02-17 18:46             ` Luis Chamberlain
2022-02-17 18:53               ` Jens Axboe
     [not found]   ` <CGME20211220142231epcas5p1482c78f91feabdbc3e62341790ab22e1@epcas5p1.samsung.com>
2021-12-20 14:17     ` [RFC 02/13] nvme: wire-up support for async-passthru on char-device Kanchan Joshi
     [not found]   ` <CGME20211220142233epcas5p3b54aa591fb7b81bfb58bc33b5f92a2d3@epcas5p3.samsung.com>
2021-12-20 14:17     ` [RFC 03/13] io_uring: mark iopoll not supported for uring-cmd Kanchan Joshi
2022-02-17  2:16       ` Luis Chamberlain
2022-02-17  2:52         ` Jens Axboe
     [not found]   ` <CGME20211220142235epcas5p3b8d56cd39d9710278ec3360be47f2cca@epcas5p3.samsung.com>
2021-12-20 14:17     ` [RFC 04/13] io_uring: modify unused field in io_uring_cmd to store flags Kanchan Joshi
     [not found]   ` <CGME20211220142237epcas5p48729a52293e4f7627e6ec53ca67b9c58@epcas5p4.samsung.com>
2021-12-20 14:17     ` [RFC 05/13] io_uring: add flag and helper for fixed-buffer uring-cmd Kanchan Joshi
     [not found]   ` <CGME20211220142239epcas5p3efc3c89bd536f3f5d728c81bc550e143@epcas5p3.samsung.com>
2021-12-20 14:17     ` [RFC 06/13] io_uring: add support for uring_cmd with fixed-buffer Kanchan Joshi
     [not found]   ` <CGME20211220142242epcas5p45dddab51a9f20a8ec3d8b8e4f1dda40a@epcas5p4.samsung.com>
2021-12-20 14:17     ` [RFC 07/13] nvme: enable passthrough " Kanchan Joshi
     [not found]   ` <CGME20211220142244epcas5p2f311ed168b8f31b9301bcc2002076db4@epcas5p2.samsung.com>
2021-12-20 14:17     ` [RFC 08/13] io_uring: plug for async bypass Kanchan Joshi
     [not found]   ` <CGME20211220142246epcas5p303c64b6b1b832c7fcd5ac31fc79c91d1@epcas5p3.samsung.com>
2021-12-20 14:17     ` [RFC 09/13] block: wire-up support for plugging Kanchan Joshi
     [not found]   ` <CGME20211220142248epcas5p1e5904e10396f8cdea54bbd8d7aeca9a6@epcas5p1.samsung.com>
2021-12-20 14:17     ` [RFC 10/13] block: factor out helper for bio allocation from cache Kanchan Joshi
     [not found]   ` <CGME20211220142250epcas5p34b9d93b1dd3388af6209a4223befe40f@epcas5p3.samsung.com>
2021-12-20 14:17     ` [RFC 11/13] nvme: enable bio-cache for fixed-buffer passthru Kanchan Joshi
     [not found]   ` <CGME20211220142252epcas5p4611297f9970acbc8ee3b0e325ca5ceec@epcas5p4.samsung.com>
2021-12-20 14:17     ` [RFC 12/13] nvme: allow user passthrough commands to poll Kanchan Joshi
     [not found]   ` <CGME20211220142256epcas5p49e0804ff8b075e8063259f94ccc9ced0@epcas5p4.samsung.com>
2021-12-20 14:17     ` [RFC 13/13] nvme: Add async passthru polling support Kanchan Joshi
2021-12-21  3:45   ` [RFC 00/13] uring-passthru for nvme Jens Axboe
2021-12-21 14:36     ` Kanchan Joshi

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    [email protected] \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox