From: Kanchan Joshi <[email protected]>
To: [email protected], [email protected],
[email protected]
Cc: [email protected], [email protected], [email protected],
[email protected], [email protected], [email protected],
[email protected]
Subject: [RFC 00/13] uring-passthru for nvme
Date: Mon, 20 Dec 2021 19:47:21 +0530 [thread overview]
Message-ID: <[email protected]> (raw)
In-Reply-To: CGME20211220142227epcas5p280851b0a62baa78379979eb81af7a096@epcas5p2.samsung.com
Here is a revamped series on uring-passthru which is on top of Jens
"nvme-passthru-wip.2" branch.
https://git.kernel.dk/cgit/linux-block/commit/?h=nvme-passthru-wip.2
This scales much better than before with the addition of following:
- plugging
- passthru polling (sync and async; sync part comes from a patch that
Keith did earlier)
- bio-cache (this is regardless of irq/polling since we submit/complete in
task-contex anyway. Currently kicks in when fixed-buffer option is
also passed, but that's primarily to keep the plumbing simple)
Also the feedback from Christoph (previous fixed-buffer series) is in
which has streamlined the plumbing.
I look forward to further feedback/comments.
KIOPS(512b) on P5800x looked like this:
QD uring pt uring-poll pt-poll
8 538 589 831 902
64 967 1131 1351 1378
256 1043 1230 1376 1429
Here uring is operating on block-interface (nvme0n1) while 'pt' refers
to uring-passthru operating on char-interface (ng0n1).
Perf/testing is with this custom fio that turnes regular io into
passthru on supplying "uring_cmd=1" option.
https://github.com/joshkan/fio/tree/nvme-passthru-wip-polling
Example command-line:
fio -iodepth=256 -rw=randread -ioengine=io_uring -bs=512 -numjobs=1 -runtime=60 -group_reporting -iodepth_batch_submit=64 -iodepth_batch_complete_min=1 -iodepth_batch_complete_max=64 -fixedbufs=1 -hipri=1 -sqthread_poll=0 -filename=/dev/ng0n1 -name=io_uring_256 -uring_cmd=1
background/context:
https://linuxplumbersconf.org/event/11/contributions/989/attachments/747/1723/lpc-2021-building-a-fast-passthru.pdf
Changes from v5:
https://lore.kernel.org/linux-nvme/[email protected]/
1. Fixed-buffer passthru with same ioctl code + other feedback from hch
2. Plugging (from Jens)
3. Sync polling (from Keith)
3. Async polling via io_uring
4. Enable bio-cache for fixed-buffer passthru
Changes from v4:
https://lore.kernel.org/linux-nvme/[email protected]/
1. Moved to v5 branch of Jens, adapted to task-work changes in io_uring
2. Removed support for block-passthrough (over nvme0n1) for now
3. Added support for char-passthrough (over ng0n1)
4. Added fixed-buffer passthrough in io_uring and nvme plumbing
Anuj Gupta (3):
io_uring: mark iopoll not supported for uring-cmd
io_uring: modify unused field in io_uring_cmd to store flags
io_uring: add support for uring_cmd with fixed-buffer
Jens Axboe (2):
io_uring: plug for async bypass
block: wire-up support for plugging
Kanchan Joshi (6):
io_uring: add infra for uring_cmd completion in submitter-task
nvme: wire-up support for async-passthru on char-device.
io_uring: add flag and helper for fixed-buffer uring-cmd
nvme: enable passthrough with fixed-buffer
block: factor out helper for bio allocation from cache
nvme: enable bio-cache for fixed-buffer passthru
Keith Busch (1):
nvme: allow user passthrough commands to poll
Pankaj Raghav (1):
nvme: Add async passthru polling support
block/bio.c | 43 +++--
block/blk-map.c | 46 ++++++
block/blk-mq.c | 93 +++++------
drivers/nvme/host/core.c | 21 ++-
drivers/nvme/host/ioctl.c | 271 ++++++++++++++++++++++++++++----
drivers/nvme/host/multipath.c | 2 +
drivers/nvme/host/nvme.h | 13 +-
drivers/nvme/host/pci.c | 4 +-
drivers/nvme/target/passthru.c | 2 +-
fs/io_uring.c | 113 +++++++++++--
include/linux/bio.h | 1 +
include/linux/blk-mq.h | 4 +
include/linux/io_uring.h | 26 ++-
include/uapi/linux/io_uring.h | 6 +-
include/uapi/linux/nvme_ioctl.h | 4 +
15 files changed, 542 insertions(+), 107 deletions(-)
--
2.25.1
next parent reply other threads:[~2021-12-21 2:56 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CGME20211220142227epcas5p280851b0a62baa78379979eb81af7a096@epcas5p2.samsung.com>
2021-12-20 14:17 ` Kanchan Joshi [this message]
[not found] ` <CGME20211220142228epcas5p2978d92d38f2015148d5f72913d6dbc3e@epcas5p2.samsung.com>
2021-12-20 14:17 ` [RFC 01/13] io_uring: add infra for uring_cmd completion in submitter-task Kanchan Joshi
2022-02-17 2:13 ` Luis Chamberlain
2022-02-17 15:39 ` Kanchan Joshi
2022-02-17 15:50 ` Jens Axboe
2022-02-17 17:56 ` Luis Chamberlain
2022-02-18 17:41 ` Kanchan Joshi
2022-02-17 18:46 ` Luis Chamberlain
2022-02-17 18:53 ` Jens Axboe
[not found] ` <CGME20211220142231epcas5p1482c78f91feabdbc3e62341790ab22e1@epcas5p1.samsung.com>
2021-12-20 14:17 ` [RFC 02/13] nvme: wire-up support for async-passthru on char-device Kanchan Joshi
[not found] ` <CGME20211220142233epcas5p3b54aa591fb7b81bfb58bc33b5f92a2d3@epcas5p3.samsung.com>
2021-12-20 14:17 ` [RFC 03/13] io_uring: mark iopoll not supported for uring-cmd Kanchan Joshi
2022-02-17 2:16 ` Luis Chamberlain
2022-02-17 2:52 ` Jens Axboe
[not found] ` <CGME20211220142235epcas5p3b8d56cd39d9710278ec3360be47f2cca@epcas5p3.samsung.com>
2021-12-20 14:17 ` [RFC 04/13] io_uring: modify unused field in io_uring_cmd to store flags Kanchan Joshi
[not found] ` <CGME20211220142237epcas5p48729a52293e4f7627e6ec53ca67b9c58@epcas5p4.samsung.com>
2021-12-20 14:17 ` [RFC 05/13] io_uring: add flag and helper for fixed-buffer uring-cmd Kanchan Joshi
[not found] ` <CGME20211220142239epcas5p3efc3c89bd536f3f5d728c81bc550e143@epcas5p3.samsung.com>
2021-12-20 14:17 ` [RFC 06/13] io_uring: add support for uring_cmd with fixed-buffer Kanchan Joshi
[not found] ` <CGME20211220142242epcas5p45dddab51a9f20a8ec3d8b8e4f1dda40a@epcas5p4.samsung.com>
2021-12-20 14:17 ` [RFC 07/13] nvme: enable passthrough " Kanchan Joshi
[not found] ` <CGME20211220142244epcas5p2f311ed168b8f31b9301bcc2002076db4@epcas5p2.samsung.com>
2021-12-20 14:17 ` [RFC 08/13] io_uring: plug for async bypass Kanchan Joshi
[not found] ` <CGME20211220142246epcas5p303c64b6b1b832c7fcd5ac31fc79c91d1@epcas5p3.samsung.com>
2021-12-20 14:17 ` [RFC 09/13] block: wire-up support for plugging Kanchan Joshi
[not found] ` <CGME20211220142248epcas5p1e5904e10396f8cdea54bbd8d7aeca9a6@epcas5p1.samsung.com>
2021-12-20 14:17 ` [RFC 10/13] block: factor out helper for bio allocation from cache Kanchan Joshi
[not found] ` <CGME20211220142250epcas5p34b9d93b1dd3388af6209a4223befe40f@epcas5p3.samsung.com>
2021-12-20 14:17 ` [RFC 11/13] nvme: enable bio-cache for fixed-buffer passthru Kanchan Joshi
[not found] ` <CGME20211220142252epcas5p4611297f9970acbc8ee3b0e325ca5ceec@epcas5p4.samsung.com>
2021-12-20 14:17 ` [RFC 12/13] nvme: allow user passthrough commands to poll Kanchan Joshi
[not found] ` <CGME20211220142256epcas5p49e0804ff8b075e8063259f94ccc9ced0@epcas5p4.samsung.com>
2021-12-20 14:17 ` [RFC 13/13] nvme: Add async passthru polling support Kanchan Joshi
2021-12-21 3:45 ` [RFC 00/13] uring-passthru for nvme Jens Axboe
2021-12-21 14:36 ` Kanchan Joshi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox