From: David Ahern <[email protected]>
To: Jens Axboe <[email protected]>, Stefan Metzmacher <[email protected]>,
[email protected]
Subject: Re: io-uring and tcp sockets
Date: Wed, 4 Nov 2020 08:38:27 -0700 [thread overview]
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
[-- Attachment #1: Type: text/plain, Size: 1904 bytes --]
On 11/4/20 7:50 AM, Jens Axboe wrote:
> On 11/4/20 4:21 AM, Stefan Metzmacher wrote:
>> Hi David,
>>
>>> New to io_uring but can't find this answer online, so reaching out.
>>>
>>> I was trying out io_uring with netperf - tcp stream sockets - and
>>> noticed a submission is called complete even with a partial send
>>> (io_send(), ret < sr->len). Saving the offset of what succeeded (plus
>>> some other adjustments) and retrying the sqe again solves the problem.
>>> But the issue seems fundamental so wondering if is intentional?
>>
>> I guess this is just the way it is currently.
>>
>> For Samba I'd also like to be sure to never get short write to a socket.
>>
>> There I'd like to keep the pipeline full by submitting as much sqe's as possible
>> (without waiting for completions on every single IORING_OP_SENDMSG/IORING_OP_SPLICE)
>> using IOSQE_IO_DRAIN or IOSQE_IO_LINK and maybe IOSQE_ASYNC or IORING_SETUP_SQPOLL.
>>
>> But for now I just used a single sqe with IOSQE_ASYNC at a time.
>>
>> Jens, do you see a way to overcome that limitation?
>>
>> As far as I understand the situation is completely fixed now and
>> it's no possible to get short reads and writes for file io anymore, is that correct?
>
> Right, the regular file IO will not return short reads or writes, unless a
> blocking attempt returns 0 (or short). Which would be expected. The send/recvmsg
> side just returns what the socket read/write would return, similarly to if you
> did the normal system call variants of those calls.
>
> It would not be impossible to make recvmsg/sendmsg handle this internally as
> well, we just need a good way to indicate the intent of "please satisfy the
> whole thing before return".
>
Attached patch handles the full send request; sendmsg can be handled
similarly.
I take your comment to mean there should be an sq flag to opt-in to the
behavior change? Pointers to which flag set?
[-- Attachment #2: 0001-io_uring-Handle-incomplete-sends-for-stream-sockets.patch --]
[-- Type: text/plain, Size: 1846 bytes --]
From 9d6ca280512d3a539c771879d82645a0f7b5a27d Mon Sep 17 00:00:00 2001
From: David Ahern <[email protected]>
Date: Mon, 2 Nov 2020 18:31:00 -0700
Subject: [PATCH] io_uring: Handle incomplete sends for stream sockets
Signed-off-by: David Ahern <[email protected]>
---
fs/io_uring.c | 19 +++++++++++++++++--
1 file changed, 17 insertions(+), 2 deletions(-)
diff --git a/fs/io_uring.c b/fs/io_uring.c
index aae0ef2ec34d..d15511d1e284 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -421,6 +421,7 @@ struct io_sr_msg {
int msg_flags;
int bgid;
size_t len;
+ size_t offset;
struct io_buffer *kbuf;
};
@@ -4149,7 +4150,8 @@ static int io_send(struct io_kiocb *req, bool force_nonblock,
if (unlikely(!sock))
return ret;
- ret = import_single_range(WRITE, sr->buf, sr->len, &iov, &msg.msg_iter);
+ ret = import_single_range(WRITE, sr->buf + sr->offset, sr->len, &iov,
+ &msg.msg_iter);
if (unlikely(ret))
return ret;;
@@ -4171,8 +4173,18 @@ static int io_send(struct io_kiocb *req, bool force_nonblock,
if (ret == -ERESTARTSYS)
ret = -EINTR;
- if (ret < 0)
+ if (ret < 0) {
req_set_fail_links(req);
+ } else if (ret > 0 && sock->type == SOCK_STREAM) {
+ if (unlikely(ret < sr->len)) {
+ pr_debug("req %px sr->offset %lu sr->len %lu ret %d\n",
+ req, sr->offset, sr->len, ret);
+ sr->len -= ret;
+ sr->offset += ret;
+ return -EAGAIN;
+ }
+ ret += sr->offset;
+ }
__io_req_complete(req, ret, 0, cs);
return 0;
}
@@ -6460,6 +6472,9 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req,
/* same numerical values with corresponding REQ_F_*, safe to copy */
req->flags |= sqe_flags;
+ if (req->opcode == IORING_OP_SEND || req->opcode == IORING_OP_SENDMSG)
+ req->sr_msg.offset = 0;
+
if (!io_op_defs[req->opcode].needs_file)
return 0;
--
2.25.1
next prev parent reply other threads:[~2020-11-04 15:38 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-03 1:56 io-uring and tcp sockets David Ahern
2020-11-04 11:21 ` Stefan Metzmacher
2020-11-04 14:50 ` Jens Axboe
2020-11-04 15:38 ` David Ahern [this message]
2020-11-04 17:08 ` Stefan Metzmacher
2020-11-08 23:18 ` David Ahern
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox