From: Jens Axboe <[email protected]>
To: [email protected]
Cc: Jens Axboe <[email protected]>
Subject: [PATCH 6/8] io_uring: terminate multishot poll for CQ ring overflow
Date: Fri, 19 Mar 2021 14:35:14 -0600 [thread overview]
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
If we hit overflow and fail to allocate an overflow entry for the
completion, terminate the multishot poll mode.
Signed-off-by: Jens Axboe <[email protected]>
---
fs/io_uring.c | 20 ++++++++++++--------
1 file changed, 12 insertions(+), 8 deletions(-)
diff --git a/fs/io_uring.c b/fs/io_uring.c
index 53378003bd3b..103daef0db34 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -1044,6 +1044,7 @@ static void io_rsrc_put_work(struct work_struct *work);
static void io_req_task_queue(struct io_kiocb *req);
static void io_submit_flush_completions(struct io_comp_state *cs,
struct io_ring_ctx *ctx);
+static bool io_poll_remove_waitqs(struct io_kiocb *req);
static int io_req_prep_async(struct io_kiocb *req);
static struct kmem_cache *req_cachep;
@@ -1513,7 +1514,7 @@ static inline void req_ref_get(struct io_kiocb *req)
atomic_inc(&req->refs);
}
-static void __io_cqring_fill_event(struct io_kiocb *req, long res,
+static bool __io_cqring_fill_event(struct io_kiocb *req, long res,
unsigned int cflags)
{
struct io_ring_ctx *ctx = req->ctx;
@@ -1531,7 +1532,7 @@ static void __io_cqring_fill_event(struct io_kiocb *req, long res,
WRITE_ONCE(cqe->user_data, req->user_data);
WRITE_ONCE(cqe->res, res);
WRITE_ONCE(cqe->flags, cflags);
- return;
+ return true;
}
if (!ctx->cq_overflow_flushed &&
!atomic_read(&req->task->io_uring->in_idle)) {
@@ -1549,7 +1550,7 @@ static void __io_cqring_fill_event(struct io_kiocb *req, long res,
ocqe->cqe.res = res;
ocqe->cqe.flags = cflags;
list_add_tail(&ocqe->list, &ctx->cq_overflow_list);
- return;
+ return true;
}
overflow:
/*
@@ -1558,6 +1559,7 @@ static void __io_cqring_fill_event(struct io_kiocb *req, long res,
* on the floor.
*/
WRITE_ONCE(ctx->rings->cq_overflow, ++ctx->cached_cq_overflow);
+ return false;
}
static void io_cqring_fill_event(struct io_kiocb *req, long res)
@@ -4905,14 +4907,14 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask, int error)
error = -ECANCELED;
req->poll.events |= EPOLLONESHOT;
}
- if (error || (req->poll.events & EPOLLONESHOT)) {
- io_poll_remove_double(req);
+ if (!error)
+ error = mangle_poll(mask);
+ if (!__io_cqring_fill_event(req, error, flags) ||
+ (req->poll.events & EPOLLONESHOT)) {
+ io_poll_remove_waitqs(req);
req->poll.done = true;
flags = 0;
}
- if (!error)
- error = mangle_poll(mask);
- __io_cqring_fill_event(req, error, flags);
io_commit_cqring(ctx);
return !(flags & IORING_CQE_F_MORE);
}
@@ -5203,6 +5205,8 @@ static bool __io_poll_remove_one(struct io_kiocb *req,
{
bool do_complete = false;
+ if (!poll->head)
+ return false;
spin_lock(&poll->head->lock);
WRITE_ONCE(poll->canceled, true);
if (!list_empty(&poll->wait.entry)) {
--
2.31.0
next prev parent reply other threads:[~2021-03-19 20:36 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-19 20:35 [PATCHSET v2 0/8] Poll improvements Jens Axboe
2021-03-19 20:35 ` [PATCH 1/8] io_uring: mask in error/nval/hangup consistently for poll Jens Axboe
2021-03-19 20:35 ` [PATCH 2/8] io_uring: allocate memory for overflowed CQEs Jens Axboe
2021-03-19 20:35 ` [PATCH 3/8] io_uring: include cflags in completion trace event Jens Axboe
2021-03-19 20:35 ` [PATCH 4/8] io_uring: add multishot mode for IORING_OP_POLL_ADD Jens Axboe
2021-03-19 20:35 ` [PATCH 5/8] io_uring: abstract out helper for removing poll waitqs/hashes Jens Axboe
2021-03-19 20:35 ` Jens Axboe [this message]
2021-03-19 20:35 ` [PATCH 7/8] io_uring: abstract out a io_poll_find_helper() Jens Axboe
2021-03-19 20:35 ` [PATCH 8/8] io_uring: allow events and user_data update of running poll requests Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
[email protected] \
[email protected] \
[email protected] \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox