From: Pavel Begunkov <[email protected]>
To: Jens Axboe <[email protected]>, [email protected]
Subject: [PATCH 02/16] io_uring: optimise io_uring_enter()
Date: Fri, 19 Mar 2021 17:22:30 +0000 [thread overview]
Message-ID: <6c391e07e5c2de877d256a288ff751c8d892a751.1616167719.git.asml.silence@gmail.com> (raw)
In-Reply-To: <[email protected]>
Add unlikely annotations, because my compiler pretty much mispredicts
every first check, and apart jumping around in the fast path, it also
generates extra instructions, like in advance setting ret value.
Signed-off-by: Pavel Begunkov <[email protected]>
---
fs/io_uring.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/fs/io_uring.c b/fs/io_uring.c
index 8ef8809b851f..439f95111b18 100644
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -9138,31 +9138,31 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit,
size_t, argsz)
{
struct io_ring_ctx *ctx;
- long ret = -EBADF;
int submitted = 0;
struct fd f;
+ long ret;
io_run_task_work();
- if (flags & ~(IORING_ENTER_GETEVENTS | IORING_ENTER_SQ_WAKEUP |
- IORING_ENTER_SQ_WAIT | IORING_ENTER_EXT_ARG))
+ if (unlikely(flags & ~(IORING_ENTER_GETEVENTS | IORING_ENTER_SQ_WAKEUP |
+ IORING_ENTER_SQ_WAIT | IORING_ENTER_EXT_ARG)))
return -EINVAL;
f = fdget(fd);
- if (!f.file)
+ if (unlikely(!f.file))
return -EBADF;
ret = -EOPNOTSUPP;
- if (f.file->f_op != &io_uring_fops)
+ if (unlikely(f.file->f_op != &io_uring_fops))
goto out_fput;
ret = -ENXIO;
ctx = f.file->private_data;
- if (!percpu_ref_tryget(&ctx->refs))
+ if (unlikely(!percpu_ref_tryget(&ctx->refs)))
goto out_fput;
ret = -EBADFD;
- if (ctx->flags & IORING_SETUP_R_DISABLED)
+ if (unlikely(ctx->flags & IORING_SETUP_R_DISABLED))
goto out;
/*
--
2.24.0
next prev parent reply other threads:[~2021-03-19 17:27 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-19 17:22 [PATCH 00/16] random 5.13 bits Pavel Begunkov
2021-03-19 17:22 ` [PATCH 01/16] io_uring: don't take ctx refs in task_work handler Pavel Begunkov
2021-03-19 17:22 ` Pavel Begunkov [this message]
2021-03-19 17:22 ` [PATCH 03/16] io_uring: optimise tctx node checks/alloc Pavel Begunkov
2021-03-19 17:22 ` [PATCH 04/16] io_uring: keep io_req_free_batch() call locality Pavel Begunkov
2021-03-19 17:22 ` [PATCH 05/16] io_uring: inline __io_queue_linked_timeout() Pavel Begunkov
2021-03-19 17:22 ` [PATCH 06/16] io_uring: optimise success case of __io_queue_sqe Pavel Begunkov
2021-03-19 17:22 ` [PATCH 07/16] io_uring: refactor io_flush_cached_reqs() Pavel Begunkov
2021-03-19 17:22 ` [PATCH 08/16] io_uring: refactor rsrc refnode allocation Pavel Begunkov
2021-03-19 17:22 ` [PATCH 09/16] io_uring: inline io_put_req and friends Pavel Begunkov
2021-03-19 17:22 ` [PATCH 10/16] io_uring: refactor io_free_req_deferred() Pavel Begunkov
2021-03-19 17:22 ` [PATCH 11/16] io_uring: add helper flushing locked_free_list Pavel Begunkov
2021-03-19 17:22 ` [PATCH 12/16] io_uring: remove __io_req_task_cancel() Pavel Begunkov
2021-03-19 17:22 ` [PATCH 13/16] io_uring: inline io_clean_op()'s fast path Pavel Begunkov
2021-03-19 17:22 ` [PATCH 14/16] io_uring: optimise io_dismantle_req() " Pavel Begunkov
2021-03-19 17:22 ` [PATCH 15/16] io_uring: abolish old io_put_file() Pavel Begunkov
2021-03-19 17:22 ` [PATCH 16/16] io_uring: optimise io_req_task_work_add() Pavel Begunkov
2021-03-19 18:59 ` [PATCH 00/16] random 5.13 bits Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6c391e07e5c2de877d256a288ff751c8d892a751.1616167719.git.asml.silence@gmail.com \
[email protected] \
[email protected] \
[email protected] \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox