From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2042DC49EA2 for ; Fri, 18 Jun 2021 23:32:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0379560FE9 for ; Fri, 18 Jun 2021 23:32:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234637AbhFRXev (ORCPT ); Fri, 18 Jun 2021 19:34:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55246 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234509AbhFRXeu (ORCPT ); Fri, 18 Jun 2021 19:34:50 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA810C061574 for ; Fri, 18 Jun 2021 16:32:40 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id u5-20020a7bc0450000b02901480e40338bso7263697wmc.1 for ; Fri, 18 Jun 2021 16:32:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=cn0g3QO5TGXbEX7L1hdWWqIJra1pf0Ee731+XCYnLGI=; b=cF3MrTdK/tvkcmmEi0Cz0vmJuSp/TNGSqaKqSTer7vP0kkGYbooH2r4m2U8g0ZDI5O KgoYcgPmH5NTVrLMUZs/9n1ydiruZcWwpwafmhW1xSKrmE5RwnFPUE1gskqLWIB23/H6 4KwTOaHi2BpBVRJW4hgpbmjnTVMXcnC55DgrKthVf6Q30/z7tCKVTDm3ryWehtqKnpe5 Fl3h/9rw0m7W1MefZ3rVE6wxruKutbiT2Ge/5yKPKxTgC1ReVD3CVey/nI8E5blR5JyL 6mnyjPJpRlaOsoMJJ3T2MOMDWLMhf37MnsVBt47H8NHd6p3eLLURNpbdq6QHcRVLrsiW Gtjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cn0g3QO5TGXbEX7L1hdWWqIJra1pf0Ee731+XCYnLGI=; b=WIi4ol81cMDHUQqor1DtnY87ldH8y4aixQW+J+ZclMyATwo4cOVmI9dZ/xWtnz/4Sv OahwATECLQyEymCjZ6CZzS46ay9PEfRH5DxROuGWk0y+4xUcCGCfQfICHCxbH6y4L2b5 ZkKRyIWiJ+EoU1tvnxvOMMTeACNHTPB/grvUr3KRYs4Na4sIafE1QTUEURRlWDbSbBLW MNYpTF8hXTYT+TzVmouaqJXkDPIJ8R2cCsghcloheTYWr1MjfSrzrknrGdLAtdqEAva7 UZlF3Xm5xp9jWalqLuMWwilb2Gqx6wjACXacuDcrhqH3nUI4SDa4VHQG9kym9wdyWM88 yh2w== X-Gm-Message-State: AOAM5322xq6OXklUMczBU1y1mzi1wJ9JvOj415wKVp3vu01gaL8RsMSm zWyIfzAfy3Mfjtuh34C1cKibZBpJ1KZ86g== X-Google-Smtp-Source: ABdhPJzwP1aJrbOjpehn4RtAW2XuyTPLna1gvD5Zs6OebaAZw6jirLKx/vPbnS/6nVOI6SUiNt2iww== X-Received: by 2002:a05:600c:88a:: with SMTP id l10mr13927385wmp.7.1624059159365; Fri, 18 Jun 2021 16:32:39 -0700 (PDT) Received: from localhost.localdomain ([185.69.145.72]) by smtp.gmail.com with ESMTPSA id o20sm11765774wms.3.2021.06.18.16.32.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Jun 2021 16:32:39 -0700 (PDT) From: Pavel Begunkov To: Jens Axboe , io-uring@vger.kernel.org Subject: [PATCH 2/2] io_uring: inline fixed part of io_file_get() Date: Sat, 19 Jun 2021 00:32:17 +0100 Message-Id: <21325f0526f63649c85301fdf960d870db09b1c9.1624058853.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Optimise io_file_get() with registered files, which is in a hot path, by inlining parts of the function. Saves a function call, and inefficiencies of passing arguments, e.g. evaluating (sqe_flags & IOSQE_FIXED_FILE). Note that there was enough of instructions generated by two-level table dereference to impede inlining, but it's now slim enough. Signed-off-by: Pavel Begunkov --- fs/io_uring.c | 65 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 39 insertions(+), 26 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 2fd54a21ed8b..b966c65da23a 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -1053,7 +1053,8 @@ static int __io_register_rsrc_update(struct io_ring_ctx *ctx, unsigned type, struct io_uring_rsrc_update2 *up, unsigned nr_args); static void io_clean_op(struct io_kiocb *req); -static struct file *io_file_get(struct io_submit_state *state, +static struct file *io_file_get(struct io_ring_ctx *ctx, + struct io_submit_state *state, struct io_kiocb *req, int fd, bool fixed); static void __io_queue_sqe(struct io_kiocb *req); static void io_rsrc_put_work(struct work_struct *work); @@ -3602,7 +3603,8 @@ static int __io_splice_prep(struct io_kiocb *req, if (unlikely(sp->flags & ~valid_flags)) return -EINVAL; - sp->file_in = io_file_get(NULL, req, READ_ONCE(sqe->splice_fd_in), + sp->file_in = io_file_get(req->ctx, NULL, req, + READ_ONCE(sqe->splice_fd_in), (sp->flags & SPLICE_F_FD_IN_FIXED)); if (!sp->file_in) return -EBADF; @@ -6340,36 +6342,48 @@ static void io_fixed_file_set(struct io_fixed_file *file_slot, struct file *file file_slot->file_ptr = file_ptr; } -static struct file *io_file_get(struct io_submit_state *state, - struct io_kiocb *req, int fd, bool fixed) +static inline struct file *io_file_get_fixed(struct io_ring_ctx *ctx, + struct io_kiocb *req, int fd) { - struct io_ring_ctx *ctx = req->ctx; struct file *file; + unsigned long file_ptr; - if (fixed) { - unsigned long file_ptr; + if (unlikely((unsigned int)fd >= ctx->nr_user_files)) + return NULL; + fd = array_index_nospec(fd, ctx->nr_user_files); + file_ptr = io_fixed_file_slot(&ctx->file_table, fd)->file_ptr; + file = (struct file *) (file_ptr & FFS_MASK); + file_ptr &= ~FFS_MASK; + /* mask in overlapping REQ_F and FFS bits */ + req->flags |= (file_ptr << REQ_F_ASYNC_READ_BIT); + io_req_set_rsrc_node(req); + return file; +} - if (unlikely((unsigned int)fd >= ctx->nr_user_files)) - return NULL; - fd = array_index_nospec(fd, ctx->nr_user_files); - file_ptr = io_fixed_file_slot(&ctx->file_table, fd)->file_ptr; - file = (struct file *) (file_ptr & FFS_MASK); - file_ptr &= ~FFS_MASK; - /* mask in overlapping REQ_F and FFS bits */ - req->flags |= (file_ptr << REQ_F_ASYNC_READ_BIT); - io_req_set_rsrc_node(req); - } else { - trace_io_uring_file_get(ctx, fd); - file = __io_file_get(state, fd); +static struct file *io_file_get_normal(struct io_ring_ctx *ctx, + struct io_submit_state *state, + struct io_kiocb *req, int fd) +{ + struct file *file = __io_file_get(state, fd); - /* we don't allow fixed io_uring files */ - if (file && unlikely(file->f_op == &io_uring_fops)) - io_req_track_inflight(req); - } + trace_io_uring_file_get(ctx, fd); + /* we don't allow fixed io_uring files */ + if (file && unlikely(file->f_op == &io_uring_fops)) + io_req_track_inflight(req); return file; } +static inline struct file *io_file_get(struct io_ring_ctx *ctx, + struct io_submit_state *state, + struct io_kiocb *req, int fd, bool fixed) +{ + if (fixed) + return io_file_get_fixed(ctx, req, fd); + else + return io_file_get_normal(ctx, state, req, fd); +} + static enum hrtimer_restart io_link_timeout_fn(struct hrtimer *timer) { struct io_timeout_data *data = container_of(timer, @@ -6561,9 +6575,8 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, } if (io_op_defs[req->opcode].needs_file) { - bool fixed = req->flags & REQ_F_FIXED_FILE; - - req->file = io_file_get(state, req, READ_ONCE(sqe->fd), fixed); + req->file = io_file_get(ctx, state, req, READ_ONCE(sqe->fd), + (sqe_flags & IOSQE_FIXED_FILE)); if (unlikely(!req->file)) ret = -EBADF; } -- 2.31.1