From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46BA4C2D0C3 for ; Sat, 21 Dec 2019 20:13:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 15FD5206D8 for ; Sat, 21 Dec 2019 20:13:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NI/MpqL5" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727417AbfLUUNc (ORCPT ); Sat, 21 Dec 2019 15:13:32 -0500 Received: from mail-wm1-f48.google.com ([209.85.128.48]:39364 "EHLO mail-wm1-f48.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727370AbfLUUNb (ORCPT ); Sat, 21 Dec 2019 15:13:31 -0500 Received: by mail-wm1-f48.google.com with SMTP id 20so12378691wmj.4; Sat, 21 Dec 2019 12:13:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=UipHCb83ruP90ECQEg5f6zgvM+zJSXnBrw+F0UKt9U0=; b=NI/MpqL5XTZJ9FFg62mmBeL3cVB0ZvhSCPPT7LkOaYDGXqEOnBNORyh5Y8oQVq+FXH d+s9ONsAurSDiKl3wLjuzmcLRjmd6THHOmqncoKetazwxZSylaOCij+HyhJ6dt64ZfA4 nZ8fWp5IAVOWqbmuEqPtiCKS9NMWDM+4VPqc9THPUck7f29+o8xb6tR5AWcLJRWkLiEF 3YjoO29GXCJtDhqQu7l2+LRW6aVSIEkEiKNTIqGJyjh0c3ajYUUPZH0isRhAnWldUL5K ZHWxnz6jrrA3zR8zaTEfHqSvBM4nF78ixrMWZDqVmjNx3B7Axze6AgdkdAWxZAQmpx3D J+2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UipHCb83ruP90ECQEg5f6zgvM+zJSXnBrw+F0UKt9U0=; b=F89wf+GKfn4px/vE8a5qZh4uhtOGLWjm3w8sKYZEDQn6f7tC8vAlbBUYlNvOAGVMoO 2lA3f3T0PBK1C/YeaqtATOD1V5E5InGBmkJSTDcUpMvyLL5N+DeN+FbjASfKU3EeksZB L/1nsUsC7T6Du4XnltOa+hABzYbSqpYFInYUSVdx+a2jYuxgK6dtJpw/kE+KRmGSJlbc mmUkOF85a6/xY+Kx2pjYZwHNjAv3RZa5QTWJ5AfNwTLpBAo2vjZ5RbZMDYgQZEHRss57 hbQWeoNzNELmbAG5Y6i/HtI608sM8n2QLwFOhhrejSPp9EFA99i9Nc7A2p0nP3bJU7ST yaCQ== X-Gm-Message-State: APjAAAV5OeASPzn6WX7mS57Ac/HN1TxOo5yZdYo3PBXXPxHXZ54vxXt7 tHcwiXJlh6t1zsN5yz92y78= X-Google-Smtp-Source: APXvYqy0OElbCHLIZqVJxJkxd8OHloWK98hEstrYI7xsYK/wmiehUbqT0GYZc+F+Ct3ApRAqgN5viQ== X-Received: by 2002:a1c:a543:: with SMTP id o64mr22203963wme.73.1576959208605; Sat, 21 Dec 2019 12:13:28 -0800 (PST) Received: from localhost.localdomain ([109.126.149.134]) by smtp.gmail.com with ESMTPSA id l7sm14470821wrq.61.2019.12.21.12.13.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Dec 2019 12:13:28 -0800 (PST) From: Pavel Begunkov To: Jens Axboe , io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 2/2] io_uring: batch getting pcpu references Date: Sat, 21 Dec 2019 23:12:54 +0300 Message-Id: X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: io-uring-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org percpu_ref_tryget() has its own overhead. Instead getting a reference for each request, grab a bunch once per io_submit_sqes(). basic benchmark with submit and wait 128 non-linked nops showed ~5% performance gain. (7044 KIOPS vs 7423) Signed-off-by: Pavel Begunkov --- It's just becoming more bulky with ret for me, and would love to hear, hot to make it clearer. This version removes all error handling from hot path, though with goto. fs/io_uring.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 513f1922ce6a..b89a8b975c69 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -1045,9 +1045,6 @@ static struct io_kiocb *io_get_req(struct io_ring_ctx *ctx, gfp_t gfp = GFP_KERNEL | __GFP_NOWARN; struct io_kiocb *req; - if (!percpu_ref_tryget(&ctx->refs)) - return NULL; - if (!state) { req = kmem_cache_alloc(req_cachep, gfp); if (unlikely(!req)) @@ -4400,6 +4397,9 @@ static int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr, return -EBUSY; } + if (!percpu_ref_tryget_many(&ctx->refs, nr)) + return -EAGAIN; + if (nr > IO_PLUG_THRESHOLD) { io_submit_state_start(&state, nr); statep = &state; @@ -4408,16 +4408,22 @@ static int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr, for (i = 0; i < nr; i++) { const struct io_uring_sqe *sqe; struct io_kiocb *req; + unsigned int unused_refs; req = io_get_req(ctx, statep); if (unlikely(!req)) { + unused_refs = nr - submitted; if (!submitted) submitted = -EAGAIN; +put_refs: + percpu_ref_put_many(&ctx->refs, unused_refs); break; } if (!io_get_sqring(ctx, req, &sqe)) { __io_free_req(req); - break; + /* __io_free_req() puts a ref */ + unused_refs = nr - submitted - 1; + goto put_refs; } /* will complete beyond this point, count as submitted */ -- 2.24.0