From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on gnuweeb.org X-Spam-Level: * X-Spam-Status: No, score=1.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_PASS, SPF_SOFTFAIL autolearn=no autolearn_force=no version=3.4.6 Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF927C43334 for ; Thu, 7 Jul 2022 11:53:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235602AbiGGLxj (ORCPT ); Thu, 7 Jul 2022 07:53:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235374AbiGGLww (ORCPT ); Thu, 7 Jul 2022 07:52:52 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D34235725A; Thu, 7 Jul 2022 04:52:18 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id a5so11266915wrx.12; Thu, 07 Jul 2022 04:52:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hdUqBQRL0M8k3/ERJQ8Tpj8kF4Bki9R+wj8zDNk4L4U=; b=CHqpDq7PFolUPTrHXiliVGpHvg84HNgoCSpt3X5917qZvoibQp6mpo9tW0rnC/XfA1 6c8wMwfXI9kago0piOD8B8edJ4/BjGgIH69n1xgc7NuMISMUXFGpXgA3ztgPQlZbRUOn GQ/JcYtlO57iVNqKyTazWMsMNSYBSlH7Oc20sfl7ynZDCDPixpj5DuYvZE7eSok6OnE2 7XdREWZpqCDFN/4k6v3kLCsOZjo4jLky/Ku1UkWKOeEu1mkBao5g12yyW0pQS1jh0kzr 1EXjw2X3vt7sitctPy+enDMeF6Z9tB0tEt4LIyYbvY1WyanMI9cPCR7+a0dfbDWxy05c e6Zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hdUqBQRL0M8k3/ERJQ8Tpj8kF4Bki9R+wj8zDNk4L4U=; b=FekPAN70Vo0gE3AYxoxxoAC9RGpWk65uCtoXQMElItI0qLFsFAQWRv2H34qcjMk/Nl Jjv9AJ4gmgv8EPbCx9C2+ihN8HQp66JtZSR3iMOSHLCxFEjylp80bXD78DGU5uoTzit3 a5i5WKkXrG4L/iyi2Oaucy4XGVjyfH1QJAnc7dPLOl85CavQBaRlv0cLR5IzvXCmlJdO ul5m2UdxO0zJCV3LJAbR66rJ9U75LI3Et0IPLjQJq2CcGt+a5hn/VXZuHVvgfIFc6x0B 3CkD9IIoMHs1QgCyjEvyctZxbb67+rqI/fETc0hmv8sX11WsVZiK2QYdrp1Lsgn2MKnW qx7w== X-Gm-Message-State: AJIora/0ujyNK02X89GQZUpTYs71ythfMSMrH7XjnQRAMwVRbTV4gXbV rnIAyGAQCsP9ghYguzhzGcg+zwsCbKQxggB9fPA= X-Google-Smtp-Source: AGRyM1sDgBrksV2wP21F4DQYtpBSViC4H9NXhwp+0PZA8OMBIpGC6vbzcyfmRD90WXrCPAgd0c4vsg== X-Received: by 2002:adf:d1ea:0:b0:21b:a6cb:fcf6 with SMTP id g10-20020adfd1ea000000b0021ba6cbfcf6mr42348139wrd.477.1657194736776; Thu, 07 Jul 2022 04:52:16 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id u2-20020a5d5142000000b0021b966abc19sm37982131wrt.19.2022.07.07.04.52.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Jul 2022 04:52:16 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Cc: "David S . Miller" , Jakub Kicinski , Jonathan Lemon , Willem de Bruijn , Jens Axboe , David Ahern , kernel-team@fb.com, Pavel Begunkov Subject: [PATCH net-next v4 26/27] io_uring: enable managed frags with register buffers Date: Thu, 7 Jul 2022 12:49:57 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_uring's registered buffers infra has a good performant way of pinning pages, so let's use SKBFL_MANAGED_FRAG_REFS when our requests are purely register buffer backed. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 56 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 55 insertions(+), 1 deletion(-) diff --git a/io_uring/net.c b/io_uring/net.c index bf9916d5e50c..a4e863dce7ec 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -704,6 +704,60 @@ int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return 0; } +static int io_sg_from_iter(struct sock *sk, struct sk_buff *skb, + struct iov_iter *from, size_t length) +{ + struct skb_shared_info *shinfo = skb_shinfo(skb); + int frag = shinfo->nr_frags; + int ret = 0; + struct bvec_iter bi; + ssize_t copied = 0; + unsigned long truesize = 0; + + if (!shinfo->nr_frags) + shinfo->flags |= SKBFL_MANAGED_FRAG_REFS; + + if (!skb_zcopy_managed(skb) || !iov_iter_is_bvec(from)) { + skb_zcopy_downgrade_managed(skb); + return __zerocopy_sg_from_iter(NULL, sk, skb, from, length); + } + + bi.bi_size = min(from->count, length); + bi.bi_bvec_done = from->iov_offset; + bi.bi_idx = 0; + + while (bi.bi_size && frag < MAX_SKB_FRAGS) { + struct bio_vec v = mp_bvec_iter_bvec(from->bvec, bi); + + copied += v.bv_len; + truesize += PAGE_ALIGN(v.bv_len + v.bv_offset); + __skb_fill_page_desc_noacc(shinfo, frag++, v.bv_page, + v.bv_offset, v.bv_len); + bvec_iter_advance_single(from->bvec, &bi, v.bv_len); + } + if (bi.bi_size) + ret = -EMSGSIZE; + + shinfo->nr_frags = frag; + from->bvec += bi.bi_idx; + from->nr_segs -= bi.bi_idx; + from->count = bi.bi_size; + from->iov_offset = bi.bi_bvec_done; + + skb->data_len += copied; + skb->len += copied; + skb->truesize += truesize; + + if (sk && sk->sk_type == SOCK_STREAM) { + sk_wmem_queued_add(sk, truesize); + if (!skb_zcopy_pure(skb)) + sk_mem_charge(sk, truesize); + } else { + refcount_add(truesize, &skb->sk->sk_wmem_alloc); + } + return ret; +} + int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) { struct sockaddr_storage address; @@ -768,7 +822,7 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) msg.msg_flags = msg_flags; msg.msg_ubuf = ¬if->uarg; - msg.sg_from_iter = NULL; + msg.sg_from_iter = io_sg_from_iter; ret = sock_sendmsg(sock, &msg); if (unlikely(ret < min_ret)) { -- 2.36.1