From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 245D7C19F2D for ; Tue, 9 Aug 2022 06:46:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235582AbiHIGqT (ORCPT ); Tue, 9 Aug 2022 02:46:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233045AbiHIGqS (ORCPT ); Tue, 9 Aug 2022 02:46:18 -0400 Received: from verein.lst.de (verein.lst.de [213.95.11.211]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F21DC1929E; Mon, 8 Aug 2022 23:46:17 -0700 (PDT) Received: by verein.lst.de (Postfix, from userid 2407) id BCD3D68AA6; Tue, 9 Aug 2022 08:46:13 +0200 (CEST) Date: Tue, 9 Aug 2022 08:46:13 +0200 From: Christoph Hellwig To: Keith Busch Cc: linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, io-uring@vger.kernel.org, linux-fsdevel@vger.kernel.org, axboe@kernel.dk, hch@lst.de, Alexander Viro , Kernel Team , Keith Busch Subject: Re: [PATCHv3 0/7] dma mapping optimisations Message-ID: <20220809064613.GA9040@lst.de> References: <20220805162444.3985535-1-kbusch@fb.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220805162444.3985535-1-kbusch@fb.com> User-Agent: Mutt/1.5.17 (2007-11-01) Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org I finally found some time to look over this, and I have some very high level problems with the implementations: - the DMA streaming ops (dma_map_*) are only intended for short term mappings, not for long living ones that can lead to starvation. Especially with swiotlb, but also with some IOMMUs. I think this needs to be changed to an API that can allocate DMA mapped memory using dma_alloc_noncoherent/dma_alloc_noncontigious for the device and then give access to that to the user instead - I really do not like the special casing in the bio. Did you try to just stash away the DMA mapping information in an efficient lookup data structure (e.g. rthashtable, but details might need analysis and benchmarking) and thus touch very little code outside of the driver I/O path and the place that performs the mapping? - the design seems to ignore DMA ownership. Every time data in transfered data needs to be transferred to and from the device, take a look at Documentation/core-api/dma-api.rst and Documentation/core-api/dma-api-howto.rst. As for the multiple devices discussion: mapping memory for multiple devices is possible, but nontrivial to get right, mostly due to the ownership. So unless we have a really good reason I'd suggest to not support this initially.