From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 568AB5A4D8; Mon, 18 Mar 2024 22:02:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.48 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799343; cv=none; b=igM3/YbNNev4jRQ2RiTa0Kknc+3+wLMFFcD2ZB7XwKZ3FAo8Mv5kD4gjUbmjr9V2jVDMO2cAosMDlRutMsaUcSSdkQ9Zy+jmxJGMinzn2YHmSt/cF2yxZNiC0sUhi68dsK5MFg/DjnwL8diAMTDZ3LGmbkL3B9q+HusscN1Zat0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710799343; c=relaxed/simple; bh=Ae97tVzbWkRAu0LFf9aZ9RBHFn37wBgTKbXqflC4/ns=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uJV2Bb8rcDZ0SLV/4kTY4qC7dTxj3AEyI8Hy9WYfdUSU9Z/8XnW5gQdD2F7xR3Nrj8rdf+7PGTA/+XUxCb2zkmkpal25VsI15K97aVtJnIbznV1T8S74HIUK5nbSSGx/S0vAt0supXRmoEarPS3onq0wDMgr7jKQouB5SPquqeg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Jq9pgAKe; arc=none smtp.client-ip=209.85.221.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Jq9pgAKe" Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-33ed4d8e9edso2026543f8f.2; Mon, 18 Mar 2024 15:02:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710799339; x=1711404139; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TcROH2LXl8gKrYr6qUKYiPVrUt+epYbzK5tF/Y3MseE=; b=Jq9pgAKem6APyf9gZq2lsuYVZSEeiInBsyDv8XVP5vlVckoxiZzUorTu9/s4r6FOTB S+22suEQBTkSTMp5wSTTjYvZuJizSkyyBJCwGa3BcwPjs7vG/DpmSefaPErwQfllJ3Q/ 2X/RFQmhBPPn9FDW46gj8n34akigfLVhL4p4Pgv51vYdPjogl3ujtk9l4lkppW876Eg4 w6c0rKmaKsyqIfQnWXzKHwFLZJZoS9law9waqhZN+AEETYYrbP+rlzIV6nFRkgc55k4l GAX6FdQo/LF/Hn80YmCyFSijrOpRZw1gwBK1wlpitQcSIjThKz0XJ2+9OHOUzzhSRU++ 4Aog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710799339; x=1711404139; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TcROH2LXl8gKrYr6qUKYiPVrUt+epYbzK5tF/Y3MseE=; b=EvP7oOnMdVd3Hc5HJaSj7IHvm2xYnZQWG4sWb+KwYAD3BbwZTOlUmnLRNyhqIyVhPP llKLexi9EhY+e3L3EZenZPYafiEG8tvWmsOPOQOB97QyDoQqOLSakNwDnWpov6sK1qvD pwmF/mZtOCocXLdV02qcxzdVrJNahGTXG0SQzJbZCMJ2odG9Qa0qedjv8w7cFS6YPGHH +hmo9gTSuWU88yx6FoUlpjbt9vFqDwtv7dN2E2PwykulmDXwWFNTU9btcaSiORS0AOsJ OMUiNG3NgZIHeAPeky/iE9Jl3FyjQIVWLmQq3OyPUrrTEM0bJUsF8/0ulnq4Um8ixp5v 4dMQ== X-Gm-Message-State: AOJu0YxyIaGByqBu1NshC6E7+Y28p0BZFAmq86Wdq9ei7+Ob3hwHczvD oFzk+WXJAjuOp5w/dpac1StrtF73mCf4DBfxNyduwYbvj29fDoCeYUh/kH1Z X-Google-Smtp-Source: AGHT+IGXgwHD5pHiwwmQT18GEQKxv6BNPQAqIYvsLZnQWeRdzzw7fpBEHTCqH4kr+zOOtQ/U8RWLxg== X-Received: by 2002:a5d:5051:0:b0:33e:aea8:6969 with SMTP id h17-20020a5d5051000000b0033eaea86969mr10272351wrt.27.1710799339086; Mon, 18 Mar 2024 15:02:19 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.232.181]) by smtp.gmail.com with ESMTPSA id bj25-20020a0560001e1900b0033e68338fbasm2771038wrb.81.2024.03.18.15.02.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Mar 2024 15:02:17 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: linux-block@vger.kernel.org, Jens Axboe , asml.silence@gmail.com, Kanchan Joshi , Ming Lei Subject: [PATCH v3 07/13] io_uring: force tw ctx locking Date: Mon, 18 Mar 2024 22:00:29 +0000 Message-ID: <6ae858f2ef562e6ed9f13c60978c0d48926954ba.1710799188.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit We can run normal task_work without locking the ctx, however we try to lock anyway and most handlers prefer or require it locked. It might have been interesting to multi-submitter ring with high contention completing async read/write requests via task_work, however that will still need to go through io_req_complete_post() and potentially take the lock for rsrc node putting or some other case. In other words, it's hard to care about it, so alawys force the locking. The case described would also because of various io_uring caches. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index feff8f530c22..66669cc9a675 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1185,8 +1185,9 @@ struct llist_node *io_handle_tw_list(struct llist_node *node, if (req->ctx != ctx) { ctx_flush_and_put(ctx, &ts); ctx = req->ctx; - /* if not contended, grab and improve batching */ - ts.locked = mutex_trylock(&ctx->uring_lock); + + ts.locked = true; + mutex_lock(&ctx->uring_lock); percpu_ref_get(&ctx->refs); } INDIRECT_CALL_2(req->io_task_work.func, @@ -1447,11 +1448,9 @@ static int __io_run_local_work(struct io_ring_ctx *ctx, struct io_tw_state *ts, if (io_run_local_work_continue(ctx, ret, min_events)) goto again; - if (ts->locked) { - io_submit_flush_completions(ctx); - if (io_run_local_work_continue(ctx, ret, min_events)) - goto again; - } + io_submit_flush_completions(ctx); + if (io_run_local_work_continue(ctx, ret, min_events)) + goto again; trace_io_uring_local_work_run(ctx, ret, loops); return ret; @@ -1475,14 +1474,12 @@ static inline int io_run_local_work_locked(struct io_ring_ctx *ctx, static int io_run_local_work(struct io_ring_ctx *ctx, int min_events) { - struct io_tw_state ts = {}; + struct io_tw_state ts = { .locked = true }; int ret; - ts.locked = mutex_trylock(&ctx->uring_lock); + mutex_lock(&ctx->uring_lock); ret = __io_run_local_work(ctx, &ts, min_events); - if (ts.locked) - mutex_unlock(&ctx->uring_lock); - + mutex_unlock(&ctx->uring_lock); return ret; } -- 2.44.0