From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_2 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6526C4338F for ; Fri, 23 Jul 2021 01:41:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B858B60EBC for ; Fri, 23 Jul 2021 01:41:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232892AbhGWBBC (ORCPT ); Thu, 22 Jul 2021 21:01:02 -0400 Received: from mail.kernel.org ([198.145.29.99]:57344 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230318AbhGWBBC (ORCPT ); Thu, 22 Jul 2021 21:01:02 -0400 Received: from rorschach.local.home (cpe-66-24-58-225.stny.res.rr.com [66.24.58.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 11B8060E9A; Fri, 23 Jul 2021 01:41:35 +0000 (UTC) Date: Thu, 22 Jul 2021 21:41:34 -0400 From: Steven Rostedt To: Stefan Metzmacher Cc: Ingo Molnar , linux-trace-devel@vger.kernel.org, io-uring , "linux-kernel@vger.kernel.org" Subject: Re: sched_waking vs. set_event_pid crash (Re: Tracing busy processes/threads freezes/stalls the whole machine) Message-ID: <20210722214134.11bc2a6d@rorschach.local.home> In-Reply-To: <4ebea8f0-58c9-e571-fd30-0ce4f6f09c70@samba.org> References: <293cfb1d-8a53-21e1-83c1-cdb6e2f32c65@samba.org> <20210504092404.6b12aba4@gandalf.local.home> <20210504093550.5719d4bd@gandalf.local.home> <8bb757fb-a83b-0ed5-5247-8273be3925c5@samba.org> <90c806a0-8a2f-1257-7337-6761100217c9@samba.org> <4ebea8f0-58c9-e571-fd30-0ce4f6f09c70@samba.org> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org On Fri, 23 Jul 2021 00:43:13 +0200 Stefan Metzmacher wrote: > Hi Steve, Hi Stefan, > > After some days of training: > https://training.linuxfoundation.org/training/linux-kernel-debugging-and-security/ > I was able to get much closer to the problem :-) > > In order to reproduce it and get reliable kexec crash dumps, > I needed to give the VM at least 3 cores. > > While running './io-uring_cp-forever link-cp.c file' (from: > https://github.com/metze-samba/liburing/commits/io_uring-cp-forever ) > in one window, the following simple sequence triggered the problem in most cases: > > echo 1 > /sys/kernel/tracing/events/sched/sched_waking/enable > echo 1 > /sys/kernel/tracing/set_event_pid I was able to reproduce it with running hackbench in a while loop and in another terminal, executing the above two lines. I think I found the bug. Can you test this patch? Thanks, -- Steve diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c index 976bf8ce8039..fc32821f8240 100644 --- a/kernel/tracepoint.c +++ b/kernel/tracepoint.c @@ -299,8 +299,8 @@ static int tracepoint_add_func(struct tracepoint *tp, * a pointer to it. This array is referenced by __DO_TRACE from * include/linux/tracepoint.h using rcu_dereference_sched(). */ - rcu_assign_pointer(tp->funcs, tp_funcs); tracepoint_update_call(tp, tp_funcs, false); + rcu_assign_pointer(tp->funcs, tp_funcs); static_key_enable(&tp->key); release_probes(old);