From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on gnuweeb.org X-Spam-Level: X-Spam-Status: No, score=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 Received: from out02.mta.xmission.com (out02.mta.xmission.com [166.70.13.232]) by gnuweeb.org (Postfix) with ESMTPS id 2C7417F88C for ; Mon, 6 Jun 2022 15:19:51 +0000 (UTC) Received: from in01.mta.xmission.com ([166.70.13.51]:47270) by out02.mta.xmission.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1nyEWC-0048zM-0H; Mon, 06 Jun 2022 09:19:48 -0600 Received: from ip68-227-174-4.om.om.cox.net ([68.227.174.4]:34008 helo=email.froward.int.ebiederm.org.xmission.com) by in01.mta.xmission.com with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1nyEWA-007Py3-L2; Mon, 06 Jun 2022 09:19:47 -0600 From: "Eric W. Biederman" To: Linus Torvalds Cc: Ammar Faizi , John Johansen , James Morris , LSM List , Linux Kernel Mailing List , Al Viro , Kees Cook , , Linux-MM , gwml@vger.gnuweeb.org References: <226cee6a-6ca1-b603-db08-8500cd8f77b7@gnuweeb.org> Date: Mon, 06 Jun 2022 10:19:40 -0500 In-Reply-To: (Linus Torvalds's message of "Wed, 27 Apr 2022 11:31:54 -0700") Message-ID: <87r1414y5v.fsf@email.froward.int.ebiederm.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-XM-SPF: eid=1nyEWA-007Py3-L2;;;mid=<87r1414y5v.fsf@email.froward.int.ebiederm.org>;;;hst=in01.mta.xmission.com;;;ip=68.227.174.4;;;frm=ebiederm@xmission.com;;;spf=softfail X-XM-AID: U2FsdGVkX1+zbjDt+uq0sjGhGkekjhmgxAfVd6XvNGw= X-SA-Exim-Connect-IP: 68.227.174.4 X-SA-Exim-Mail-From: ebiederm@xmission.com Subject: Re: Linux 5.18-rc4 X-SA-Exim-Version: 4.2.1 (built Sat, 08 Feb 2020 21:53:50 +0000) X-SA-Exim-Scanned: Yes (on in01.mta.xmission.com) List-Id: Linus Torvalds writes: > This looks like it might be AppArmor-related. > > Adding AppArmor and security module people to the participants. > > Sorry for top-posting and quoting the whole thing, but this is really > just bringing in more people to the discussion. > > So on the exec path we have > > apparmor_bprm_committing_creds() -> > aa_inherit_files() -> > iterate_fd (takes files->file_lock) -> > aa_file_perm -> > update_file_ctx (takes aa_file_ctx->lock) > > which gives us that file_lock -> ctx lock order. All within AppArmor. > > And then we apparently _also_ have the reverse ctx lock -> file_lock > order by way of 'alloc_lock', which is the 'task_lock()' thing > > That one is a horror to decode and I didn't, but seems to go through > ipcget -> newseg.. > > Anybody? Has anyone looked into this lock ordering issues? Eric > > On Wed, Apr 27, 2022 at 11:00 AM Ammar Faizi wrote: >> >> On 4/25/22 5:22 AM, Linus Torvalds wrote: >> > Fairly slow and calm week - which makes me just suspect that the other >> > shoe will drop at some point. >> > >> > But maybe things are just going really well this release. It's bound >> > to happen _occasionally_, after all. >> >> + fs/exec.c maintainers. >> >> Testing Linux 5.18-rc4 on my laptop, it has been running for 2 days. Got >> the following lockdep splat this night. I don't have the reproducer. If >> you need more information, feel free to let me know. >> >> [78140.503644] ====================================================== >> [78140.503646] WARNING: possible circular locking dependency detected >> [78140.503648] 5.18.0-rc4-superb-owl-00006-gd615b5416f8a #12 Tainted: G W >> [78140.503650] ------------------------------------------------------ >> [78140.503651] preconv/111629 is trying to acquire lock: >> [78140.503653] ffff88834d633248 (&ctx->lock){+.+.}-{2:2}, at: update_file_ctx+0x19/0xe0 >> [78140.503663] >> but task is already holding lock: >> [78140.503664] ffff888103d80458 (&newf->file_lock){+.+.}-{2:2}, at: iterate_fd+0x34/0x150 >> [78140.503669] >> which lock already depends on the new lock. >> >> [78140.503671] >> the existing dependency chain (in reverse order) is: >> [78140.503672] >> -> #4 (&newf->file_lock){+.+.}-{2:2}: >> [78140.503675] _raw_spin_lock+0x2f/0x40 >> [78140.503679] seq_show+0x72/0x280 >> [78140.503681] seq_read_iter+0x125/0x3c0 >> [78140.503684] seq_read+0xd0/0xe0 >> [78140.503686] vfs_read+0xf5/0x2f0 >> [78140.503688] ksys_read+0x58/0xb0 >> [78140.503690] do_syscall_64+0x3d/0x90 >> [78140.503693] entry_SYSCALL_64_after_hwframe+0x44/0xae >> [78140.503695] >> -> #3 (&p->alloc_lock){+.+.}-{2:2}: >> [78140.503699] _raw_spin_lock+0x2f/0x40 >> [78140.503700] newseg+0x25b/0x360 >> [78140.503703] ipcget+0x3fb/0x480 >> [78140.503705] __x64_sys_shmget+0x48/0x50 >> [78140.503708] do_syscall_64+0x3d/0x90 >> [78140.503710] entry_SYSCALL_64_after_hwframe+0x44/0xae >> [78140.503713] >> -> #2 (&new->lock){+.+.}-{2:2}: >> [78140.503716] _raw_spin_lock+0x2f/0x40 >> [78140.503718] ipc_addid+0xb3/0x700 >> [78140.503720] newseg+0x238/0x360 >> [78140.503722] ipcget+0x3fb/0x480 >> [78140.503724] __x64_sys_shmget+0x48/0x50 >> [78140.503727] do_syscall_64+0x3d/0x90 >> [78140.503729] entry_SYSCALL_64_after_hwframe+0x44/0xae >> [78140.503731] >> -> #1 (lock#3){+.+.}-{2:2}: >> [78140.503735] local_lock_acquire+0x1d/0x70 >> [78140.503738] __radix_tree_preload+0x38/0x150 >> [78140.503740] idr_preload+0xa/0x40 >> [78140.503743] aa_alloc_secid+0x15/0xb0 >> [78140.503745] aa_label_alloc+0x6c/0x1b0 >> [78140.503747] aa_label_merge+0x52/0x430 >> [78140.503750] update_file_ctx+0x3f/0xe0 >> [78140.503752] aa_file_perm+0x56e/0x5c0 >> [78140.503754] common_file_perm+0x70/0xd0 >> [78140.503756] security_mmap_file+0x4b/0xd0 >> [78140.503759] vm_mmap_pgoff+0x50/0x150 >> [78140.503761] elf_map+0x9f/0x120 >> [78140.503763] load_elf_binary+0x521/0xc80 >> [78140.503767] bprm_execve+0x39f/0x660 >> [78140.503769] do_execveat_common+0x1d0/0x220 >> [78140.503771] __x64_sys_execveat+0x3d/0x50 >> [78140.503773] do_syscall_64+0x3d/0x90 >> [78140.503775] entry_SYSCALL_64_after_hwframe+0x44/0xae >> [78140.503777] >> -> #0 (&ctx->lock){+.+.}-{2:2}: >> [78140.503780] __lock_acquire+0x1573/0x2ce0 >> [78140.503783] lock_acquire+0xbd/0x190 >> [78140.503785] _raw_spin_lock+0x2f/0x40 >> [78140.503787] update_file_ctx+0x19/0xe0 >> [78140.503788] aa_file_perm+0x56e/0x5c0 >> [78140.503790] match_file+0x78/0x90 >> [78140.503792] iterate_fd+0xae/0x150 >> [78140.503794] aa_inherit_files+0xbe/0x170 >> [78140.503796] apparmor_bprm_committing_creds+0x50/0x80 >> [78140.503798] security_bprm_committing_creds+0x1d/0x30 >> [78140.503800] begin_new_exec+0x3c5/0x450 >> [78140.503802] load_elf_binary+0x269/0xc80 >> [78140.503804] bprm_execve+0x39f/0x660 >> [78140.503806] do_execveat_common+0x1d0/0x220 >> [78140.503808] __x64_sys_execve+0x36/0x40 >> [78140.503809] do_syscall_64+0x3d/0x90 >> [78140.503812] entry_SYSCALL_64_after_hwframe+0x44/0xae >> [78140.503815] >> other info that might help us debug this: >> >> [78140.503816] Chain exists of: >> &ctx->lock --> &p->alloc_lock --> &newf->file_lock >> >> [78140.503820] Possible unsafe locking scenario: >> >> [78140.503821] CPU0 CPU1 >> [78140.503823] ---- ---- >> [78140.503824] lock(&newf->file_lock); >> [78140.503826] lock(&p->alloc_lock); >> [78140.503828] lock(&newf->file_lock); >> [78140.503830] lock(&ctx->lock); >> [78140.503832] >> *** DEADLOCK *** >> >> [78140.503833] 3 locks held by preconv/111629: >> [78140.503835] #0: ffff888111b62550 (&sig->cred_guard_mutex){+.+.}-{3:3}, at: bprm_execve+0x39/0x660 >> [78140.503840] #1: ffff888111b625e8 (&sig->exec_update_lock){++++}-{3:3}, at: exec_mmap+0x4e/0x250 >> [78140.503844] #2: ffff888103d80458 (&newf->file_lock){+.+.}-{2:2}, at: iterate_fd+0x34/0x150 >> [78140.503849] >> stack backtrace: >> [78140.503851] CPU: 3 PID: 111629 Comm: preconv Tainted: G W 5.18.0-rc4-superb-owl-00006-gd615b5416f8a #12 6fd282a37da6f0e0172ecfa29689f3d250476a2b >> [78140.503855] Hardware name: HP HP Laptop 14s-dq2xxx/87FD, BIOS F.15 09/15/2021 >> [78140.503856] Call Trace: >> [78140.503858] >> [78140.503860] dump_stack_lvl+0x5a/0x74 >> [78140.503863] check_noncircular+0xd3/0xe0 >> [78140.503866] ? register_lock_class+0x35/0x2a0 >> [78140.503870] __lock_acquire+0x1573/0x2ce0 >> [78140.503872] ? prepend_path+0x375/0x410 >> [78140.503876] ? d_absolute_path+0x48/0x80 >> [78140.503879] ? aa_path_name+0x132/0x470 >> [78140.503883] ? lock_is_held_type+0xd0/0x130 >> [78140.503886] lock_acquire+0xbd/0x190 >> [78140.503888] ? update_file_ctx+0x19/0xe0 >> [78140.503892] _raw_spin_lock+0x2f/0x40 >> [78140.503894] ? update_file_ctx+0x19/0xe0 >> [78140.503896] update_file_ctx+0x19/0xe0 >> [78140.503899] aa_file_perm+0x56e/0x5c0 >> [78140.503904] ? aa_inherit_files+0x170/0x170 >> [78140.503906] match_file+0x78/0x90 >> [78140.503909] iterate_fd+0xae/0x150 >> [78140.503912] aa_inherit_files+0xbe/0x170 >> [78140.503915] apparmor_bprm_committing_creds+0x50/0x80 >> [78140.503918] security_bprm_committing_creds+0x1d/0x30 >> [78140.503921] begin_new_exec+0x3c5/0x450 >> [78140.503924] load_elf_binary+0x269/0xc80 >> [78140.503928] ? lock_release+0x1ee/0x260 >> [78140.503930] ? bprm_execve+0x399/0x660 >> [78140.503933] bprm_execve+0x39f/0x660 >> [78140.503936] do_execveat_common+0x1d0/0x220 >> [78140.503940] __x64_sys_execve+0x36/0x40 >> [78140.503942] do_syscall_64+0x3d/0x90 >> [78140.503946] entry_SYSCALL_64_after_hwframe+0x44/0xae >> [78140.503948] RIP: 0033:0x7f700a8ea33b >> [78140.503954] Code: Unable to access opcode bytes at RIP 0x7f700a8ea311. >> [78140.503955] RSP: 002b:00007fff315e7db8 EFLAGS: 00000246 ORIG_RAX: 000000000000003b >> [78140.503958] RAX: ffffffffffffffda RBX: 00007fff315e7dc0 RCX: 00007f700a8ea33b >> [78140.503960] RDX: 000056419e9ea7e0 RSI: 000056419e9e9160 RDI: 00007fff315e7dc0 >> [78140.503962] RBP: 00007fff315e7f60 R08: 0000000000000008 R09: 0000000000000000 >> [78140.503964] R10: 0000000000000001 R11: 0000000000000246 R12: 000056419e9ea760 >> [78140.503965] R13: 000056419e9e9160 R14: 00007fff315e9eb4 R15: 00007fff315e9ebc >> [78140.503971] >> >> -- >> Ammar Faizi