From: "Eric W. Biederman" <[email protected]>
To: Linus Torvalds <[email protected]>
Cc: Ammar Faizi <[email protected]>,
John Johansen <[email protected]>,
James Morris <[email protected]>,
LSM List <[email protected]>,
Linux Kernel Mailing List <[email protected]>,
Al Viro <[email protected]>,
Kees Cook <[email protected]>,
<[email protected]> <[email protected]>,
Linux-MM <[email protected]>,
[email protected]
Subject: Re: Linux 5.18-rc4
Date: Mon, 06 Jun 2022 10:19:40 -0500 [thread overview]
Message-ID: <[email protected]> (raw)
In-Reply-To: <CAHk-=whayT+o58FrPCXVVJ3Bn-3SeoDkMA77TOd9jg4yMGNExw@mail.gmail.com> (Linus Torvalds's message of "Wed, 27 Apr 2022 11:31:54 -0700")
Linus Torvalds <[email protected]> writes:
> This looks like it might be AppArmor-related.
>
> Adding AppArmor and security module people to the participants.
>
> Sorry for top-posting and quoting the whole thing, but this is really
> just bringing in more people to the discussion.
>
> So on the exec path we have
>
> apparmor_bprm_committing_creds() ->
> aa_inherit_files() ->
> iterate_fd (takes files->file_lock) ->
> aa_file_perm ->
> update_file_ctx (takes aa_file_ctx->lock)
>
> which gives us that file_lock -> ctx lock order. All within AppArmor.
>
> And then we apparently _also_ have the reverse ctx lock -> file_lock
> order by way of 'alloc_lock', which is the 'task_lock()' thing
>
> That one is a horror to decode and I didn't, but seems to go through
> ipcget -> newseg..
>
> Anybody?
Has anyone looked into this lock ordering issues?
Eric
>
> On Wed, Apr 27, 2022 at 11:00 AM Ammar Faizi <[email protected]> wrote:
>>
>> On 4/25/22 5:22 AM, Linus Torvalds wrote:
>> > Fairly slow and calm week - which makes me just suspect that the other
>> > shoe will drop at some point.
>> >
>> > But maybe things are just going really well this release. It's bound
>> > to happen _occasionally_, after all.
>>
>> + fs/exec.c maintainers.
>>
>> Testing Linux 5.18-rc4 on my laptop, it has been running for 2 days. Got
>> the following lockdep splat this night. I don't have the reproducer. If
>> you need more information, feel free to let me know.
>>
>> [78140.503644] ======================================================
>> [78140.503646] WARNING: possible circular locking dependency detected
>> [78140.503648] 5.18.0-rc4-superb-owl-00006-gd615b5416f8a #12 Tainted: G W
>> [78140.503650] ------------------------------------------------------
>> [78140.503651] preconv/111629 is trying to acquire lock:
>> [78140.503653] ffff88834d633248 (&ctx->lock){+.+.}-{2:2}, at: update_file_ctx+0x19/0xe0
>> [78140.503663]
>> but task is already holding lock:
>> [78140.503664] ffff888103d80458 (&newf->file_lock){+.+.}-{2:2}, at: iterate_fd+0x34/0x150
>> [78140.503669]
>> which lock already depends on the new lock.
>>
>> [78140.503671]
>> the existing dependency chain (in reverse order) is:
>> [78140.503672]
>> -> #4 (&newf->file_lock){+.+.}-{2:2}:
>> [78140.503675] _raw_spin_lock+0x2f/0x40
>> [78140.503679] seq_show+0x72/0x280
>> [78140.503681] seq_read_iter+0x125/0x3c0
>> [78140.503684] seq_read+0xd0/0xe0
>> [78140.503686] vfs_read+0xf5/0x2f0
>> [78140.503688] ksys_read+0x58/0xb0
>> [78140.503690] do_syscall_64+0x3d/0x90
>> [78140.503693] entry_SYSCALL_64_after_hwframe+0x44/0xae
>> [78140.503695]
>> -> #3 (&p->alloc_lock){+.+.}-{2:2}:
>> [78140.503699] _raw_spin_lock+0x2f/0x40
>> [78140.503700] newseg+0x25b/0x360
>> [78140.503703] ipcget+0x3fb/0x480
>> [78140.503705] __x64_sys_shmget+0x48/0x50
>> [78140.503708] do_syscall_64+0x3d/0x90
>> [78140.503710] entry_SYSCALL_64_after_hwframe+0x44/0xae
>> [78140.503713]
>> -> #2 (&new->lock){+.+.}-{2:2}:
>> [78140.503716] _raw_spin_lock+0x2f/0x40
>> [78140.503718] ipc_addid+0xb3/0x700
>> [78140.503720] newseg+0x238/0x360
>> [78140.503722] ipcget+0x3fb/0x480
>> [78140.503724] __x64_sys_shmget+0x48/0x50
>> [78140.503727] do_syscall_64+0x3d/0x90
>> [78140.503729] entry_SYSCALL_64_after_hwframe+0x44/0xae
>> [78140.503731]
>> -> #1 (lock#3){+.+.}-{2:2}:
>> [78140.503735] local_lock_acquire+0x1d/0x70
>> [78140.503738] __radix_tree_preload+0x38/0x150
>> [78140.503740] idr_preload+0xa/0x40
>> [78140.503743] aa_alloc_secid+0x15/0xb0
>> [78140.503745] aa_label_alloc+0x6c/0x1b0
>> [78140.503747] aa_label_merge+0x52/0x430
>> [78140.503750] update_file_ctx+0x3f/0xe0
>> [78140.503752] aa_file_perm+0x56e/0x5c0
>> [78140.503754] common_file_perm+0x70/0xd0
>> [78140.503756] security_mmap_file+0x4b/0xd0
>> [78140.503759] vm_mmap_pgoff+0x50/0x150
>> [78140.503761] elf_map+0x9f/0x120
>> [78140.503763] load_elf_binary+0x521/0xc80
>> [78140.503767] bprm_execve+0x39f/0x660
>> [78140.503769] do_execveat_common+0x1d0/0x220
>> [78140.503771] __x64_sys_execveat+0x3d/0x50
>> [78140.503773] do_syscall_64+0x3d/0x90
>> [78140.503775] entry_SYSCALL_64_after_hwframe+0x44/0xae
>> [78140.503777]
>> -> #0 (&ctx->lock){+.+.}-{2:2}:
>> [78140.503780] __lock_acquire+0x1573/0x2ce0
>> [78140.503783] lock_acquire+0xbd/0x190
>> [78140.503785] _raw_spin_lock+0x2f/0x40
>> [78140.503787] update_file_ctx+0x19/0xe0
>> [78140.503788] aa_file_perm+0x56e/0x5c0
>> [78140.503790] match_file+0x78/0x90
>> [78140.503792] iterate_fd+0xae/0x150
>> [78140.503794] aa_inherit_files+0xbe/0x170
>> [78140.503796] apparmor_bprm_committing_creds+0x50/0x80
>> [78140.503798] security_bprm_committing_creds+0x1d/0x30
>> [78140.503800] begin_new_exec+0x3c5/0x450
>> [78140.503802] load_elf_binary+0x269/0xc80
>> [78140.503804] bprm_execve+0x39f/0x660
>> [78140.503806] do_execveat_common+0x1d0/0x220
>> [78140.503808] __x64_sys_execve+0x36/0x40
>> [78140.503809] do_syscall_64+0x3d/0x90
>> [78140.503812] entry_SYSCALL_64_after_hwframe+0x44/0xae
>> [78140.503815]
>> other info that might help us debug this:
>>
>> [78140.503816] Chain exists of:
>> &ctx->lock --> &p->alloc_lock --> &newf->file_lock
>>
>> [78140.503820] Possible unsafe locking scenario:
>>
>> [78140.503821] CPU0 CPU1
>> [78140.503823] ---- ----
>> [78140.503824] lock(&newf->file_lock);
>> [78140.503826] lock(&p->alloc_lock);
>> [78140.503828] lock(&newf->file_lock);
>> [78140.503830] lock(&ctx->lock);
>> [78140.503832]
>> *** DEADLOCK ***
>>
>> [78140.503833] 3 locks held by preconv/111629:
>> [78140.503835] #0: ffff888111b62550 (&sig->cred_guard_mutex){+.+.}-{3:3}, at: bprm_execve+0x39/0x660
>> [78140.503840] #1: ffff888111b625e8 (&sig->exec_update_lock){++++}-{3:3}, at: exec_mmap+0x4e/0x250
>> [78140.503844] #2: ffff888103d80458 (&newf->file_lock){+.+.}-{2:2}, at: iterate_fd+0x34/0x150
>> [78140.503849]
>> stack backtrace:
>> [78140.503851] CPU: 3 PID: 111629 Comm: preconv Tainted: G W 5.18.0-rc4-superb-owl-00006-gd615b5416f8a #12 6fd282a37da6f0e0172ecfa29689f3d250476a2b
>> [78140.503855] Hardware name: HP HP Laptop 14s-dq2xxx/87FD, BIOS F.15 09/15/2021
>> [78140.503856] Call Trace:
>> [78140.503858] <TASK>
>> [78140.503860] dump_stack_lvl+0x5a/0x74
>> [78140.503863] check_noncircular+0xd3/0xe0
>> [78140.503866] ? register_lock_class+0x35/0x2a0
>> [78140.503870] __lock_acquire+0x1573/0x2ce0
>> [78140.503872] ? prepend_path+0x375/0x410
>> [78140.503876] ? d_absolute_path+0x48/0x80
>> [78140.503879] ? aa_path_name+0x132/0x470
>> [78140.503883] ? lock_is_held_type+0xd0/0x130
>> [78140.503886] lock_acquire+0xbd/0x190
>> [78140.503888] ? update_file_ctx+0x19/0xe0
>> [78140.503892] _raw_spin_lock+0x2f/0x40
>> [78140.503894] ? update_file_ctx+0x19/0xe0
>> [78140.503896] update_file_ctx+0x19/0xe0
>> [78140.503899] aa_file_perm+0x56e/0x5c0
>> [78140.503904] ? aa_inherit_files+0x170/0x170
>> [78140.503906] match_file+0x78/0x90
>> [78140.503909] iterate_fd+0xae/0x150
>> [78140.503912] aa_inherit_files+0xbe/0x170
>> [78140.503915] apparmor_bprm_committing_creds+0x50/0x80
>> [78140.503918] security_bprm_committing_creds+0x1d/0x30
>> [78140.503921] begin_new_exec+0x3c5/0x450
>> [78140.503924] load_elf_binary+0x269/0xc80
>> [78140.503928] ? lock_release+0x1ee/0x260
>> [78140.503930] ? bprm_execve+0x399/0x660
>> [78140.503933] bprm_execve+0x39f/0x660
>> [78140.503936] do_execveat_common+0x1d0/0x220
>> [78140.503940] __x64_sys_execve+0x36/0x40
>> [78140.503942] do_syscall_64+0x3d/0x90
>> [78140.503946] entry_SYSCALL_64_after_hwframe+0x44/0xae
>> [78140.503948] RIP: 0033:0x7f700a8ea33b
>> [78140.503954] Code: Unable to access opcode bytes at RIP 0x7f700a8ea311.
>> [78140.503955] RSP: 002b:00007fff315e7db8 EFLAGS: 00000246 ORIG_RAX: 000000000000003b
>> [78140.503958] RAX: ffffffffffffffda RBX: 00007fff315e7dc0 RCX: 00007f700a8ea33b
>> [78140.503960] RDX: 000056419e9ea7e0 RSI: 000056419e9e9160 RDI: 00007fff315e7dc0
>> [78140.503962] RBP: 00007fff315e7f60 R08: 0000000000000008 R09: 0000000000000000
>> [78140.503964] R10: 0000000000000001 R11: 0000000000000246 R12: 000056419e9ea760
>> [78140.503965] R13: 000056419e9e9160 R14: 00007fff315e9eb4 R15: 00007fff315e9ebc
>> [78140.503971] </TASK>
>>
>> --
>> Ammar Faizi
next prev parent reply other threads:[~2022-06-06 15:19 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CAHk-=whmtHMzjaVUF9bS+7vE_rrRctcCTvsAeB8fuLYcyYLN-g@mail.gmail.com>
2022-04-27 17:59 ` Linux 5.18-rc4 Ammar Faizi
2022-04-27 18:31 ` Linus Torvalds
2022-06-06 15:19 ` Eric W. Biederman [this message]
2022-06-06 18:28 ` Linus Torvalds
2022-06-06 19:19 ` John Johansen
2022-06-06 19:47 ` Linus Torvalds
2022-06-06 20:23 ` Matthew Wilcox
2022-06-06 21:00 ` John Johansen
2022-06-13 22:48 ` Matthew Wilcox
2022-06-21 20:27 ` John Johansen
2022-07-13 9:37 ` Ammar Faizi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
[email protected] \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox