Kernel: Deadlock in SysFS when SMP is enabled

We still have many issues when SMP is enabled, but this was the first time I saw this specific deadlock, so it seemed noteworthy.

It is probably quite hard to reproduce, I only hit this bug once even if I used these steps many times (I was trying to repro another bug):

• Boot serenity with smp=on
• Not sure if this is relevant, but I used 8 cores
• I had at least one terminal doing loop { /usr/Tests/Kernel/TestKernelAlarm }
• And at least one instance of SystemMonitor running

After the deadlock, I got the backtraces with GDB:

Stack Traces Thread 8 (Thread 1.8 (CPU#7 [running])): #0 Kernel::ProcessorBase::wait_check () at ././Kernel/Arch/x86_64/Processor.h:268 #1 Kernel::RecursiveSpinlock<(Kernel::LockRank)0>::lock () at ././Kernel/Locking/Spinlock.h:79 #2 Kernel::SpinlockLocker >::SpinlockLocker () at ././Kernel/Locking/Spinlock.h:135 #3 0x000000200072cd1f in Kernel::Scheduler::pick_next () at ./Kernel/Tasks/Scheduler.cpp:221 #4 0x000000200072cf15 in Kernel::Scheduler::yield () at ./Kernel/Tasks/Scheduler.cpp:274 #5 0x000000200072d095 in Kernel::Scheduler::idle_loop () at ./Kernel/Tasks/Scheduler.cpp:521 #6 0x0000000000000000 in ?? ()

Thread 7 (Thread 1.7 (CPU#6 [running])): #0 Kernel::RecursiveSpinlock<(Kernel::LockRank)0>::lock () at ././Kernel/Locking/Spinlock.h:76 #1 Kernel::SpinlockLocker<Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::SpinlockLocker () at ././Kernel/Locking/Spinlock.h:135 #2 0x000000200072cd1f in Kernel::Scheduler::pick_next () at ./Kernel/Tasks/Scheduler.cpp:221 #3 0x000000200072cf15 in Kernel::Scheduler::yield () at ./Kernel/Tasks/Scheduler.cpp:274 #4 0x000000200072d095 in Kernel::Scheduler::idle_loop () at ./Kernel/Tasks/Scheduler.cpp:521 #5 0x0000000000000000 in ?? ()

Thread 6 (Thread 1.6 (CPU#5 [running])): #0 Kernel::Processor::smp_process_pending_messages () at ./Kernel/Arch/x86_64/Processor.cpp:970 #1 0x00000020002037f1 in Kernel::ProcessorBase::wait_check () at ././Kernel/Arch/x86_64/Processor.h:269 #2 Kernel::RecursiveSpinlock<(Kernel::LockRank)0>::lock () at ././Kernel/Locking/Spinlock.h:79 #3 Kernel::SpinlockLocker<Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::SpinlockLocker () at ././Kernel/Locking/Spinlock.h:135 #4 0x000000200072cd1f in Kernel::Scheduler::pick_next () at ./Kernel/Tasks/Scheduler.cpp:221 #5 0x000000200072cf15 in Kernel::Scheduler::yield () at ./Kernel/Tasks/Scheduler.cpp:274 #6 0x000000200072d095 in Kernel::Scheduler::idle_loop () at ./Kernel/Tasks/Scheduler.cpp:521 #7 0x0000000000000000 in ?? ()

Thread 5 (Thread 1.5 (CPU#4 [running])): #0 Kernel::RecursiveSpinlock<(Kernel::LockRank)0>::lock () at ././Kernel/Locking/Spinlock.h:76 #1 Kernel::SpinlockLocker<Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::SpinlockLocker () at ././Kernel/Locking/Spinlock.h:135 #2 0x00000020007388ff in Kernel::Thread::block_impl () at ./Kernel/Tasks/Thread.cpp:116 #3 0x0000002000739945 in Kernel::Thread::block<Kernel::Thread::SleepBlocker, Kernel::Thread::BlockTimeout, AK::Duration*&> () at ././Kernel/Tasks/Thread.h:824 #4 Kernel::Thread::sleep () at ./Kernel/Tasks/Thread.cpp:481 #5 0x00000020005dccc8 in Kernel::Thread::sleep () at ././Kernel/Tasks/Thread.h:841 #6 Kernel::MDIO::Clause22::Interface::mdio_handling_thread () at ./Kernel/Net/MDIO.cpp:98 #7 0x00000020005dd1e1 in operator() () at ./Kernel/Net/MDIO.cpp:22 #8 kernel_process_trampoline<Kernel::MDIO::Clause22::Interface::spawn_mdio_handling_task(u8)::<lambda()> >(void) () at ././Kernel/Tasks/Process.h:199 #9 0x0000000000000000 in ?? ()

Thread 4 (Thread 1.4 (CPU#3 [running])): #0 Kernel::RecursiveSpinlock<(Kernel::LockRank)0>::lock () at ././Kernel/Locking/Spinlock.h:76 #1 Kernel::SpinlockLocker<Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::SpinlockLocker () at ././Kernel/Locking/Spinlock.h:135 #2 0x00000020007404fe in Kernel::Thread::check_dispatch_pending_signal () at ./Kernel/Tasks/Thread.cpp:637 #3 0x00000020006b546f in syscall_handler () at ./Kernel/Syscalls/SyscallHandler.cpp:173 #4 0x0000002000795b14 in syscall_entry () at ./Kernel/Arch/x86_64/SyscallEntry.cpp:25 #5 0x0000000000000000 in ?? ()

Thread 3 (Thread 1.3 (CPU#2 [running])): #0 Kernel::InterruptDisabler::InterruptDisabler () at ././Kernel/Interrupts/InterruptDisabler.h:17 #1 Kernel::ProcessorBase::leave_critical () at ./Kernel/Arch/Processor.cpp:91 #2 0x00000020007927c5 in Kernel::Processor::smp_process_pending_messages () at ./Kernel/Arch/x86_64/Processor.cpp:1032 #3 0x00000020004cbdc1 in Kernel::ProcessorBase::wait_check () at ././Kernel/Arch/x86_64/Processor.h:269 #4 Kernel::RecursiveSpinlock<(Kernel::LockRank)8>::lock () at ././Kernel/Locking/Spinlock.h:79 #5 Kernel::SpinlockLocker<Kernel::RecursiveSpinlock<(Kernel::LockRank)8> >::SpinlockLocker () at ././Kernel/Locking/Spinlock.h:135 #6 operator() () at ./Kernel/FileSystem/SysFS/Subsystems/Kernel/Processes.cpp:117 #7 0x00000020004d1ee1 in call () at ././AK/Function.h:201 #8 AK::Function<AK::ErrorOr<void, AK::Error>(Kernel::Thread const&)>::operator() () at ././AK/Function.h:135 #9 Kernel::Process::try_for_each_thread(AK::Function<AK::ErrorOr<void, AK::Error> (Kernel::Thread const&)>) const::{lambda(auto:1&)#1}::operator()<AK::Detail::IntrusiveList<Kernel::Thread, Kernel::Thread*, &Kernel::Thread::m_process_thread_list_node> const>(AK::Detail::IntrusiveList<Kernel::Thread, Kernel::Thread*, &Kernel::Thread::m_process_thread_list_node> const&) const () at ././Kernel/Tasks/Process.h:1170 #10 0x00000020004cdf38 in Kernel::SpinlockProtectedBase<AK::Detail::IntrusiveList<Kernel::Thread, Kernel::Thread*, &Kernel::Thread::m_process_thread_list_node>, Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::with<Kernel::Process::try_for_each_thread(AK::Function<AK::ErrorOr<void, AK::Error> (Kernel::Thread const&)>) const::{lambda(auto:1&)#1}>(Kernel::Process::try_for_each_thread(AK::Function<AK::ErrorOr<void, AK::Error> (Kernel::Thread const&)>) const::{lambda(auto:1&)#1}) const () at ././Kernel/Locking/SpinlockProtectedBase.h:62 #11 Kernel::Process::try_for_each_thread () at ././Kernel/Tasks/Process.h:1168 #12 operator() () at ./Kernel/FileSystem/SysFS/Subsystems/Kernel/Processes.cpp:117 #13 0x00000020004d04d8 in operator() () at ./Kernel/FileSystem/SysFS/Subsystems/Kernel/Processes.cpp:154 #14 call () at ././AK/Function.h:201 #15 0x00000020007260b5 in AK::Function<AK::ErrorOr<void, AK::Error>(Kernel::Process&)>::operator() () at ././AK/Function.h:135 #16 0x0000002000710f81 in operator()<AK::Detail::IntrusiveListRelaxedConst<Kernel::Process, Kernel::Process*, &Kernel::Process::m_all_processes_list_node> > () at ./Kernel/Tasks/Process.cpp:82 #17 with<Kernel::Process::for_each_in_same_process_list(AK::Function<AK::ErrorOr(Kernel::Process&)>)::<lambda(const auto:198&)>::<lambda(const auto:200&)> > () at ././Kernel/Locking/SpinlockProtectedBase.h:69 #18 operator()<AK::RefPtrKernel::ScopedProcessList > () at ./Kernel/Tasks/Process.cpp:80 #19 with<Kernel::Process::for_each_in_same_process_list(AK::Function<AK::ErrorOr(Kernel::Process&)>)::<lambda(const auto:198&)> > () at ././Kernel/Locking/SpinlockProtectedBase.h:69 #20 Kernel::Process::for_each_in_same_process_list () at ./Kernel/Tasks/Process.cpp:68 #21 0x00000020004d08b8 in Kernel::SysFSOverallProcesses::try_generate () at ./Kernel/FileSystem/SysFS/Subsystems/Kernel/Processes.cpp:154 #22 0x00000020004e5929 in Kernel::SysFSGlobalInformation::refresh_data () at ./Kernel/FileSystem/SysFS/Subsystems/Kernel/GlobalInformation.cpp:57 #23 0x000000200045ffee in Kernel::OpenFileDescription::attach () at ./Kernel/FileSystem/OpenFileDescription.cpp:69 #24 0x0000002000469010 in Kernel::OpenFileDescription::try_create () at ./Kernel/FileSystem/OpenFileDescription.cpp:34 #25 0x00000020005155c5 in Kernel::VirtualFileSystem::open () at ./Kernel/FileSystem/VirtualFileSystem.cpp:453 #26 0x00000020005162a6 in Kernel::VirtualFileSystem::open () at ./Kernel/FileSystem/VirtualFileSystem.cpp:362 #27 0x0000002000674119 in Kernel::Process::open_impl () at ./Kernel/Syscalls/open.cpp:59 #28 0x00000020006851f9 in Kernel::Process::sys$open () at ./Kernel/Syscalls/profiled_syscalls.cpp:47 #29 0x00000020006b43b8 in Kernel::Syscall::handle () at ./Kernel/Syscalls/SyscallHandler.cpp:95 #30 0x00000020006b50f5 in syscall_handler () at ./Kernel/Syscalls/SyscallHandler.cpp:158 #31 0x0000002000795b14 in syscall_entry () at ./Kernel/Arch/x86_64/SyscallEntry.cpp:25 #32 0x0000000000000000 in ?? ()

Thread 2 (Thread 1.2 (CPU#1 [running])): #0 Kernel::RecursiveSpinlock<(Kernel::LockRank)0>::lock () at ././Kernel/Locking/Spinlock.h:76 #1 Kernel::SpinlockLocker<Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::SpinlockLocker () at ././Kernel/Locking/Spinlock.h:135 #2 0x0000002000731b98 in Kernel::Thread::reset_signals_for_exec () at ./Kernel/Tasks/Thread.cpp:746 #3 0x000000200063d66a in Kernel::Process::do_exec () at ./Kernel/Syscalls/execve.cpp:571 #4 0x0000002000640052 in Kernel::Process::exec () at ./Kernel/Syscalls/execve.cpp:951 #5 0x0000002000641d7f in Kernel::Process::exec () at ./Kernel/Syscalls/execve.cpp:1015 #6 Kernel::Process::sys$execve () at ./Kernel/Syscalls/execve.cpp:1015 #7 0x00000020006b43b8 in Kernel::Syscall::handle () at ./Kernel/Syscalls/SyscallHandler.cpp:95 #8 0x00000020006b50f5 in syscall_handler () at ./Kernel/Syscalls/SyscallHandler.cpp:158 #9 0x0000002000795b14 in syscall_entry () at ./Kernel/Arch/x86_64/SyscallEntry.cpp:25 #10 0x0000000000000000 in ?? ()

Thread 1 (Thread 1.1 (CPU#0 [running])): #0 Kernel::cli () at ././Kernel/Arch/x86_64/ASM_wrapper.h:18 #1 Kernel::ProcessorBase::disable_interrupts () at ././Kernel/Arch/x86_64/Processor.h:242 #2 Kernel::InterruptDisabler::InterruptDisabler () at ././Kernel/Interrupts/InterruptDisabler.h:19 #3 Kernel::ProcessorBase::leave_critical () at ./Kernel/Arch/Processor.cpp:91 #4 0x00000020007927c5 in Kernel::Processor::smp_process_pending_messages () at ./Kernel/Arch/x86_64/Processor.cpp:1032 #5 0x00000020002037f1 in Kernel::ProcessorBase::wait_check () at ././Kernel/Arch/x86_64/Processor.h:269 #6 Kernel::RecursiveSpinlock<(Kernel::LockRank)0>::lock () at ././Kernel/Locking/Spinlock.h:79 #7 Kernel::SpinlockLocker<Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::SpinlockLocker () at ././Kernel/Locking/Spinlock.h:135 #8 0x0000002000710737 in Kernel::SpinlockProtectedBase<AK::Detail::IntrusiveListRelaxedConst<Kernel::Process, Kernel::Process*, &Kernel::Process::m_all_processes_list_node>, Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::Locked<AK::Detail::IntrusiveListRelaxedConst<Kernel::Process, Kernel::Process*, &Kernel::Process::m_all_processes_list_node> >::Locked () at ././Kernel/Locking/SpinlockProtectedBase.h:30 #9 Kernel::SpinlockProtectedBase<AK::Detail::IntrusiveListRelaxedConst<Kernel::Process, Kernel::Process*, &Kernel::Process::m_all_processes_list_node>, Kernel::RecursiveSpinlock<(Kernel::LockRank)0> >::lock_mutable () at ././Kernel/Locking/SpinlockProtectedBase.h:49 #10 with<Kernel::Process::from_pid_ignoring_process_lists(Kernel::ProcessID)::<lambda(const auto:222&)> > () at ././Kernel/Locking/SpinlockProtectedBase.h:68 #11 Kernel::Process::from_pid_ignoring_process_lists () at ./Kernel/Tasks/Process.cpp:664 #12 0x000000200073dcb6 in Kernel::Thread::dispatch_signal () at ./Kernel/Tasks/Thread.cpp:919 #13 0x000000200072bec3 in Kernel::Scheduler::context_switch () at ./Kernel/Tasks/Scheduler.cpp:327 #14 0x000000200072cd7c in Kernel::Scheduler::pick_next () at ./Kernel/Tasks/Scheduler.cpp:242 #15 0x000000200072d105 in Kernel::Scheduler::invoke_async () at ./Kernel/Tasks/Scheduler.cpp:498 #16 0x0000002000791848 in Kernel::ProcessorBase::clear_critical () at ./Kernel/Arch/x86_64/Processor.cpp:1283 #17 0x000000200072fba5 in Kernel::Thread::yield_without_releasing_big_lock () at ./Kernel/Tasks/Thread.cpp:434 #18 0x0000002000738c4d in Kernel::Thread::block_impl () at ./Kernel/Tasks/Thread.cpp:185 #19 0x0000002000739945 in Kernel::Thread::block<Kernel::Thread::SleepBlocker, Kernel::Thread::BlockTimeout, AK::Duration*&> () at ././Kernel/Tasks/Thread.h:824 #20 Kernel::Thread::sleep () at ./Kernel/Tasks/Thread.cpp:481 #21 0x000000200062ec49 in Kernel::Process::sys$clock_nanosleep () at ./Kernel/Syscalls/clock.cpp:86 #22 0x00000020006b43b8 in Kernel::Syscall::handle () at ./Kernel/Syscalls/SyscallHandler.cpp:95 #23 0x00000020006b50f5 in syscall_handler () at ./Kernel/Syscalls/SyscallHandler.cpp:158 #24 0x0000002000795b14 in syscall_entry () at ./Kernel/Arch/x86_64/SyscallEntry.cpp:25 #25 0x0000000000000000 in ?? ()

The interesting ones are Thread 1 and 3.

Thread 1 blocks while waiting for Process::all_instances() while holding: 1. the scheduler lock, 2. The thread's own lock (Thread::m_lock, aquired in Scheduler::context_switch).

Thread 3 blocks while waiting for Thread::m_lock while holding all_instances's lock (aquired in for_each_in_same_process_list).

This is the usual ABBA locking order issue. I think the ranking system should have been able to catch that but all_instances's lock is not annotated.