Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-26941

Windows deadlock during shutdown (3.2)

    • ALL
    • Platforms 2017-03-06
    • 0

      On Windows v3.2 builds, which uses VS 2013, there is a deadlock during shutdown if a condition variable is signaled at the same time as a thread is being shutdown. This is because std::condition_variable is using the Concurrency Runtime.

      Deadlock details:
      Thread 1: Holds Client Mutex, Waits on Loader Lock

       # Child-SP          RetAddr           Call Site
      00 00000000`0201f008 00000000`77848db8 ntdll!ZwWaitForSingleObject+0xa
      01 00000000`0201f010 00000000`77848cb4 ntdll!RtlpWaitOnCriticalSection+0xe8
      02 00000000`0201f0c0 00000000`7782219b ntdll!RtlEnterCriticalSection+0xd1
      03 00000000`0201f0f0 000007fe`fd8baa85 ntdll!LdrUnloadDll+0x2b
      04 00000000`0201f120 000007fe`f155f382 KERNELBASE!FreeLibrary+0x1d
      05 00000000`0201f150 000007fe`f1544d96 msvcr120!Concurrency::details::DeleteAsyncTimerAndUnloadLibrary+0x42 [f:\dd\vctools\crt\crtw32\concrt\utils.cpp @ 707]
      06 (Inline Function) --------`-------- msvcr120!Concurrency::details::TimedSingleWaitBlock::destroyTimer+0x22 [f:\dd\vctools\crt\crtw32\concrt\event.cpp @ 469]
      07 00000000`0201f180 000007fe`f15451e3 msvcr120!Concurrency::details::TimedSingleWaitBlock::Satisfy+0x56 [f:\dd\vctools\crt\crtw32\concrt\event.cpp @ 497]
      08 (Inline Function) --------`-------- msvcr120!Concurrency::details::EventWaitNode::Satisfy+0x14 [f:\dd\vctools\crt\crtw32\concrt\event.cpp @ 336]
      09 00000000`0201f1b0 000007fe`f16ac989 msvcr120!Concurrency::details::_Condition_variable::notify_one+0x53 [f:\dd\vctools\crt\crtw32\concrt\event.cpp @ 665]
      0a (Inline Function) --------`-------- msvcp120!do_signal+0x9 [f:\dd\vctools\crt\crtw32\stdcpp\thr\cond.c @ 74]
      0b 00000000`0201f220 00000001`3fd7772a msvcp120!_Cnd_signal+0xd [f:\dd\vctools\crt\crtw32\stdcpp\thr\cond.c @ 91]
      0c 00000000`0201f250 00000001`3fc33188 mongod!mongo::executor::NetworkInterfaceASIO::signalWorkAvailable+0x5a [c:\data\mci\12eb2148debb93ecaa2d5b93b14b3d93\src\src\mongo\executor\network_interface_asio.cpp @ 165]
      0d 00000000`0201f2a0 00000001`3fc09830 mongod!mongo::repl::ReplicationExecutor::scheduleWork+0x58 [c:\data\mci\12eb2148debb93ecaa2d5b93b14b3d93\src\src\mongo\db\repl\replication_executor.cpp @ 363]
      0e 00000000`0201f2e0 00000001`3fc04766 mongod!<lambda_d0c8b516c3ceb462ea69ef312169de9f>::operator()+0x20 [c:\data\mci\12eb2148debb93ecaa2d5b93b14b3d93\src\src\mongo\db\repl\replication_coordinator_impl.cpp @ 3622]
      0f 00000000`0201f320 00000001`3fc0b28a mongod!std::_Callable_obj<<lambda_d0c8b516c3ceb462ea69ef312169de9f>,0>::_ApplyX<mongo::StatusWith<mongo::executor::TaskExecutor::CallbackHandle>,std::function<void __cdecl(mongo::executor::TaskExecutor::CallbackArgs const & __ptr64)> const & __ptr64>+0x16 [c:\program files (x86)\microsoft visual studio 12.0\vc\include\xrefwrap @ 283]
      10 00000000`0201f360 00000001`3fbc0c67 mongod!std::_Func_impl<std::_Callable_obj<<lambda_d0c8b516c3ceb462ea69ef312169de9f>,0>,std::allocator<std::_Func_class<mongo::StatusWith<mongo::executor::TaskExecutor::CallbackHandle>,std::function<void __cdecl(mongo::executor::TaskExecutor::CallbackArgs const & __ptr64)> const & __ptr64> >,mongo::StatusWith<mongo::executor::TaskExecutor::CallbackHandle>,std::function<void __cdecl(mongo::executor::TaskExecutor::CallbackArgs const & __ptr64)> const & __ptr64>::_Do_call+0x1a [c:\program files (x86)\microsoft visual studio 12.0\vc\include\functional @ 228]
      11 00000000`0201f3a0 00000001`3fc1564f mongod!std::_Func_class<mongo::StatusWith<mongo::executor::TaskExecutor::CallbackHandle>,std::function<void __cdecl(mongo::executor::TaskExecutor::CallbackArgs const & __ptr64)> const & __ptr64>::operator()+0x27 [c:\program files (x86)\microsoft visual studio 12.0\vc\include\functional @ 315]
      12 00000000`0201f3e0 00000001`3fc107a2 mongod!mongo::repl::ReplicationCoordinatorImpl::_wrapAndScheduleWork+0xef [c:\data\mci\12eb2148debb93ecaa2d5b93b14b3d93\src\src\mongo\db\repl\replication_coordinator_impl.cpp @ 3655]
      13 00000000`0201f4d0 00000001`3fc18713 mongod!mongo::repl::ReplicationCoordinatorImpl::_scheduleWork+0x42 [c:\data\mci\12eb2148debb93ecaa2d5b93b14b3d93\src\src\mongo\db\repl\replication_coordinator_impl.cpp @ 3623]
      14 00000000`0201f540 00000001`3fca369e mongod!mongo::repl::ReplicationCoordinatorImpl::interruptAll+0x163 [c:\data\mci\12eb2148debb93ecaa2d5b93b14b3d93\src\src\mongo\db\repl\replication_coordinator_impl.cpp @ 1284]
      15 00000000`0201f5e0 00000001`3fa83bc3 mongod!mongo::ServiceContextMongoD::setKillAllOperations+0x5e [c:\data\mci\12eb2148debb93ecaa2d5b93b14b3d93\src\src\mongo\db\service_context_d.cpp @ 212]
      16 00000000`0201f620 00000001`3fed6782 mongod!mongo::exitCleanly+0xb3 [c:\data\mci\12eb2148debb93ecaa2d5b93b14b3d93\src\src\mongo\db\instance.cpp @ 1276]
      17 00000000`0201f700 00000001`3fbcd242 mongod!mongo::`anonymous namespace'::eventProcessingThread+0x232 [c:\data\mci\12eb2148debb93ecaa2d5b93b14b3d93\src\src\mongo\util\signal_handlers.cpp @ 156]
      18 00000000`0201f820 000007fe`f16ad24c mongod!std::_LaunchPad<std::_Bind<1,void,void (__cdecl*const)(void)> >::_Go+0x12 [c:\program files (x86)\microsoft visual studio 12.0\vc\include\thr\xthread @ 187]
      19 00000000`0201f850 000007fe`f1564f7f msvcp120!_Call_func+0x14 [f:\dd\vctools\crt\crtw32\stdcpp\thr\threadcall.cpp @ 28]
      1a 00000000`0201f890 000007fe`f1565126 msvcr120!_callthreadstartex+0x17 [f:\dd\vctools\crt\crtw32\startup\threadex.c @ 376]
      1b 00000000`0201f8c0 00000000`776f59bd msvcr120!_threadstartex+0x102 [f:\dd\vctools\crt\crtw32\startup\threadex.c @ 354]
      1c 00000000`0201f8f0 00000000`7782a2e1 kernel32!BaseThreadInitThunk+0xd
      1d 00000000`0201f920 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
      

      Thread 2: Holds load lock, Waits on client mutex

      0:052> k
       # Child-SP          RetAddr           Call Site
      00 00000000`097af6a8 000007fe`fd8b10ac ntdll!ZwWaitForSingleObject+0xa
      01 00000000`097af6b0 000007fe`f154744f KERNELBASE!WaitForSingleObjectEx+0x79
      02 00000000`097af750 000007fe`f154b223 msvcr120!Concurrency::details::ExternalContextBase::Block+0x4b [f:\dd\vctools\crt\crtw32\concrt\externalcontextbase.cpp @ 151]
      03 00000000`097af790 000007fe`f154ae47 msvcr120!Concurrency::critical_section::_Acquire_lock+0x143 [f:\dd\vctools\crt\crtw32\concrt\rtlocks.cpp @ 1209]
      04 00000000`097af7f0 000007fe`f16acd49 msvcr120!Concurrency::critical_section::lock+0x3f [f:\dd\vctools\crt\crtw32\concrt\rtlocks.cpp @ 1037]
      05 00000000`097af850 00000001`3fca01ea msvcp120!mtx_do_lock+0x85 [f:\dd\vctools\crt\crtw32\stdcpp\thr\mutex.c @ 67]
      06 00000000`097af8a0 00000001`3f91f7bd mongod!mongo::ServiceContext::ClientDeleter::operator()+0x3a [c:\data\mci\12eb2148debb93ecaa2d5b93b14b3d93\src\src\mongo\db\service_context.cpp @ 165]
      07 00000000`097af8f0 00000001`3ff1f296 mongod!boost::thread_specific_ptr<std::unique_ptr<mongo::Client,mongo::ServiceContext::ClientDeleter> >::delete_data::operator()+0x1d [c:\data\mci\12eb2148debb93ecaa2d5b93b14b3d93\src\src\third_party\boost-1.56.0\boost\thread\tss.hpp @ 42]
      08 00000000`097af920 00000000`7785616e mongod!boost::`anonymous namespace'::run_thread_exit_callbacks+0x126 [c:\data\mci\12eb2148debb93ecaa2d5b93b14b3d93\src\src\third_party\boost-1.56.0\libs\thread\src\win32\thread.cpp @ 199]
      09 00000000`097af970 00000000`7785792d ntdll!LdrpCallTlsInitializers+0xc8
      0a 00000000`097af9e0 00000000`77841818 ntdll!LdrShutdownThread+0x1cf
      0b 00000000`097afae0 000007fe`f1565023 ntdll!RtlExitUserThread+0x38
      0c 00000000`097afb20 000007fe`f1564f86 msvcr120!_endthreadex+0x8f [f:\dd\vctools\crt\crtw32\startup\threadex.c @ 431]
      0d 00000000`097afb50 000007fe`f1565126 msvcr120!_callthreadstartex+0x1e [f:\dd\vctools\crt\crtw32\startup\threadex.c @ 376]
      0e 00000000`097afb80 00000000`776f59bd msvcr120!_threadstartex+0x102 [f:\dd\vctools\crt\crtw32\startup\threadex.c @ 354]
      0f 00000000`097afbb0 00000000`7782a2e1 kernel32!BaseThreadInitThunk+0xd
      10 00000000`097afbe0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
      

            Assignee:
            andrew.morrow@mongodb.com Andrew Morrow (Inactive)
            Reporter:
            mark.benvenuto@mongodb.com Mark Benvenuto
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: