Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-5988

Do not write log records as part recovery rollback to stable operation

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • WT10.0.0, 4.4.0-rc0, 4.7.0
    • Affects Version/s: None
    • Component/s: None
    • None
    • 5
    • Storage - Ra 2020-04-20

      WT recovery rollback to stable checkpoint hangs on writing checkpoint log record.
      As described in BF-16822, server restart hangs during recovery. Following is the call stack where it waits

      Thread 1 (Thread 0x7ffff7fd6180 (LWP 76029)):
      #0  __log_wait_for_earlier_slot () at src/third_party/wiredtiger/src/log/log.c:155
      #1  0x0000555556849c4b in __wt_log_release () at src/third_party/wiredtiger/src/log/log.c:1909
      #2  0x000055555684c0bf in __log_write_internal (flags=8, lsnp=0x0, record=<optimized out>, session=0x55555a851108) at src/third_party/wiredtiger/src/log/log.c:2668
      #3  __wt_log_write () at src/third_party/wiredtiger/src/log/log.c:2551
      #4  0x00005555568a9bcd in __wt_txn_checkpoint_log (session=0x55555a851108, full=<optimized out>, flags=<optimized out>, lsnp=0x0) at src/third_party/wiredtiger/src/txn/txn_log.c:543
      #5  0x00005555568a7177 in __txn_checkpoint (cfg=<optimized out>, session=0x55555a851108) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1037
      #6  __txn_checkpoint_wrapper () at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1076
      #7  0x00005555568a7f39 in __wt_txn_checkpoint () at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1133
      #8  0x00005555561c327c in __session_checkpoint (wt_session=0x55555a851108, config=<optimized out>) at src/third_party/wiredtiger/src/session/session_api.c:1979
      #9  0x00005555568b33c0 in __wt_rollback_to_stable () at src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c:1127
      #10 0x00005555568abe09 in __wt_txn_recover () at src/third_party/wiredtiger/src/txn/txn_recover.c:758
      #11 0x000055555681c6a7 in __wt_connection_workers (session=0x55555a84f000, cfg=0x7fffffffac20) at src/third_party/wiredtiger/src/conn/conn_open.c:211
      #12 0x000055555681834d in wiredtiger_open () at src/third_party/wiredtiger/src/conn/conn_api.c:2677
      #13 0x00005555567d137a in mongo::WiredTigerKVEngine::_openWiredTiger(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/basic_string.h:2281
      #14 0x00005555567d6bf8 in mongo::WiredTigerKVEngine::WiredTigerKVEngine(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::ClockSource*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long, bool, bool, bool, bool) ()
          at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:908
      #15 0x000055555679dc28 in mongo::(anonymous namespace)::WiredTigerFactory::create(mongo::StorageGlobalParams const&, mongo::StorageEngineLockFile const*) const ()
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/unique_ptr.h:342
      ---Type <return> to continue, or q <return> to quit---
      #16 0x0000555556ef13d5 in mongo::initializeStorageEngine(mongo::ServiceContext*, mongo::StorageEngineInitFlags) () at src/third_party/boost-1.70.0/boost/optional/optional.hpp:1448
      #17 0x0000555556543796 in mongo::(anonymous namespace)::_initAndListen(mongo::ServiceContext*, int) [clone .isra.1294] () at src/mongo/db/db.cpp:357
      #18 0x0000555556547199 in mongo::(anonymous namespace)::initAndListen (listenPort=<optimized out>, service=0x55555a642400) at src/mongo/db/db.cpp:1319
      #19 mongo::(anonymous namespace)::mongoDbMain(int, char**, char**) () at src/mongo/db/db.cpp:1319
      #20 0x00005555564a9269 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at src/mongo/db/db.cpp:1340
      (gdb) c
      

      The code is looping in that function to let the log->write_lsn to match with lot->slot_release_lsn. It seems there is no progress with log->write_lsn lead to wait forever. Values of write_lsn and slot_release_lsn are:

      (gdb) p log->write_lsn
      $2 = {l = {offset = 256, file = 694}, file_offset = 2980707303680}
      (gdb) p slot->slot_release_lsn
      $3 = {l = {offset = 1101824, file = 694}, file_offset = 2980708405248}
      

      Following are the call stacks of all the threads that are running in the system when this issue happened.

      Thread 9 (Thread 0x7fffeb7bb700 (LWP 76040)):
      #0  0x00007ffff495359e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000555556864499 in __wt_cond_wait_signal () at src/third_party/wiredtiger/src/os_posix/os_mtx_cond.c:115
      #2  0x00005555568907d6 in __wt_cond_auto_wait_signal (session=session@entry=0x55555a850c50, cond=0x55555a7cc380, progress=progress@entry=false, run_func=run_func@entry=0x0, signalled=signalled@entry=0x7fffeb7ba847)
          at src/third_party/wiredtiger/src/support/cond_auto.c:61
      #3  0x0000555556890823 in __wt_cond_auto_wait (session=session@entry=0x55555a850c50, cond=<optimized out>, progress=progress@entry=false, run_func=run_func@entry=0x0)
          at src/third_party/wiredtiger/src/support/cond_auto.c:80
      #4  0x000055555683d63c in __wt_evict_thread_run () at src/third_party/wiredtiger/src/evict/evict_lru.c:307
      #5  0x000055555689f1c9 in __thread_run () at src/third_party/wiredtiger/src/support/thread_group.c:31
      #6  0x00007ffff494d40b in start_thread () from /lib64/libpthread.so.0
      #7  0x00007ffff4687e7f in clone () from /lib64/libc.so.6
      
      Thread 8 (Thread 0x7fffebfbc700 (LWP 76039)):
      #0  0x00007ffff495359e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000555556864499 in __wt_cond_wait_signal () at src/third_party/wiredtiger/src/os_posix/os_mtx_cond.c:115
      #2  0x00005555568907d6 in __wt_cond_auto_wait_signal (session=session@entry=0x55555a850798, cond=0x55555a7cc380, progress=progress@entry=false, run_func=run_func@entry=0x0, signalled=signalled@entry=0x7fffebfbb847)
          at src/third_party/wiredtiger/src/support/cond_auto.c:61
      #3  0x0000555556890823 in __wt_cond_auto_wait (session=session@entry=0x55555a850798, cond=<optimized out>, progress=progress@entry=false, run_func=run_func@entry=0x0)
          at src/third_party/wiredtiger/src/support/cond_auto.c:80
      #4  0x000055555683d63c in __wt_evict_thread_run () at src/third_party/wiredtiger/src/evict/evict_lru.c:307
      #5  0x000055555689f1c9 in __thread_run () at src/third_party/wiredtiger/src/support/thread_group.c:31
      #6  0x00007ffff494d40b in start_thread () from /lib64/libpthread.so.0
      #7  0x00007ffff4687e7f in clone () from /lib64/libc.so.6
      
      Thread 7 (Thread 0x7fffec7bd700 (LWP 76038)):
      #0  0x00007ffff495359e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000555556864499 in __wt_cond_wait_signal () at src/third_party/wiredtiger/src/os_posix/os_mtx_cond.c:115
      #2  0x00005555568907d6 in __wt_cond_auto_wait_signal (session=session@entry=0x55555a8502e0, cond=0x55555a7cc380, progress=progress@entry=false, run_func=run_func@entry=0x0, signalled=signalled@entry=0x7fffec7bc847)
          at src/third_party/wiredtiger/src/support/cond_auto.c:61
      #3  0x0000555556890823 in __wt_cond_auto_wait (session=session@entry=0x55555a8502e0, cond=<optimized out>, progress=progress@entry=false, run_func=run_func@entry=0x0)
          at src/third_party/wiredtiger/src/support/cond_auto.c:80
      #4  0x000055555683d63c in __wt_evict_thread_run () at src/third_party/wiredtiger/src/evict/evict_lru.c:307
      #5  0x000055555689f1c9 in __thread_run () at src/third_party/wiredtiger/src/support/thread_group.c:31
      #6  0x00007ffff494d40b in start_thread () from /lib64/libpthread.so.0
      #7  0x00007ffff4687e7f in clone () from /lib64/libc.so.6
      
      Thread 6 (Thread 0x7fffecfbe700 (LWP 76037)):
      #0  0x00007ffff495359e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000555556864499 in __wt_cond_wait_signal () at src/third_party/wiredtiger/src/os_posix/os_mtx_cond.c:115
      #2  0x0000555556839b52 in __wt_cond_wait (run_func=0x0, usecs=10000, cond=<optimized out>, session=0x55555a84fe28) at src/third_party/wiredtiger/src/include/misc.i:19
      #3  __evict_lru_pages (session=session@entry=0x55555a84fe28, is_server=is_server@entry=false) at src/third_party/wiredtiger/src/evict/evict_lru.c:1123
      #4  0x000055555683b979 in __wt_evict_thread_run () at src/third_party/wiredtiger/src/evict/evict_lru.c:311
      #5  0x000055555689f1c9 in __thread_run () at src/third_party/wiredtiger/src/support/thread_group.c:31
      #6  0x00007ffff494d40b in start_thread () from /lib64/libpthread.so.0
      #7  0x00007ffff4687e7f in clone () from /lib64/libc.so.6
      
      Thread 5 (Thread 0x7fffed7bf700 (LWP 76036)):
      #0  0x00007ffff495361e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000555558207d97 in __gthread_cond_timedwait (__abs_timeout=0x7fffed7be900, __mutex=<optimized out>, __cond=0x55555a7c62d0)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/x86_64-mongodb-linux/bits/gthr-default.h:871
      #2  std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x55555a7c62d0)
      ---Type <return> to continue, or q <return> to quit--- 
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/condition_variable:178
      #3  std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x55555a7c62d0)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/condition_variable:106
      #4  std::_V2::condition_variable_any::wait_until<mongo::BasicLockableAdapter, std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x55555a7c62d0)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/condition_variable:286
      #5  mongo::Waitable::wait_until<mongo::BasicLockableAdapter> (timeout_time=..., lk=..., cv=..., clkSource=0x55555a121f10, waitable=<optimized out>) at src/mongo/util/waitable.h:102
      #6  mongo::ClockSource::waitForConditionUntil(mongo::stdx::condition_variable&, mongo::BasicLockableAdapter, mongo::Date_t, mongo::Waitable*) () at src/mongo/util/clock_source.cpp:48
      #7  0x0000555556bc7ad2 in mongo::ClockSource::waitForConditionUntil<std::unique_lock<mongo::latch_detail::Latch>, mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()> mutable::<lambda()> > (waitable=0x0, 
          pred=..., deadline=..., m=..., cv=..., this=0x55555a121f10) at src/mongo/util/lockable_adapter.h:50
      #8  mongo::PeriodicRunnerImpl::PeriodicJobImpl::<lambda()>::operator() (__closure=<optimized out>) at src/mongo/util/periodic_runner_impl.cpp:100
      #9  std::__invoke_impl<void, mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()> > (__f=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:60
      #10 std::__invoke<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()> > (__fn=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:95
      #11 std::__apply_impl<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()>, std::tuple<> > (__t=..., __f=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/tuple:1678
      #12 std::apply<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()>, std::tuple<> > (__t=..., __f=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/tuple:1687
      #13 mongo::stdx::thread::<lambda()>::operator() (this=<optimized out>) at src/mongo/stdx/thread.h:186
      #14 std::__invoke_impl<void, mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > (__f=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:60
      #15 std::__invoke<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > (__fn=...) at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:95
      #16 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > >::_M_invoke<0> (this=<optimized out>)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:234
      #17 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > >::operator() (this=<optimized out>)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:243
      #18 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::{lambda()#1}, , 0>(mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::{lambda()#1})::{lambda()#1}> > >::_M_run() () at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:186
      #19 0x00005555584e48af in execute_native_thread_routine () at ../../../../../src/combined/libstdc++-v3/src/c++11/thread.cc:80
      #20 0x00007ffff494d40b in start_thread () from /lib64/libpthread.so.0
      #21 0x00007ffff4687e7f in clone () from /lib64/libc.so.6
      
      Thread 4 (Thread 0x7fffedfc0700 (LWP 76035)):
      #0  0x00007ffff4953277 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005555584e170c in __gthread_cond_wait (__mutex=<optimized out>, __cond=__cond@entry=0x55555a7cdb70)
          at /data/mci/4aa8627b70212eb05b788e4bc6eb2ee3/toolchain-builder/tmp/build-gcc-v3.sh-uyz/build/x86_64-mongodb-linux/libstdc++-v3/include/x86_64-mongodb-linux/bits/gthr-default.h:864
      #2  std::condition_variable::wait (this=this@entry=0x55555a7cdb70, __lock=...) at ../../../../../src/combined/libstdc++-v3/src/c++11/condition_variable.cc:53
      #3  0x0000555557e63c43 in std::_V2::condition_variable_any::wait<std::unique_lock<mongo::latch_detail::Latch> > (__lock=..., this=0x55555a7cdb70)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/std_mutex.h:238
      #4  mongo::ThreadPool::_consumeTasks() () at src/mongo/util/concurrency/thread_pool.cpp:341
      #5  0x0000555557e645ea in mongo::ThreadPool::_workerThreadBody(mongo::ThreadPool*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
          at src/mongo/util/concurrency/thread_pool.cpp:273
      #6  0x0000555557e64930 in mongo::ThreadPool::<lambda()>::operator() (__closure=0x55555a6caf50) at src/mongo/util/concurrency/thread_pool.cpp:452
      #7  std::__invoke_impl<void, mongo::ThreadPool::_startWorkerThread_inlock()::<lambda()> > (__f=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:60
      #8  std::__invoke<mongo::ThreadPool::_startWorkerThread_inlock()::<lambda()> > (__fn=...) at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:95
      #9  std::__apply_impl<mongo::ThreadPool::_startWorkerThread_inlock()::<lambda()>, std::tuple<> > (__t=..., __f=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/tuple:1678
      ---Type <return> to continue, or q <return> to quit---
      #10 std::apply<mongo::ThreadPool::_startWorkerThread_inlock()::<lambda()>, std::tuple<> > (__t=..., __f=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/tuple:1687
      #11 mongo::stdx::thread::<lambda()>::operator() (this=0x55555a6caf48) at src/mongo/stdx/thread.h:186
      #12 std::__invoke_impl<void, mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::ThreadPool::_startWorkerThread_inlock()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > (__f=...) at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:60
      #13 std::__invoke<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::ThreadPool::_startWorkerThread_inlock()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > (__fn=...) at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:95
      #14 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::ThreadPool::_startWorkerThread_inlock()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > >::_M_invoke<0> (this=0x55555a6caf48)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:234
      #15 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::ThreadPool::_startWorkerThread_inlock()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > >::operator() (this=0x55555a6caf48)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:243
      #16 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::ThreadPool::_startWorkerThread_inlock()::{lambda()#4}, , 0>(mongo::ThreadPool::_startWorkerThread_inlock()::{lambda()#4})::{lambda()#1}> > >::_M_run() () at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:186
      #17 0x00005555584e48af in execute_native_thread_routine () at ../../../../../src/combined/libstdc++-v3/src/c++11/thread.cc:80
      #18 0x00007ffff494d40b in start_thread () from /lib64/libpthread.so.0
      #19 0x00007ffff4687e7f in clone () from /lib64/libc.so.6
      
      Thread 3 (Thread 0x7fffee7c1700 (LWP 76034)):
      #0  0x00007ffff4953277 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x00005555584e170c in __gthread_cond_wait (__mutex=<optimized out>, __cond=__cond@entry=0x55555a790c50)
          at /data/mci/4aa8627b70212eb05b788e4bc6eb2ee3/toolchain-builder/tmp/build-gcc-v3.sh-uyz/build/x86_64-mongodb-linux/libstdc++-v3/include/x86_64-mongodb-linux/bits/gthr-default.h:864
      #2  std::condition_variable::wait (this=this@entry=0x55555a790c50, __lock=...) at ../../../../../src/combined/libstdc++-v3/src/c++11/condition_variable.cc:53
      #3  0x00005555582070b7 in std::_V2::condition_variable_any::wait<std::unique_lock<std::mutex> > (__lock=..., this=0x55555a790c50)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/std_mutex.h:238
      #4  mongo::BackgroundThreadClockSource::<lambda()>::operator() (__closure=0x55555a7b1e30) at src/mongo/util/background_thread_clock_source.cpp:188
      #5  std::__invoke_impl<void, mongo::BackgroundThreadClockSource::_startTimerThread()::<lambda()> > (__f=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:60
      #6  std::__invoke<mongo::BackgroundThreadClockSource::_startTimerThread()::<lambda()> > (__fn=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:95
      #7  std::__apply_impl<mongo::BackgroundThreadClockSource::_startTimerThread()::<lambda()>, std::tuple<> > (__t=..., __f=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/tuple:1678
      #8  std::apply<mongo::BackgroundThreadClockSource::_startTimerThread()::<lambda()>, std::tuple<> > (__t=..., __f=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/tuple:1687
      #9  mongo::stdx::thread::<lambda()>::operator() (this=0x55555a7b1e28) at src/mongo/stdx/thread.h:186
      #10 std::__invoke_impl<void, mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::BackgroundThreadClockSource::_startTimerThread()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > (__f=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:60
      #11 std::__invoke<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::BackgroundThreadClockSource::_startTimerThread()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > (__fn=...) at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:95
      #12 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::BackgroundThreadClockSource::_startTimerThread()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > >::_M_invoke<0> (this=0x55555a7b1e28)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:234
      #13 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::BackgroundThreadClockSource::_startTimerThread()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > >::operator() (this=0x55555a7b1e28)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:243
      #14 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::BackgroundThreadClockSource::_startTimerThread()::{lambda()#1}, , 0>(mongo::BackgroundThreadClockSource::_startTimerThread()::{lambda()#1})::{lambda()#1}> > >::_M_run() () at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:186
      #15 0x00005555584e48af in execute_native_thread_routine () at ../../../../../src/combined/libstdc++-v3/src/c++11/thread.cc:80
      #16 0x00007ffff494d40b in start_thread () from /lib64/libpthread.so.0
      #17 0x00007ffff4687e7f in clone () from /lib64/libc.so.6
      ---Type <return> to continue, or q <return> to quit---
      
      Thread 2 (Thread 0x7fffeefc2700 (LWP 76033)):
      #0  0x00007ffff45cf9f1 in sigwaitinfo () from /lib64/libc.so.6
      #1  0x0000555556dd37a9 in mongo::(anonymous namespace)::waitForSignal (result=0x7fffeefc17b0, sigset=...) at src/mongo/util/signal_handlers.cpp:177
      #2  mongo::(anonymous namespace)::signalProcessingThread(mongo::LogFileStatus) () at src/mongo/util/signal_handlers.cpp:295
      #3  0x0000555556dd3ff5 in std::__invoke_impl<void, void (*)(mongo::LogFileStatus), mongo::LogFileStatus> (__f=@0x55555a7b1d50: 0x555556dd35a0 <mongo::(anonymous namespace)::signalProcessingThread(mongo::LogFileStatus)>)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:89
      #4  std::__invoke<void (*)(mongo::LogFileStatus), mongo::LogFileStatus> (__fn=@0x55555a7b1d50: 0x555556dd35a0 <mongo::(anonymous namespace)::signalProcessingThread(mongo::LogFileStatus)>)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:95
      #5  std::__apply_impl<void (*)(mongo::LogFileStatus), std::tuple<mongo::LogFileStatus>, 0ul> (__f=@0x55555a7b1d50: 0x555556dd35a0 <mongo::(anonymous namespace)::signalProcessingThread(mongo::LogFileStatus)>, __t=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/tuple:1678
      #6  std::apply<void (*)(mongo::LogFileStatus), std::tuple<mongo::LogFileStatus> > (__f=@0x55555a7b1d50: 0x555556dd35a0 <mongo::(anonymous namespace)::signalProcessingThread(mongo::LogFileStatus)>, __t=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/tuple:1687
      #7  mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}::operator()() (this=0x55555a7b1d48)
          at src/mongo/stdx/thread.h:186
      #8  std::__invoke_impl<void, mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}>(std::__invoke_other, mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}&&) (__f=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:60
      #9  std::__invoke<mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}>(mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}&&) (__fn=...)
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:95
      #10 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x55555a7b1d48) at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:234
      #11 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}> >::operator()() (
          this=0x55555a7b1d48) at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:243
      #12 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}> > >::_M_run() (this=0x55555a7b1d40) at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:186
      #13 0x00005555584e48af in execute_native_thread_routine () at ../../../../../src/combined/libstdc++-v3/src/c++11/thread.cc:80
      #14 0x00007ffff494d40b in start_thread () from /lib64/libpthread.so.0
      #15 0x00007ffff4687e7f in clone () from /lib64/libc.so.6
      
      Thread 1 (Thread 0x7ffff7fd6180 (LWP 76029)):
      #0  __log_wait_for_earlier_slot () at src/third_party/wiredtiger/src/log/log.c:155
      #1  0x0000555556849c4b in __wt_log_release () at src/third_party/wiredtiger/src/log/log.c:1909
      #2  0x000055555684c0bf in __log_write_internal (flags=8, lsnp=0x0, record=<optimized out>, session=0x55555a851108) at src/third_party/wiredtiger/src/log/log.c:2668
      #3  __wt_log_write () at src/third_party/wiredtiger/src/log/log.c:2551
      #4  0x00005555568a9bcd in __wt_txn_checkpoint_log (session=0x55555a851108, full=<optimized out>, flags=<optimized out>, lsnp=0x0) at src/third_party/wiredtiger/src/txn/txn_log.c:543
      #5  0x00005555568a7177 in __txn_checkpoint (cfg=<optimized out>, session=0x55555a851108) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1037
      #6  __txn_checkpoint_wrapper () at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1076
      #7  0x00005555568a7f39 in __wt_txn_checkpoint () at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1133
      #8  0x00005555561c327c in __session_checkpoint (wt_session=0x55555a851108, config=<optimized out>) at src/third_party/wiredtiger/src/session/session_api.c:1979
      #9  0x00005555568b33c0 in __wt_rollback_to_stable () at src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c:1127
      #10 0x00005555568abe09 in __wt_txn_recover () at src/third_party/wiredtiger/src/txn/txn_recover.c:758
      #11 0x000055555681c6a7 in __wt_connection_workers (session=0x55555a84f000, cfg=0x7fffffffac20) at src/third_party/wiredtiger/src/conn/conn_open.c:211
      #12 0x000055555681834d in wiredtiger_open () at src/third_party/wiredtiger/src/conn/conn_api.c:2677
      #13 0x00005555567d137a in mongo::WiredTigerKVEngine::_openWiredTiger(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/basic_string.h:2281
      #14 0x00005555567d6bf8 in mongo::WiredTigerKVEngine::WiredTigerKVEngine(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::ClockSource*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long, bool, bool, bool, bool) ()
          at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:908
      #15 0x000055555679dc28 in mongo::(anonymous namespace)::WiredTigerFactory::create(mongo::StorageGlobalParams const&, mongo::StorageEngineLockFile const*) const ()
          at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/unique_ptr.h:342
      ---Type <return> to continue, or q <return> to quit---
      #16 0x0000555556ef13d5 in mongo::initializeStorageEngine(mongo::ServiceContext*, mongo::StorageEngineInitFlags) () at src/third_party/boost-1.70.0/boost/optional/optional.hpp:1448
      #17 0x0000555556543796 in mongo::(anonymous namespace)::_initAndListen(mongo::ServiceContext*, int) [clone .isra.1294] () at src/mongo/db/db.cpp:357
      #18 0x0000555556547199 in mongo::(anonymous namespace)::initAndListen (listenPort=<optimized out>, service=0x55555a642400) at src/mongo/db/db.cpp:1319
      #19 mongo::(anonymous namespace)::mongoDbMain(int, char**, char**) () at src/mongo/db/db.cpp:1319
      #20 0x00005555564a9269 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at src/mongo/db/db.cpp:1340
      (gdb) c
      Continuing.
      

            Assignee:
            haribabu.kommi@mongodb.com Haribabu Kommi
            Reporter:
            haribabu.kommi@mongodb.com Haribabu Kommi
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: