-
Type: Bug
-
Resolution: Fixed
-
Priority: Critical - P2
-
Affects Version/s: 4.0.0
-
Component/s: Replication
-
None
-
Fully Compatible
-
ALL
-
v4.0
-
Repl 2018-07-16, Repl 2018-07-30, Repl 2018-08-13
-
62
Unstashing a transaction's lock state also involves reacquiring a WiredTiger write ticket. The call to Locker::reacquireTicket() has no lock timeout and occurs while holding the Session::_mutex. Since the killAllExpiredTransactions() function calls Session::abortArbitraryTransactionIfExpired() which attempts to acquire the Session::_mutex, it isn't possible to reap expired transactions while a transaction's lock state is being unstashed and no WiredTiger write tickets are available.
Thread 36: "conn1" (Thread 0x7f8e579ef700 (LWP 14971)) #0 sem_timedwait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101 #1 0x00007f8e595b861e in mongo::TicketHolder::waitForTicketUntil (this=<optimized out>, opCtx=<optimized out>, until=...) at src/mongo/util/concurrency/ticketholder.cpp:104 #2 0x00007f8e595b8673 in mongo::TicketHolder::waitForTicket (this=<optimized out>, opCtx=<optimized out>) at src/mongo/util/concurrency/ticketholder.cpp:90 #3 0x00007f8e5951eacd in mongo::LockerImpl<false>::_acquireTicket (this=<optimized out>, opCtx=<optimized out>, mode=<optimized out>, deadline=...) at src/mongo/db/concurrency/lock_state.cpp:338 #4 0x00007f8e5951eb1d in mongo::LockerImpl<false>::reacquireTicket (this=<optimized out>, opCtx=<optimized out>) at src/mongo/db/concurrency/lock_state.cpp:322 #5 0x00007f8e592fd06f in mongo::Session::TxnResources::release (this=<optimized out>, opCtx=<optimized out>) at src/mongo/db/session.cpp:658 #6 0x00007f8e593021e3 in mongo::Session::unstashTransactionResources (this=<optimized out>, opCtx=<optimized out>, cmdName=...) at src/mongo/db/session.cpp:775 #7 0x00007f8e584adc80 in mongo::(anonymous namespace)::invokeInTransaction (opCtx=<optimized out>, invocation=<optimized out>, replyBuilder=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:466 #8 0x00007f8e584b0dd9 in runCommandImpl (sessionOptions=..., extraFieldsBuilder=<optimized out>, behaviors=..., startOperationTime=..., replyBuilder=<optimized out>, request=..., invocation=<optimized out>, opCtx=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:537 #9 mongo::(anonymous namespace)::execCommandDatabase (opCtx=<optimized out>, command=<optimized out>, request=..., replyBuilder=<optimized out>, behaviors=...) at src/mongo/db/service_entry_point_common.cpp:873 #10 0x00007f8e584b2291 in mongo::(anonymous namespace)::<lambda()>::operator()(void) const (__closure=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:1018 #11 0x00007f8e584b2291 in mongo::ServiceEntryPointCommon::handleRequest (opCtx=<optimized out>, m=..., behaviors=...) #12 0x00007f8e584b31d1 in runCommands (behaviors=..., message=..., opCtx=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:1033 #13 mongo::ServiceEntryPointCommon::handleRequest (opCtx=<optimized out>, m=..., behaviors=...) at src/mongo/db/service_entry_point_common.cpp:1307 #14 0x00007f8e584a09da in mongo::ServiceEntryPointMongod::handleRequest (this=<optimized out>, opCtx=<optimized out>, m=...) at src/mongo/db/service_entry_point_mongod.cpp:123 #15 0x00007f8e584ab76a in mongo::ServiceStateMachine::_processMessage (this=<optimized out>, guard=...) at src/mongo/transport/service_state_machine.cpp:378 ... #36 mongo::(anonymous namespace)::runFunc (ctx=<optimized out>) at src/mongo/transport/service_entry_point_utils.cpp:55 #37 0x00007f8e56844184 in start_thread (arg=0x7f8e579ef700) at pthread_create.c:312 #38 0x00007f8e5657103d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Thread 39: "startPe.actions" (Thread 0x7f8e4551d700 (LWP 14968)) #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f8e56846649 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f8e56846470 in __GI___pthread_mutex_lock (mutex=0x7f8e5f6f8080) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f8e592fdcc7 in __gthread_mutex_lock (__mutex=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/x86_64-mongodb-linux/bits/gthr-default.h:748 #4 lock (this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/mutex:135 #5 lock_guard (__m=..., this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/mutex:386 #6 mongo::Session::abortArbitraryTransactionIfExpired (this=<optimized out>) at src/mongo/db/session.cpp:858 #7 0x00007f8e58a90d7b in mongo::<lambda(mongo::OperationContext*, mongo::Session*)>::operator()(mongo::Session *, mongo::OperationContext *) (session=<optimized out>, opCtx=<optimized out>, __closure=<optimized out>) at src/mongo/db/kill_sessions_local.cpp:76 #8 0x00007f8e59306504 in operator() (__args#1=<optimized out>, __args#0=<optimized out>, this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267 #9 mongo::SessionCatalog::scanSessions(mongo::OperationContext*, mongo::SessionKiller::Matcher const&, std::function<void (mongo::OperationContext*, mongo::Session*)>) (this=<optimized out>, opCtx=<optimized out>, matcher=..., workerFn=...) at src/mongo/db/session_catalog.cpp:208 #10 0x00007f8e58a911b8 in mongo::killAllExpiredTransactions (opCtx=<optimized out>) at src/mongo/db/kill_sessions_local.cpp:96 #11 0x00007f8e5881041f in operator() (__closure=<optimized out>, client=<optimized out>) at src/mongo/db/periodic_runner_job_abort_expired_transactions.cpp:91 #12 std::_Function_handler<void(mongo::Client*), mongo::startPeriodicThreadToAbortExpiredTransactions(mongo::ServiceContext*)::<lambda(mongo::Client*)> >::_M_invoke(const std::_Any_data &, <unknown type in /home/maxh/debugging/mongo/mongod, CU 0x0, DIE 0x316c7>) (__functor=..., __args#0=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1871 #13 0x00007f8e5923d47c in operator() (__args#0=<optimized out>, this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267 #14 operator() (__closure=<optimized out>) at src/mongo/util/periodic_runner_impl.cpp:108 #15 _M_invoke<> (this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1531 #16 operator() (this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1520 #17 std::thread::_Impl<std::_Bind_simple<mongo::PeriodicRunnerImpl::PeriodicJobImpl::run()::<lambda()>()> >::_M_run(void) (this=<optimized out>) at /opt/mongodbtoolchain/v2/include/c++/5.4.0/thread:115 #18 0x00007f8e59e680d0 in std::execute_native_thread_routine (__p=<optimized out>) at ../../../../../gcc-5.4.0/libstdc++-v3/src/c++11/thread.cc:84 #19 0x00007f8e56844184 in start_thread (arg=0x7f8e4551d700) at pthread_create.c:312 #20 0x00007f8e5657103d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
- is related to
-
SERVER-35217 killSessions command attempts to kill a transaction while holding SessionCatalog::_mutex, which leads to deadlock
- Closed
- related to
-
SERVER-43868 Session::TxnResources::release() can throw exceptions in destructors (4.0.x only)
- Closed
-
SERVER-39320 lower wiredTigerConcurrentWriteTransactions to 64 in concurrency_simultaneous suites
- Closed