Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-1996

Use-after-free in __wt_txn_rollback after allocation failure

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • WT2.7.0
    • Affects Version/s: None
    • Component/s: None

      While simulating out-of-memory conditions in __wt_calloc, I triggered a use-after-free error in __wt_txn_rollback.

      Sequence of events:

      1. A reconciliation fails because an in-memory page gets too large, which triggers a page rewrite
      2. While the page is being rewritten, an allocation failure causes one of the modifications not to apply:
        ---> In __wt_row_modify, the call to __wt_row_insert_alloc fails with ENOMEM (line 163)
      3. The err block in __wt_row_modify is jumped to, which ends up freeing the WT_UPDATE data that's already been added to the transaction's modification list (line 221)
      4. The transaction is rolled back by the integration layer, causing __wt_txn_rollback to access already-freed WT_UPDATE data while iterating over the transaction's modification list

      Excerpted code:

      int
      __wt_row_modify(WT_SESSION_IMPL *session, WT_CURSOR_BTREE *cbt,
          WT_ITEM *key, WT_ITEM *value, WT_UPDATE *upd, int is_remove)
      {
      
      [....snip....]
      
                 /*
                  * Allocate a WT_INSERT/WT_UPDATE pair and transaction ID, and
                  * update the cursor to reference it (the WT_INSERT_HEAD might
                  * be allocated, the WT_INSERT was allocated).
                  */
      163        WT_ERR(__wt_row_insert_alloc(
      164            session, key, skipdepth, &ins, &ins_size));
      165        cbt->ins_head = ins_head;
      166        cbt->ins = ins;
      167
      168        if (upd == NULL) {
      169            WT_ERR(
      170                __wt_update_alloc(session, value, &upd, &upd_size));
      171            WT_ERR(__wt_txn_modify(session, upd));
      172            logged = 1;
      173
      174            /* Avoid WT_CURSOR.update data copy. */
      175            cbt->modify_update = upd;
      176        } else
      177            upd_size = __wt_update_list_memsize(upd);
      
      [....snip....]
      
      	if (0) {
      213 err:		/*
      214		 * Remove the update from the current transaction, so we don't
      215		 * try to modify it on rollback.
      216		 */
      217		if (logged)
      218			__wt_txn_unmodify(session);
      219		__wt_free(session, ins);
      220		cbt->ins = NULL;
      221		__wt_free(session, upd);
      	}
      
      	return (ret);
      

      ASan report:

      ==7600== ERROR: AddressSanitizer: heap-use-after-free on address 0x601e0005d1f0 at pc 0x2f7d36c bp 0x7f531676b9f0 sp 0x7f531676b9e8
      WRITE of size 8 at 0x601e0005d1f0 thread T27
          #0 0x2f7d36b in __wt_txn_rollback /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/txn/txn.c:583
          #1 0x2f5e350 in __session_rollback_transaction /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/session/session_api.c:849
          #2 0x206c3ca in mongo::WiredTigerRecoveryUnit::_txnClose(bool) /home/s/code/mongo/mongo/src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp:310
          #3 0x206aac0 in mongo::WiredTigerRecoveryUnit::_abort() /home/s/code/mongo/mongo/src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp:138
          #4 0x206b15b in mongo::WiredTigerRecoveryUnit::abortUnitOfWork() /home/s/code/mongo/mongo/src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp:174
          #5 0x15b7fb1 in mongo::WriteUnitOfWork::~WriteUnitOfWork() /home/s/code/mongo/mongo/src/mongo/db/operation_context.h:235
          #6 0x1dbb01a in mongo::repl::writeOpsToOplog(mongo::OperationContext*, std::deque<mongo::BSONObj, std::allocator<mongo::BSONObj> > const&) /home/s/code/mongo/mongo/src/mongo/db/repl/oplog.cpp:348
          #7 0x1eb8b0c in mongo::repl::SyncTail::multiApply(mongo::OperationContext*, mongo::repl::SyncTail::OpQueue const&, mongo::OldThreadPool*, mongo::OldThreadPool*, std::function<void (std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> > const&, mongo::repl::SyncTail*)>, mongo::repl::SyncTail*, bool) /home/s/code/mongo/mongo/src/mongo/db/repl/sync_tail.cpp:378
          #8 0x1eb9f24 in mongo::repl::SyncTail::oplogApplication() /home/s/code/mongo/mongo/src/mongo/db/repl/sync_tail.cpp:572
          #9 0x1eae65c in mongo::repl::runSyncThread() /home/s/code/mongo/mongo/src/mongo/db/repl/rs_sync.cpp:133
          #10 0x1dab3bd in boost::detail::thread_data<void (*)()>::run() /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/boost/thread/detail/thread.hpp:115
          #11 0x241311b in thread_proxy /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/libs/thread/src/pthread/thread.cpp:173
          #12 0x7f532724ab97 (/usr/lib/x86_64-linux-gnu/libasan.so.0+0x18b97)
          #13 0x7f53263f0181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312
          #14 0x7f532611d47c in clone /build/buildd/eglibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:111
      
      0x601e0005d1f0 is located 0 bytes inside of 166-byte region [0x601e0005d1f0,0x601e0005d296)
      freed by thread T27 here:
          #0 0x7f532724733a in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.0+0x1533a)
          #1 0x2ee51a0 in __wt_free_int /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/os_posix/os_alloc.c:258
          #2 0x2dfb42a in __wt_row_modify /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/btree/row_modify.c:221
          #3 0x2dcfe21 in __split_multi_inmem /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/btree/bt_split.c:748
          #4 0x2dd34e3 in __wt_split_rewrite /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/btree/bt_split.c:1437
          #5 0x2e9af3c in __evict_page_dirty_update /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/evict/evict_page.c:268
          #6 0x2e9a13a in __wt_evict /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/evict/evict_page.c:119
          #7 0x2e94183 in __wt_evict_page /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/evict/evict_lru.c:697
          #8 0x2dae3f1 in __wt_page_release_evict /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/include/btree.i:1149
          #9 0x2daf038 in __wt_page_in_func /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/btree/bt_page.c:153
          #10 0x2dfdbec in __wt_page_swap_func /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/include/btree.i:1244
          #11 0x2dffa56 in __wt_row_search /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/btree/row_srch.c:293
          #12 0x2d8d55e in __cursor_row_search /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/btree/bt_cursor.c:241
          #13 0x2d8f5cc in __wt_btcur_insert /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/btree/bt_cursor.c:538
          #14 0x2e49905 in __curfile_insert /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/cursor/cur_file.c:245
          #15 0x205d94c in mongo::WiredTigerRecordStore::insertRecord(mongo::OperationContext*, char const*, int, bool) /home/s/code/mongo/mongo/src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp:797
          #16 0x17a1cdb in mongo::Collection::_insertDocument(mongo::OperationContext*, mongo::BSONObj const&, bool) /home/s/code/mongo/mongo/src/mongo/db/catalog/collection.cpp:376
          #17 0x17a12d9 in mongo::Collection::insertDocument(mongo::OperationContext*, mongo::BSONObj const&, bool, bool) /home/s/code/mongo/mongo/src/mongo/db/catalog/collection.cpp:313
          #18 0x1dbad55 in mongo::repl::writeOpsToOplog(mongo::OperationContext*, std::deque<mongo::BSONObj, std::allocator<mongo::BSONObj> > const&) /home/s/code/mongo/mongo/src/mongo/db/repl/oplog.cpp:354
          #19 0x1eb8b0c in mongo::repl::SyncTail::multiApply(mongo::OperationContext*, mongo::repl::SyncTail::OpQueue const&, mongo::OldThreadPool*, mongo::OldThreadPool*, std::function<void (std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> > const&, mongo::repl::SyncTail*)>, mongo::repl::SyncTail*, bool) /home/s/code/mongo/mongo/src/mongo/db/repl/sync_tail.cpp:378
          #20 0x1eb9f24 in mongo::repl::SyncTail::oplogApplication() /home/s/code/mongo/mongo/src/mongo/db/repl/sync_tail.cpp:572
          #21 0x1eae65c in mongo::repl::runSyncThread() /home/s/code/mongo/mongo/src/mongo/db/repl/rs_sync.cpp:133
          #22 0x1dab3bd in boost::detail::thread_data<void (*)()>::run() /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/boost/thread/detail/thread.hpp:115
          #23 0x241311b in thread_proxy /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/libs/thread/src/pthread/thread.cpp:173
          #24 0x7f532724ab97 (/usr/lib/x86_64-linux-gnu/libasan.so.0+0x18b97)
      
      previously allocated by thread T27 here:
          #0 0x7f53272474e5 in calloc (/usr/lib/x86_64-linux-gnu/libasan.so.0+0x154e5)
          #1 0x2ee4379 in __wt_calloc /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/os_posix/os_alloc.c:60
          #2 0x2dfb95a in __wt_update_alloc /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/btree/row_modify.c:276
          #3 0x2dfafa1 in __wt_row_modify /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/btree/row_modify.c:169
          #4 0x2d8d656 in __cursor_row_modify /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/btree/bt_cursor.c:266
          #5 0x2d8f6ab in __wt_btcur_insert /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/btree/bt_cursor.c:547
          #6 0x2e49905 in __curfile_insert /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/cursor/cur_file.c:245
          #7 0x205d94c in mongo::WiredTigerRecordStore::insertRecord(mongo::OperationContext*, char const*, int, bool) /home/s/code/mongo/mongo/src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp:797
          #8 0x17a1cdb in mongo::Collection::_insertDocument(mongo::OperationContext*, mongo::BSONObj const&, bool) /home/s/code/mongo/mongo/src/mongo/db/catalog/collection.cpp:376
          #9 0x17a12d9 in mongo::Collection::insertDocument(mongo::OperationContext*, mongo::BSONObj const&, bool, bool) /home/s/code/mongo/mongo/src/mongo/db/catalog/collection.cpp:313
          #10 0x1dbad55 in mongo::repl::writeOpsToOplog(mongo::OperationContext*, std::deque<mongo::BSONObj, std::allocator<mongo::BSONObj> > const&) /home/s/code/mongo/mongo/src/mongo/db/repl/oplog.cpp:354
          #11 0x1eb8b0c in mongo::repl::SyncTail::multiApply(mongo::OperationContext*, mongo::repl::SyncTail::OpQueue const&, mongo::OldThreadPool*, mongo::OldThreadPool*, std::function<void (std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> > const&, mongo::repl::SyncTail*)>, mongo::repl::SyncTail*, bool) /home/s/code/mongo/mongo/src/mongo/db/repl/sync_tail.cpp:378
          #12 0x1eb9f24 in mongo::repl::SyncTail::oplogApplication() /home/s/code/mongo/mongo/src/mongo/db/repl/sync_tail.cpp:572
          #13 0x1eae65c in mongo::repl::runSyncThread() /home/s/code/mongo/mongo/src/mongo/db/repl/rs_sync.cpp:133
          #14 0x1dab3bd in boost::detail::thread_data<void (*)()>::run() /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/boost/thread/detail/thread.hpp:115
          #15 0x241311b in thread_proxy /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/libs/thread/src/pthread/thread.cpp:173
          #16 0x7f532724ab97 (/usr/lib/x86_64-linux-gnu/libasan.so.0+0x18b97)
      
      Thread T27 created by T17 here:
          #0 0x7f532723cb5b in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.0+0xab5b)
          #1 0x2413315 in boost::thread::start_thread_noexcept() /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/libs/thread/src/pthread/thread.cpp:255
          #2 0x15b8ba0 in boost::thread::start_thread() /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/boost/thread/detail/thread.hpp:178
          #3 0x1da50cd in boost::thread::thread<void (&)()>(void (&)()) /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/boost/thread/detail/thread.hpp:265
          #4 0x1dfb35b in mongo::repl::ReplicationCoordinatorExternalStateImpl::startThreads() /home/s/code/mongo/mongo/src/mongo/db/repl/replication_coordinator_external_state_impl.cpp:100
          #5 0x1e66403 in mongo::repl::ReplicationCoordinatorImpl::_heartbeatReconfigStore(mongo::executor::TaskExecutor::CallbackArgs const&, mongo::repl::ReplicaSetConfig const&) /home/s/code/mongo/mongo/src/mongo/db/repl/replication_coordinator_impl_heartbeat.cpp:392
          #6 0x1e6f72f in void std::_Mem_fn<void (mongo::repl::ReplicationCoordinatorImpl::*)(mongo::executor::TaskExecutor::CallbackArgs const&, mongo::repl::ReplicaSetConfig const&)>::operator()<mongo::executor::TaskExecutor::CallbackArgs const&, mongo::repl::ReplicaSetConfig&, void>(mongo::repl::ReplicationCoordinatorImpl*, mongo::executor::TaskExecutor::CallbackArgs const&, mongo::repl::ReplicaSetConfig&) const /usr/include/c++/4.8/functional:601
          #7 0x1e6eb9d in void std::_Bind<std::_Mem_fn<void (mongo::repl::ReplicationCoordinatorImpl::*)(mongo::executor::TaskExecutor::CallbackArgs const&, mongo::repl::ReplicaSetConfig const&)> (mongo::repl::ReplicationCoordinatorImpl*, std::_Placeholder<1>, mongo::repl::ReplicaSetConfig)>::__call<void, mongo::executor::TaskExecutor::CallbackArgs const&, 0ul, 1ul, 2ul>(std::tuple<mongo::executor::TaskExecutor::CallbackArgs const&>&&, std::_Index_tuple<0ul, 1ul, 2ul>) /usr/include/c++/4.8/functional:1296
          #8 0x1e6d668 in void std::_Bind<std::_Mem_fn<void (mongo::repl::ReplicationCoordinatorImpl::*)(mongo::executor::TaskExecutor::CallbackArgs const&, mongo::repl::ReplicaSetConfig const&)> (mongo::repl::ReplicationCoordinatorImpl*, std::_Placeholder<1>, mongo::repl::ReplicaSetConfig)>::operator()<mongo::executor::TaskExecutor::CallbackArgs const&, void>(mongo::executor::TaskExecutor::CallbackArgs const&) /usr/include/c++/4.8/functional:1355
          #9 0x1e6ae0b in std::_Function_handler<void (mongo::executor::TaskExecutor::CallbackArgs const&), std::_Bind<std::_Mem_fn<void (mongo::repl::ReplicationCoordinatorImpl::*)(mongo::executor::TaskExecutor::CallbackArgs const&, mongo::repl::ReplicaSetConfig const&)> (mongo::repl::ReplicationCoordinatorImpl*, std::_Placeholder<1>, mongo::repl::ReplicaSetConfig)> >::_M_invoke(std::_Any_data const&, mongo::executor::TaskExecutor::CallbackArgs const&) /usr/include/c++/4.8/functional:2071
          #10 0x1e7bade in std::function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>::operator()(mongo::executor::TaskExecutor::CallbackArgs const&) const /usr/include/c++/4.8/functional:2464
          #11 0x1e7432a in mongo::repl::ReplicationExecutor::_doOperation(mongo::OperationContext*, mongo::Status const&, mongo::executor::TaskExecutor::CallbackHandle const&, std::list<mongo::repl::ReplicationExecutor::WorkItem, std::allocator<mongo::repl::ReplicationExecutor::WorkItem> >*, boost::mutex*) /home/s/code/mongo/mongo/src/mongo/db/repl/replication_executor.cpp:392
          #12 0x1e858f7 in void std::_Mem_fn<void (mongo::repl::ReplicationExecutor::*)(mongo::OperationContext*, mongo::Status const&, mongo::executor::TaskExecutor::CallbackHandle const&, std::list<mongo::repl::ReplicationExecutor::WorkItem, std::allocator<mongo::repl::ReplicationExecutor::WorkItem> >*, boost::mutex*)>::operator()<mongo::OperationContext*&, mongo::Status&, mongo::executor::TaskExecutor::CallbackHandle&, std::list<mongo::repl::ReplicationExecutor::WorkItem, std::allocator<mongo::repl::ReplicationExecutor::WorkItem> >*&, decltype(nullptr)&, void>(mongo::repl::ReplicationExecutor*, mongo::OperationContext*&, mongo::Status&, mongo::executor::TaskExecutor::CallbackHandle&, std::list<mongo::repl::ReplicationExecutor::WorkItem, std::allocator<mongo::repl::ReplicationExecutor::WorkItem> >*&, decltype(nullptr)&) const /usr/include/c++/4.8/functional:601
          #13 0x1e8521a in void std::_Bind<std::_Mem_fn<void (mongo::repl::ReplicationExecutor::*)(mongo::OperationContext*, mongo::Status const&, mongo::executor::TaskExecutor::CallbackHandle const&, std::list<mongo::repl::ReplicationExecutor::WorkItem, std::allocator<mongo::repl::ReplicationExecutor::WorkItem> >*, boost::mutex*)> (mongo::repl::ReplicationExecutor*, std::_Placeholder<1>, std::_Placeholder<2>, mongo::executor::TaskExecutor::CallbackHandle, std::list<mongo::repl::ReplicationExecutor::WorkItem, std::allocator<mongo::repl::ReplicationExecutor::WorkItem> >*, decltype(nullptr))>::__call<void, mongo::OperationContext*&, mongo::Status&, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul>(std::tuple<mongo::OperationContext*&, mongo::Status&>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul, 4ul, 5ul>) /usr/include/c++/4.8/functional:1296
          #14 0x1e84ac3 in void std::_Bind<std::_Mem_fn<void (mongo::repl::ReplicationExecutor::*)(mongo::OperationContext*, mongo::Status const&, mongo::executor::TaskExecutor::CallbackHandle const&, std::list<mongo::repl::ReplicationExecutor::WorkItem, std::allocator<mongo::repl::ReplicationExecutor::WorkItem> >*, boost::mutex*)> (mongo::repl::ReplicationExecutor*, std::_Placeholder<1>, std::_Placeholder<2>, mongo::executor::TaskExecutor::CallbackHandle, std::list<mongo::repl::ReplicationExecutor::WorkItem, std::allocator<mongo::repl::ReplicationExecutor::WorkItem> >*, decltype(nullptr))>::operator()<mongo::OperationContext*&, mongo::Status&, void>(mongo::OperationContext*&, mongo::Status&) /usr/include/c++/4.8/functional:1355
          #15 0x1e83b6e in void std::_Bind<std::_Bind<std::_Mem_fn<void (mongo::repl::ReplicationExecutor::*)(mongo::OperationContext*, mongo::Status const&, mongo::executor::TaskExecutor::CallbackHandle const&, std::list<mongo::repl::ReplicationExecutor::WorkItem, std::allocator<mongo::repl::ReplicationExecutor::WorkItem> >*, boost::mutex*)> (mongo::repl::ReplicationExecutor*, std::_Placeholder<1>, std::_Placeholder<2>, mongo::executor::TaskExecutor::CallbackHandle, std::list<mongo::repl::ReplicationExecutor::WorkItem, std::allocator<mongo::repl::ReplicationExecutor::WorkItem> >*, decltype(nullptr))> (mongo::OperationContext*, mongo::Status)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /usr/include/c++/4.8/functional:1296
          #16 0x1e81dcd in void std::_Bind<std::_Bind<std::_Mem_fn<void (mongo::repl::ReplicationExecutor::*)(mongo::OperationContext*, mongo::Status const&, mongo::executor::TaskExecutor::CallbackHandle const&, std::list<mongo::repl::ReplicationExecutor::WorkItem, std::allocator<mongo::repl::ReplicationExecutor::WorkItem> >*, boost::mutex*)> (mongo::repl::ReplicationExecutor*, std::_Placeholder<1>, std::_Placeholder<2>, mongo::executor::TaskExecutor::CallbackHandle, std::list<mongo::repl::ReplicationExecutor::WorkItem, std::allocator<mongo::repl::ReplicationExecutor::WorkItem> >*, decltype(nullptr))> (mongo::OperationContext*, mongo::Status)>::operator()<, void>() /usr/include/c++/4.8/functional:1355
          #17 0x1e7f4ab in std::_Function_handler<void (), std::_Bind<std::_Bind<std::_Mem_fn<void (mongo::repl::ReplicationExecutor::*)(mongo::OperationContext*, mongo::Status const&, mongo::executor::TaskExecutor::CallbackHandle const&, std::list<mongo::repl::ReplicationExecutor::WorkItem, std::allocator<mongo::repl::ReplicationExecutor::WorkItem> >*, boost::mutex*)> (mongo::repl::ReplicationExecutor*, std::_Placeholder<1>, std::_Placeholder<2>, mongo::executor::TaskExecutor::CallbackHandle, std::list<mongo::repl::ReplicationExecutor::WorkItem, std::allocator<mongo::repl::ReplicationExecutor::WorkItem> >*, decltype(nullptr))> (mongo::OperationContext*, mongo::Status)> >::_M_invoke(std::_Any_data const&) /usr/include/c++/4.8/functional:2071
          #18 0x1d342dd in std::function<void ()>::operator()() const /usr/include/c++/4.8/functional:2464
          #19 0x1e76306 in mongo::repl::(anonymous namespace)::callNoExcept(std::function<void ()> const&) /home/s/code/mongo/mongo/src/mongo/db/repl/replication_executor.cpp:566
          #20 0x1e83dd7 in void std::_Bind<void (*(std::function<void ()>))(std::function<void ()> const&)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /usr/include/c++/4.8/functional:1296
          #21 0x1e82661 in void std::_Bind<void (*(std::function<void ()>))(std::function<void ()> const&)>::operator()<, void>() /usr/include/c++/4.8/functional:1355
          #22 0x1e8002b in std::_Function_handler<void (), std::_Bind<void (*(std::function<void ()>))(std::function<void ()> const&)> >::_M_invoke(std::_Any_data const&) /usr/include/c++/4.8/functional:2071
          #23 0x1d342dd in std::function<void ()>::operator()() const /usr/include/c++/4.8/functional:2464
          #24 0x1e73a28 in mongo::repl::ReplicationExecutor::scheduleDBWork(std::function<void (mongo::executor::TaskExecutor::CallbackArgs const&)> const&, mongo::NamespaceString const&, mongo::LockMode)::{lambda(mongo::OperationContext*, mongo::Status const&)#1}::operator()(mongo::OperationContext*, mongo::Status const&) const /home/s/code/mongo/mongo/src/mongo/db/repl/replication_executor.cpp:356
          #25 0x1e767a5 in std::_Function_handler<mongo::repl::TaskRunner::NextAction (mongo::OperationContext*, mongo::Status const&), mongo::repl::ReplicationExecutor::scheduleDBWork(std::function<void (mongo::executor::TaskExecutor::CallbackArgs const&)> const&, mongo::NamespaceString const&, mongo::LockMode)::{lambda(mongo::OperationContext*, mongo::Status const&)#1}>::_M_invoke(std::_Any_data const&, mongo::OperationContext*, mongo::Status const&) /usr/include/c++/4.8/functional:2057
          #26 0x1d7736e in std::function<mongo::repl::TaskRunner::NextAction (mongo::OperationContext*, mongo::Status const&)>::operator()(mongo::OperationContext*, mongo::Status const&) const /usr/include/c++/4.8/functional:2464
          #27 0x1ec182d in mongo::repl::(anonymous namespace)::runSingleTask(std::function<mongo::repl::TaskRunner::NextAction (mongo::OperationContext*, mongo::Status const&)> const&, mongo::OperationContext*, mongo::Status const&) /home/s/code/mongo/mongo/src/mongo/db/repl/task_runner.cpp:58
          #28 0x1ec26a2 in mongo::repl::TaskRunner::_runTasks() /home/s/code/mongo/mongo/src/mongo/db/repl/task_runner.cpp:145
          #29 0x1ec532c in void std::_Mem_fn<void (mongo::repl::TaskRunner::*)()>::operator()<, void>(mongo::repl::TaskRunner*) const /usr/include/c++/4.8/functional:601
          #30 0x1ec50dd in void std::_Bind<std::_Mem_fn<void (mongo::repl::TaskRunner::*)()> (mongo::repl::TaskRunner*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /usr/include/c++/4.8/functional:1296
          #31 0x1ec4d7f in void std::_Bind<std::_Mem_fn<void (mongo::repl::TaskRunner::*)()> (mongo::repl::TaskRunner*)>::operator()<, void>() /usr/include/c++/4.8/functional:1355
          #32 0x1ec4443 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (mongo::repl::TaskRunner::*)()> (mongo::repl::TaskRunner*)> >::_M_invoke(std::_Any_data const&) /usr/include/c++/4.8/functional:2071
          #33 0x1d342dd in std::function<void ()>::operator()() const /usr/include/c++/4.8/functional:2464
          #34 0x230fea2 in mongo::ThreadPool::_doOneTask(boost::unique_lock<boost::mutex>*) /home/s/code/mongo/mongo/src/mongo/util/concurrency/thread_pool.cpp:324
          #35 0x230f7a9 in mongo::ThreadPool::_consumeTasks() /home/s/code/mongo/mongo/src/mongo/util/concurrency/thread_pool.cpp:276
          #36 0x230f0e7 in mongo::ThreadPool::_workerThreadBody(mongo::ThreadPool*, std::string const&) /home/s/code/mongo/mongo/src/mongo/util/concurrency/thread_pool.cpp:227
          #37 0x2313bec in void std::_Bind<void (*(mongo::ThreadPool*, std::string))(mongo::ThreadPool*, std::string const&)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /usr/include/c++/4.8/functional:1296
          #38 0x2313ae1 in void std::_Bind<void (*(mongo::ThreadPool*, std::string))(mongo::ThreadPool*, std::string const&)>::operator()<, void>() /usr/include/c++/4.8/functional:1355
          #39 0x2313ab7 in boost::detail::thread_data<std::_Bind<void (*(mongo::ThreadPool*, std::string))(mongo::ThreadPool*, std::string const&)> >::run() /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/boost/thread/detail/thread.hpp:115
          #40 0x241311b in thread_proxy /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/libs/thread/src/pthread/thread.cpp:173
          #41 0x7f532724ab97 (/usr/lib/x86_64-linux-gnu/libasan.so.0+0x18b97)
      
      Thread T17 created by T12 here:
          #0 0x7f532723cb5b in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.0+0xab5b)
          #1 0x2413315 in boost::thread::start_thread_noexcept() /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/libs/thread/src/pthread/thread.cpp:255
          #2 0x15b8ba0 in boost::thread::start_thread() /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/boost/thread/detail/thread.hpp:178
          #3 0x2313142 in boost::thread::thread<std::_Bind<void (*(mongo::ThreadPool*, std::string))(mongo::ThreadPool*, std::string const&)> >(std::_Bind<void (*(mongo::ThreadPool*, std::string))(mongo::ThreadPool*, std::string const&)>&&) /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/boost/thread/detail/thread.hpp:265
          #4 0x2312edd in void __gnu_cxx::new_allocator<boost::thread>::construct<boost::thread, std::_Bind<void (*(mongo::ThreadPool*, std::string))(mongo::ThreadPool*, std::string const&)> >(boost::thread*, std::_Bind<void (*(mongo::ThreadPool*, std::string))(mongo::ThreadPool*, std::string const&)>&&) /usr/include/c++/4.8/ext/new_allocator.h:120
          #5 0x2312af8 in _ZNSt16allocator_traitsISaIN5boost6threadEEE12_S_constructIS1_ISt5_BindIFPFvPN5mongo10ThreadPoolERKSsES8_SsEEEEENSt9enable_ifIXsrNS3_18__construct_helperIT_IDpT0_EEE5valueEvE4typeERS2_PSH_DpOSI_ /usr/include/c++/4.8/bits/alloc_traits.h:254
          #6 0x23122dd in decltype (_S_construct({parm#1}, {parm#2}, (forward<std::_Bind<void (*(mongo::ThreadPool*, std::string))(mongo::ThreadPool*, std::string const&)> >)({parm#3}))) std::allocator_traits<std::allocator<boost::thread> >::construct<boost::thread, std::_Bind<void (*(mongo::ThreadPool*, std::string))(mongo::ThreadPool*, std::string const&)> >(std::allocator<boost::thread>&, boost::thread*, std::_Bind<void (*(mongo::ThreadPool*, std::string))(mongo::ThreadPool*, std::string const&)>&&) /usr/include/c++/4.8/bits/alloc_traits.h:393
          #7 0x2312367 in void std::vector<boost::thread, std::allocator<boost::thread> >::_M_emplace_back_aux<std::_Bind<void (*(mongo::ThreadPool*, std::string))(mongo::ThreadPool*, std::string const&)> >(std::_Bind<void (*(mongo::ThreadPool*, std::string))(mongo::ThreadPool*, std::string const&)>&&) /usr/include/c++/4.8/bits/vector.tcc:408
          #8 0x23117f8 in void std::vector<boost::thread, std::allocator<boost::thread> >::emplace_back<std::_Bind<void (*(mongo::ThreadPool*, std::string))(mongo::ThreadPool*, std::string const&)> >(std::_Bind<void (*(mongo::ThreadPool*, std::string))(mongo::ThreadPool*, std::string const&)>&&) /usr/include/c++/4.8/bits/vector.tcc:101
          #9 0x2310743 in mongo::ThreadPool::_startWorkerThread_inlock() /home/s/code/mongo/mongo/src/mongo/util/concurrency/thread_pool.cpp:361
          #10 0x230e0d2 in mongo::ThreadPool::startup() /home/s/code/mongo/mongo/src/mongo/util/concurrency/thread_pool.cpp:106
          #11 0x230ba79 in mongo::OldThreadPool::startThreads() /home/s/code/mongo/mongo/src/mongo/util/concurrency/old_thread_pool.cpp:64
          #12 0x1e70e27 in mongo::repl::ReplicationExecutor::run() /home/s/code/mongo/mongo/src/mongo/db/repl/replication_executor.cpp:101
          #13 0x1e712a9 in mongo::repl::ReplicationExecutor::startup()::{lambda()#1}::operator()() const /home/s/code/mongo/mongo/src/mongo/db/repl/replication_executor.cpp:123
          #14 0x1e7768b in boost::detail::thread_data<mongo::repl::ReplicationExecutor::startup()::{lambda()#1}>::run() /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/boost/thread/detail/thread.hpp:115
          #15 0x241311b in thread_proxy /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/libs/thread/src/pthread/thread.cpp:173
          #16 0x7f532724ab97 (/usr/lib/x86_64-linux-gnu/libasan.so.0+0x18b97)
      
      Thread T12 created by T0 here:
          #0 0x7f532723cb5b in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.0+0xab5b)
          #1 0x2413315 in boost::thread::start_thread_noexcept() /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/libs/thread/src/pthread/thread.cpp:255
          #2 0x15b8ba0 in boost::thread::start_thread() /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/boost/thread/detail/thread.hpp:178
          #3 0x1e7644e in boost::thread::thread<mongo::repl::ReplicationExecutor::startup()::{lambda()#1}>(mongo::repl::ReplicationExecutor::startup()::{lambda()#1}&&) /home/s/code/mongo/mongo/src/third_party/boost-1.56.0/boost/thread/detail/thread.hpp:265
          #4 0x1e7132d in mongo::repl::ReplicationExecutor::startup() /home/s/code/mongo/mongo/src/mongo/db/repl/replication_executor.cpp:123
          #5 0x1e099d2 in mongo::repl::ReplicationCoordinatorImpl::startReplication(mongo::OperationContext*) /home/s/code/mongo/mongo/src/mongo/db/repl/replication_coordinator_impl.cpp:377
          #6 0x15af472 in mongo::_initAndListen(int) /home/s/code/mongo/mongo/src/mongo/db/db.cpp:563
          #7 0x15afb33 in mongo::initAndListen(int) /home/s/code/mongo/mongo/src/mongo/db/db.cpp:594
          #8 0x15b1304 in mongoDbMain(int, char**, char**) /home/s/code/mongo/mongo/src/mongo/db/db.cpp:823
          #9 0x15afedd in main /home/s/code/mongo/mongo/src/mongo/db/db.cpp:639
          #10 0x7f5326044ec4 in __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287
      SUMMARY: AddressSanitizer: heap-use-after-free /home/s/code/mongo/mongo/src/third_party/wiredtiger/src/txn/txn.c:583 __wt_txn_rollback
      

      Version: 6c49d69bbae5d8807fc205dbca12eecf1a60258b

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            kamran.khan Kamran K.
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: