-
Type: Bug
-
Resolution: Fixed
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: Networking
-
Fully Compatible
-
ALL
-
-
Service Arch 2019-02-11, Service Arch 2019-02-25, Service Arch 2019-03-11, Service Arch 2019-03-25, Service Arch 2019-04-08, Service Arch 2019-04-22
Found by billy.donahue while working on getting our vendored gperftools 2.5 building in C++17 mode.
When executor/connection_pool_test is built with libc++ and tcmalloc, it reports memory corruption, which was confirmed with ASAN:
================================================================= ==14437==ERROR: AddressSanitizer: heap-use-after-free on address 0x6070000031c0 at pc 0x5557cd67be9f bp 0x7ffd033ff750 sp 0x7ffd033ff748 WRITE of size 8 at 0x6070000031c0 thread T0 #0 0x5557cd67be9e in std::__1::function<void ()>::operator=(std::__1::function<void ()>&&) /usr/include/c++/v1/functional:1825:10 #1 0x5557cd67be9e in mongo::executor::connection_pool_test_details::TimerImpl::cancelTimeout() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:58 #2 0x5557cd686b7b in mongo::executor::connection_pool_test_details::TimerImpl::clear() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:64:16 #3 0x5557cd686b7b in mongo::executor::connection_pool_test_details::PoolImpl::shutdown() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.h:166 #4 0x5557cd6c17d2 in mongo::executor::ConnectionPool::shutdown() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:303:15 #5 0x5557cd6c1543 in mongo::executor::ConnectionPool::~ConnectionPool() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:299:5 #6 0x5557cd5f91c1 in mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens::_doTest() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test.cpp:969:1 #7 0x5557cd6adf1d in mongo::unittest::Test::run() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:188:9 #8 0x5557cd674288 in mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}::operator()() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:420:21 #9 0x5557cd674288 in _ZNSt3__18__invokeIRZN5mongo8unittest5Suite3addINS1_8executor28connection_pool_test_details48UnitTest__ConnectionPoolTest__hostTimeoutHappensEEEvRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSI_DpOSJ_ /usr/include/c++/v1/type_traits:4482 #10 0x5557cd674288 in void std::__1::__invoke_void_return_wrapper<void>::__call<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&>(mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&) /usr/include/c++/v1/__functional_base:349 #11 0x5557cd674288 in std::__1::__function::__func<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}, std::__1::allocator<{lambda()#1}>, void ()>::operator()() /usr/include/c++/v1/functional:1562 #12 0x5557cd6b12e3 in std::__1::function<void ()>::operator()() const /usr/include/c++/v1/functional:1916:12 #13 0x5557cd6b12e3 in mongo::unittest::TestHolder::run() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:285 #14 0x5557cd6b12e3 in mongo::unittest::Suite::run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:313 #15 0x5557cd6b53c7 in mongo::unittest::Suite::run(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:373:33 #16 0x5557cd6925d1 in main /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest_main.cpp:112:12 #17 0x7fd0bb0e1b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310 #18 0x5557cd4c06d9 in _start (/home/andrew/Documents/10gen/dev/src/mongodb/build/cached/mongo/executor/connection_pool_test+0x2b26d9) 0x6070000031c0 is located 48 bytes inside of 80-byte region [0x607000003190,0x6070000031e0) freed by thread T0 here: #0 0x5557cd5b9908 in operator delete(void*) (/home/andrew/Documents/10gen/dev/src/mongodb/build/cached/mongo/executor/connection_pool_test+0x3ab908) #1 0x5557cd6cb09d in std::__1::__shared_count::__release_shared() /usr/include/c++/v1/memory:3490:9 #2 0x5557cd6cb09d in std::__1::__shared_weak_count::__release_shared() /usr/include/c++/v1/memory:3532 #3 0x5557cd6cb09d in std::__1::shared_ptr<mongo::executor::ConnectionPool::TimerInterface>::~shared_ptr() /usr/include/c++/v1/memory:4468 #4 0x5557cd6cb09d in mongo::executor::ConnectionPool::SpecificPool::~SpecificPool() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:469 #5 0x5557cd6e0739 in std::__1::default_delete<mongo::executor::ConnectionPool::SpecificPool>::operator()(mongo::executor::ConnectionPool::SpecificPool*) const /usr/include/c++/v1/memory:2285:5 #6 0x5557cd6e0739 in std::__1::__shared_ptr_pointer<mongo::executor::ConnectionPool::SpecificPool*, std::__1::default_delete<mongo::executor::ConnectionPool::SpecificPool>, std::__1::allocator<mongo::executor::ConnectionPool::SpecificPool> >::__on_zero_shared() /usr/include/c++/v1/memory:3586 #7 0x5557cd6d83c9 in std::__1::__shared_count::__release_shared() /usr/include/c++/v1/memory:3490:9 #8 0x5557cd6d83c9 in std::__1::__shared_weak_count::__release_shared() /usr/include/c++/v1/memory:3532 #9 0x5557cd6d83c9 in std::__1::shared_ptr<mongo::executor::ConnectionPool::SpecificPool>::~shared_ptr() /usr/include/c++/v1/memory:4468 #10 0x5557cd6d83c9 in mongo::executor::ConnectionPool::SpecificPool::updateStateInLock()::$_11::~$_11() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:946 #11 0x5557cd6d83c9 in std::__1::__compressed_pair_elem<mongo::executor::ConnectionPool::SpecificPool::updateStateInLock()::$_11, 0, false>::~__compressed_pair_elem() /usr/include/c++/v1/memory:2083 #12 0x5557cd6d83c9 in std::__1::__function::__func<mongo::executor::ConnectionPool::SpecificPool::updateStateInLock()::$_11, std::__1::allocator<mongo::executor::ConnectionPool::SpecificPool::updateStateInLock()::$_11>, void ()>::destroy() /usr/include/c++/v1/functional:1543 #13 0x5557cd67bc90 in std::__1::function<void ()>::operator=(std::__1::function<void ()>&&) /usr/include/c++/v1/functional #14 0x5557cd67bc90 in mongo::executor::connection_pool_test_details::TimerImpl::cancelTimeout() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:58 #15 0x5557cd686b7b in mongo::executor::connection_pool_test_details::TimerImpl::clear() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:64:16 #16 0x5557cd686b7b in mongo::executor::connection_pool_test_details::PoolImpl::shutdown() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.h:166 #17 0x5557cd6c17d2 in mongo::executor::ConnectionPool::shutdown() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:303:15 #18 0x5557cd6c1543 in mongo::executor::ConnectionPool::~ConnectionPool() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:299:5 #19 0x5557cd5f91c1 in mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens::_doTest() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test.cpp:969:1 #20 0x5557cd6adf1d in mongo::unittest::Test::run() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:188:9 #21 0x5557cd674288 in mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}::operator()() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:420:21 #22 0x5557cd674288 in _ZNSt3__18__invokeIRZN5mongo8unittest5Suite3addINS1_8executor28connection_pool_test_details48UnitTest__ConnectionPoolTest__hostTimeoutHappensEEEvRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSI_DpOSJ_ /usr/include/c++/v1/type_traits:4482 #23 0x5557cd674288 in void std::__1::__invoke_void_return_wrapper<void>::__call<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&>(mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&) /usr/include/c++/v1/__functional_base:349 #24 0x5557cd674288 in std::__1::__function::__func<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}, std::__1::allocator<{lambda()#1}>, void ()>::operator()() /usr/include/c++/v1/functional:1562 #25 0x5557cd6b12e3 in std::__1::function<void ()>::operator()() const /usr/include/c++/v1/functional:1916:12 #26 0x5557cd6b12e3 in mongo::unittest::TestHolder::run() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:285 #27 0x5557cd6b12e3 in mongo::unittest::Suite::run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:313 #28 0x5557cd6b53c7 in mongo::unittest::Suite::run(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:373:33 #29 0x5557cd6925d1 in main /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest_main.cpp:112:12 #30 0x7fd0bb0e1b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310 previously allocated by thread T0 here: #0 0x5557cd5b8b90 in operator new(unsigned long) (/home/andrew/Documents/10gen/dev/src/mongodb/build/cached/mongo/executor/connection_pool_test+0x3aab90) #1 0x5557cd68473a in std::__1::__unique_if<mongo::executor::connection_pool_test_details::TimerImpl>::__unique_single std::__1::make_unique<mongo::executor::connection_pool_test_details::TimerImpl, mongo::executor::connection_pool_test_details::PoolImpl*>(mongo::executor::connection_pool_test_details::PoolImpl*&&) /usr/include/c++/v1/memory:3078:28 #2 0x5557cd68473a in mongo::executor::connection_pool_test_details::PoolImpl::makeTimer() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:246 #3 0x5557cd6ca6c7 in mongo::executor::ConnectionPool::SpecificPool::SpecificPool(mongo::executor::ConnectionPool*, mongo::HostAndPort const&, mongo::transport::ConnectSSLMode) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:456:39 #4 0x5557cd6c531b in std::__1::__unique_if<mongo::executor::ConnectionPool::SpecificPool>::__unique_single std::__1::make_unique<mongo::executor::ConnectionPool::SpecificPool, mongo::executor::ConnectionPool*, mongo::HostAndPort const&, mongo::transport::ConnectSSLMode&>(mongo::executor::ConnectionPool*&&, mongo::HostAndPort const&, mongo::transport::ConnectSSLMode&) /usr/include/c++/v1/memory:3078:32 #5 0x5557cd6c531b in mongo::executor::ConnectionPool::get(mongo::HostAndPort const&, mongo::transport::ConnectSSLMode, mongo::Duration<std::__1::ratio<1l, 1000l> >) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:399 #6 0x5557cd6c4d65 in mongo::executor::ConnectionPool::get_forTest(mongo::HostAndPort const&, mongo::Duration<std::__1::ratio<1l, 1000l> >, std::__1::function<void (mongo::StatusWith<std::__1::unique_ptr<mongo::executor::ConnectionPool::ConnectionInterface, std::__1::function<void (mongo::executor::ConnectionPool::ConnectionInterface*)> > >)>) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:369:12 #7 0x5557cd5f8745 in mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens::_doTest() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test.cpp:943:10 #8 0x5557cd6adf1d in mongo::unittest::Test::run() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:188:9 #9 0x5557cd674288 in mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}::operator()() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:420:21 #10 0x5557cd674288 in _ZNSt3__18__invokeIRZN5mongo8unittest5Suite3addINS1_8executor28connection_pool_test_details48UnitTest__ConnectionPoolTest__hostTimeoutHappensEEEvRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSI_DpOSJ_ /usr/include/c++/v1/type_traits:4482 #11 0x5557cd674288 in void std::__1::__invoke_void_return_wrapper<void>::__call<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&>(mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&) /usr/include/c++/v1/__functional_base:349 #12 0x5557cd674288 in std::__1::__function::__func<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}, std::__1::allocator<{lambda()#1}>, void ()>::operator()() /usr/include/c++/v1/functional:1562 #13 0x5557cd6b12e3 in std::__1::function<void ()>::operator()() const /usr/include/c++/v1/functional:1916:12 #14 0x5557cd6b12e3 in mongo::unittest::TestHolder::run() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:285 #15 0x5557cd6b12e3 in mongo::unittest::Suite::run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:313 #16 0x5557cd6b53c7 in mongo::unittest::Suite::run(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:373:33 #17 0x5557cd6925d1 in main /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest_main.cpp:112:12 #18 0x7fd0bb0e1b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310 SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/v1/functional:1825:10 in std::__1::function<void ()>::operator=(std::__1::function<void ()>&&) Shadow bytes around the buggy address: 0x0c0e7fff85e0: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd 0x0c0e7fff85f0: fd fd fd fd fd fd fa fa fa fa fd fd fd fd fd fd 0x0c0e7fff8600: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd 0x0c0e7fff8610: fd fd fa fa fa fa fd fd fd fd fd fd fd fd fd fd 0x0c0e7fff8620: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fa fa =>0x0c0e7fff8630: fa fa fd fd fd fd fd fd[fd]fd fd fd fa fa fa fa 0x0c0e7fff8640: 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa 0x0c0e7fff8650: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c0e7fff8660: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c0e7fff8670: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c0e7fff8680: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==14437==ABORTING