Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-38919

ASAN heap-use-after-free in connection_pool_test

    • Fully Compatible
    • ALL
    • Hide

      On commit 524cc45b29e023c9004f51f8a90c5bb7b0d7c169, build on Ubuntu 18.04 as follows:

      python buildscripts/scons.py --dbg=on --opt=on --variables-files= CC=clang-6.0 CXX=clang++-6.0 --libc++ --disable-warnings-as-errors --sanitize=address --allocator=system -j24 build/optdebug/mongo/executor/connection_pool_test LIBS=ssl
      

      And then run

      export ASAN_SYMBOLIZER_PATH=/usr/lib/llvm-6.0/bin/llvm-symbolizer
      export ASAN_OPTIONS=symbolize=1
      ./build/optdebug/mongo/executor/connection_pool_test
      
      Show
      On commit 524cc45b29e023c9004f51f8a90c5bb7b0d7c169, build on Ubuntu 18.04 as follows: python buildscripts/scons.py --dbg=on --opt=on --variables-files= CC=clang-6.0 CXX=clang++-6.0 --libc++ --disable-warnings-as-errors --sanitize=address --allocator=system -j24 build/optdebug/mongo/executor/connection_pool_test LIBS=ssl And then run export ASAN_SYMBOLIZER_PATH=/usr/lib/llvm-6.0/bin/llvm-symbolizer export ASAN_OPTIONS=symbolize=1 ./build/optdebug/mongo/executor/connection_pool_test
    • Service Arch 2019-02-11, Service Arch 2019-02-25, Service Arch 2019-03-11, Service Arch 2019-03-25, Service Arch 2019-04-08, Service Arch 2019-04-22

      Found by billy.donahue while working on getting our vendored gperftools 2.5 building in C++17 mode.

      When executor/connection_pool_test is built with libc++ and tcmalloc, it reports memory corruption, which was confirmed with ASAN:

      =================================================================
      ==14437==ERROR: AddressSanitizer: heap-use-after-free on address 0x6070000031c0 at pc 0x5557cd67be9f bp 0x7ffd033ff750 sp 0x7ffd033ff748
      WRITE of size 8 at 0x6070000031c0 thread T0
          #0 0x5557cd67be9e in std::__1::function<void ()>::operator=(std::__1::function<void ()>&&) /usr/include/c++/v1/functional:1825:10
          #1 0x5557cd67be9e in mongo::executor::connection_pool_test_details::TimerImpl::cancelTimeout() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:58
          #2 0x5557cd686b7b in mongo::executor::connection_pool_test_details::TimerImpl::clear() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:64:16
          #3 0x5557cd686b7b in mongo::executor::connection_pool_test_details::PoolImpl::shutdown() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.h:166
          #4 0x5557cd6c17d2 in mongo::executor::ConnectionPool::shutdown() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:303:15
          #5 0x5557cd6c1543 in mongo::executor::ConnectionPool::~ConnectionPool() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:299:5
          #6 0x5557cd5f91c1 in mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens::_doTest() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test.cpp:969:1
          #7 0x5557cd6adf1d in mongo::unittest::Test::run() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:188:9
          #8 0x5557cd674288 in mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}::operator()() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:420:21
          #9 0x5557cd674288 in _ZNSt3__18__invokeIRZN5mongo8unittest5Suite3addINS1_8executor28connection_pool_test_details48UnitTest__ConnectionPoolTest__hostTimeoutHappensEEEvRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSI_DpOSJ_ /usr/include/c++/v1/type_traits:4482
          #10 0x5557cd674288 in void std::__1::__invoke_void_return_wrapper<void>::__call<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&>(mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&) /usr/include/c++/v1/__functional_base:349
          #11 0x5557cd674288 in std::__1::__function::__func<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}, std::__1::allocator<{lambda()#1}>, void ()>::operator()() /usr/include/c++/v1/functional:1562
          #12 0x5557cd6b12e3 in std::__1::function<void ()>::operator()() const /usr/include/c++/v1/functional:1916:12
          #13 0x5557cd6b12e3 in mongo::unittest::TestHolder::run() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:285
          #14 0x5557cd6b12e3 in mongo::unittest::Suite::run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:313
          #15 0x5557cd6b53c7 in mongo::unittest::Suite::run(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:373:33
          #16 0x5557cd6925d1 in main /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest_main.cpp:112:12
          #17 0x7fd0bb0e1b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
          #18 0x5557cd4c06d9 in _start (/home/andrew/Documents/10gen/dev/src/mongodb/build/cached/mongo/executor/connection_pool_test+0x2b26d9)
      
      0x6070000031c0 is located 48 bytes inside of 80-byte region [0x607000003190,0x6070000031e0)
      freed by thread T0 here:
          #0 0x5557cd5b9908 in operator delete(void*) (/home/andrew/Documents/10gen/dev/src/mongodb/build/cached/mongo/executor/connection_pool_test+0x3ab908)
          #1 0x5557cd6cb09d in std::__1::__shared_count::__release_shared() /usr/include/c++/v1/memory:3490:9
          #2 0x5557cd6cb09d in std::__1::__shared_weak_count::__release_shared() /usr/include/c++/v1/memory:3532
          #3 0x5557cd6cb09d in std::__1::shared_ptr<mongo::executor::ConnectionPool::TimerInterface>::~shared_ptr() /usr/include/c++/v1/memory:4468
          #4 0x5557cd6cb09d in mongo::executor::ConnectionPool::SpecificPool::~SpecificPool() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:469
          #5 0x5557cd6e0739 in std::__1::default_delete<mongo::executor::ConnectionPool::SpecificPool>::operator()(mongo::executor::ConnectionPool::SpecificPool*) const /usr/include/c++/v1/memory:2285:5
          #6 0x5557cd6e0739 in std::__1::__shared_ptr_pointer<mongo::executor::ConnectionPool::SpecificPool*, std::__1::default_delete<mongo::executor::ConnectionPool::SpecificPool>, std::__1::allocator<mongo::executor::ConnectionPool::SpecificPool> >::__on_zero_shared() /usr/include/c++/v1/memory:3586
          #7 0x5557cd6d83c9 in std::__1::__shared_count::__release_shared() /usr/include/c++/v1/memory:3490:9
          #8 0x5557cd6d83c9 in std::__1::__shared_weak_count::__release_shared() /usr/include/c++/v1/memory:3532
          #9 0x5557cd6d83c9 in std::__1::shared_ptr<mongo::executor::ConnectionPool::SpecificPool>::~shared_ptr() /usr/include/c++/v1/memory:4468
          #10 0x5557cd6d83c9 in mongo::executor::ConnectionPool::SpecificPool::updateStateInLock()::$_11::~$_11() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:946
          #11 0x5557cd6d83c9 in std::__1::__compressed_pair_elem<mongo::executor::ConnectionPool::SpecificPool::updateStateInLock()::$_11, 0, false>::~__compressed_pair_elem() /usr/include/c++/v1/memory:2083
          #12 0x5557cd6d83c9 in std::__1::__function::__func<mongo::executor::ConnectionPool::SpecificPool::updateStateInLock()::$_11, std::__1::allocator<mongo::executor::ConnectionPool::SpecificPool::updateStateInLock()::$_11>, void ()>::destroy() /usr/include/c++/v1/functional:1543
          #13 0x5557cd67bc90 in std::__1::function<void ()>::operator=(std::__1::function<void ()>&&) /usr/include/c++/v1/functional
          #14 0x5557cd67bc90 in mongo::executor::connection_pool_test_details::TimerImpl::cancelTimeout() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:58
          #15 0x5557cd686b7b in mongo::executor::connection_pool_test_details::TimerImpl::clear() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:64:16
          #16 0x5557cd686b7b in mongo::executor::connection_pool_test_details::PoolImpl::shutdown() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.h:166
          #17 0x5557cd6c17d2 in mongo::executor::ConnectionPool::shutdown() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:303:15
          #18 0x5557cd6c1543 in mongo::executor::ConnectionPool::~ConnectionPool() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:299:5
          #19 0x5557cd5f91c1 in mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens::_doTest() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test.cpp:969:1
          #20 0x5557cd6adf1d in mongo::unittest::Test::run() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:188:9
          #21 0x5557cd674288 in mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}::operator()() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:420:21
          #22 0x5557cd674288 in _ZNSt3__18__invokeIRZN5mongo8unittest5Suite3addINS1_8executor28connection_pool_test_details48UnitTest__ConnectionPoolTest__hostTimeoutHappensEEEvRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSI_DpOSJ_ /usr/include/c++/v1/type_traits:4482
          #23 0x5557cd674288 in void std::__1::__invoke_void_return_wrapper<void>::__call<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&>(mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&) /usr/include/c++/v1/__functional_base:349
          #24 0x5557cd674288 in std::__1::__function::__func<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}, std::__1::allocator<{lambda()#1}>, void ()>::operator()() /usr/include/c++/v1/functional:1562
          #25 0x5557cd6b12e3 in std::__1::function<void ()>::operator()() const /usr/include/c++/v1/functional:1916:12
          #26 0x5557cd6b12e3 in mongo::unittest::TestHolder::run() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:285
          #27 0x5557cd6b12e3 in mongo::unittest::Suite::run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:313
          #28 0x5557cd6b53c7 in mongo::unittest::Suite::run(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:373:33
          #29 0x5557cd6925d1 in main /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest_main.cpp:112:12
          #30 0x7fd0bb0e1b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
      
      previously allocated by thread T0 here:
          #0 0x5557cd5b8b90 in operator new(unsigned long) (/home/andrew/Documents/10gen/dev/src/mongodb/build/cached/mongo/executor/connection_pool_test+0x3aab90)
          #1 0x5557cd68473a in std::__1::__unique_if<mongo::executor::connection_pool_test_details::TimerImpl>::__unique_single std::__1::make_unique<mongo::executor::connection_pool_test_details::TimerImpl, mongo::executor::connection_pool_test_details::PoolImpl*>(mongo::executor::connection_pool_test_details::PoolImpl*&&) /usr/include/c++/v1/memory:3078:28
          #2 0x5557cd68473a in mongo::executor::connection_pool_test_details::PoolImpl::makeTimer() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:246
          #3 0x5557cd6ca6c7 in mongo::executor::ConnectionPool::SpecificPool::SpecificPool(mongo::executor::ConnectionPool*, mongo::HostAndPort const&, mongo::transport::ConnectSSLMode) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:456:39
          #4 0x5557cd6c531b in std::__1::__unique_if<mongo::executor::ConnectionPool::SpecificPool>::__unique_single std::__1::make_unique<mongo::executor::ConnectionPool::SpecificPool, mongo::executor::ConnectionPool*, mongo::HostAndPort const&, mongo::transport::ConnectSSLMode&>(mongo::executor::ConnectionPool*&&, mongo::HostAndPort const&, mongo::transport::ConnectSSLMode&) /usr/include/c++/v1/memory:3078:32
          #5 0x5557cd6c531b in mongo::executor::ConnectionPool::get(mongo::HostAndPort const&, mongo::transport::ConnectSSLMode, mongo::Duration<std::__1::ratio<1l, 1000l> >) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:399
          #6 0x5557cd6c4d65 in mongo::executor::ConnectionPool::get_forTest(mongo::HostAndPort const&, mongo::Duration<std::__1::ratio<1l, 1000l> >, std::__1::function<void (mongo::StatusWith<std::__1::unique_ptr<mongo::executor::ConnectionPool::ConnectionInterface, std::__1::function<void (mongo::executor::ConnectionPool::ConnectionInterface*)> > >)>) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:369:12
          #7 0x5557cd5f8745 in mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens::_doTest() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test.cpp:943:10
          #8 0x5557cd6adf1d in mongo::unittest::Test::run() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:188:9
          #9 0x5557cd674288 in mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}::operator()() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:420:21
          #10 0x5557cd674288 in _ZNSt3__18__invokeIRZN5mongo8unittest5Suite3addINS1_8executor28connection_pool_test_details48UnitTest__ConnectionPoolTest__hostTimeoutHappensEEEvRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSI_DpOSJ_ /usr/include/c++/v1/type_traits:4482
          #11 0x5557cd674288 in void std::__1::__invoke_void_return_wrapper<void>::__call<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&>(mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&) /usr/include/c++/v1/__functional_base:349
          #12 0x5557cd674288 in std::__1::__function::__func<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}, std::__1::allocator<{lambda()#1}>, void ()>::operator()() /usr/include/c++/v1/functional:1562
          #13 0x5557cd6b12e3 in std::__1::function<void ()>::operator()() const /usr/include/c++/v1/functional:1916:12
          #14 0x5557cd6b12e3 in mongo::unittest::TestHolder::run() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:285
          #15 0x5557cd6b12e3 in mongo::unittest::Suite::run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:313
          #16 0x5557cd6b53c7 in mongo::unittest::Suite::run(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:373:33
          #17 0x5557cd6925d1 in main /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest_main.cpp:112:12
          #18 0x7fd0bb0e1b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
      
      SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/v1/functional:1825:10 in std::__1::function<void ()>::operator=(std::__1::function<void ()>&&)
      Shadow bytes around the buggy address:
        0x0c0e7fff85e0: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
        0x0c0e7fff85f0: fd fd fd fd fd fd fa fa fa fa fd fd fd fd fd fd
        0x0c0e7fff8600: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
        0x0c0e7fff8610: fd fd fa fa fa fa fd fd fd fd fd fd fd fd fd fd
        0x0c0e7fff8620: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fa fa
      =>0x0c0e7fff8630: fa fa fd fd fd fd fd fd[fd]fd fd fd fa fa fa fa
        0x0c0e7fff8640: 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa
        0x0c0e7fff8650: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c0e7fff8660: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c0e7fff8670: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c0e7fff8680: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      Shadow byte legend (one shadow byte represents 8 application bytes):
        Addressable:           00
        Partially addressable: 01 02 03 04 05 06 07
        Heap left redzone:       fa
        Freed heap region:       fd
        Stack left redzone:      f1
        Stack mid redzone:       f2
        Stack right redzone:     f3
        Stack after return:      f5
        Stack use after scope:   f8
        Global redzone:          f9
        Global init order:       f6
        Poisoned by user:        f7
        Container overflow:      fc
        Array cookie:            ac
        Intra object redzone:    bb
        ASan internal:           fe
        Left alloca redzone:     ca
        Right alloca redzone:    cb
      ==14437==ABORTING
      

            Assignee:
            jesse@mongodb.com A. Jesse Jiryu Davis
            Reporter:
            andrew.morrow@mongodb.com Andrew Morrow (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: