Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-54090

SSLConfiguration use after free when running concurrent migrations

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 5.0.0
    • Affects Version/s: None
    • Component/s: None
    • Fully Compatible
    • ALL
    • Sharding 2021-02-22
    • 59

      One ASAN example:

      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:46.179+0000 d20020| ==113923==ERROR: AddressSanitizer: heap-use-after-free on address 0x607000574ad0 at pc 0x5606994700ea bp 0x7f1e80775650 sp 0x7f1e80774dd8
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:46.179+0000 d20020| READ of size 1 at 0x607000574ad0 thread T373 (TenantM.vice-59)
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.275+0000 d20020|     #0 0x5606994700e9 in strcmp /data/mci/a0a92963d3836bdfa3c974d486c10ee5/toolchain-builder/tmp/build-llvm.sh-DWL/llvm/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:432:3
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.276+0000 d20020|     #1 0x7f1f4da3abb8 in OBJ_bsearch_ex_ (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x185bb8)
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.277+0000 d20020|     #2 0x7f1f4da3b241 in OBJ_ln2nid (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x186241)
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.277+0000 d20020|     #3 0x7f1f4da3b365 in OBJ_txt2obj (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x186365)
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.277+0000 d20020|     #4 0x7f1f4da3b49c in OBJ_txt2nid (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x18649c)
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.301+0000 d20020|     #5 0x5606a226dc9e in mongo::x509OidToShortName[abi:cxx11](mongo::StringData) /data/mci/00a3370018bfacd9240233748dd9166f/src/src/mongo/util/net/ssl_manager.cpp:464:22
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.301+0000 d20020|     #6 0x5606a22709f8 in mongo::operator<<(mongo::StringBuilderImpl<mongo::BufBuilder>&, mongo::SSLX509Name const&) /data/mci/00a3370018bfacd9240233748dd9166f/src/src/mongo/util/net/ssl_manager.cpp:697:27
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.302+0000 d20020|     #7 0x5606a2261f30 in mongo::SSLX509Name::toString[abi:cxx11]() const /data/mci/00a3370018bfacd9240233748dd9166f/src/src/mongo/util/net/ssl_manager.cpp:707:8
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.321+0000 d20020|     #8 0x5606a1cf3af1 in mongo::DBClientBase::_auth(mongo::BSONObj const&) /data/mci/3da64909ec0a2680321efc4ca3c244ba/src/src/mongo/client/dbclient_base.cpp:510:58
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.338+0000 d20020|     #9 0x5606a1276256 in mongo::DBClientConnection::_auth(mongo::BSONObj const&) /data/mci/3da64909ec0a2680321efc4ca3c244ba/src/src/mongo/client/dbclient_connection.cpp:264:19
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.395+0000 d20020|     #10 0x56069b3f5ad7 in mongo::repl::TenantMigrationRecipientService::Instance::_connectAndAuth(mongo::HostAndPort const&, mongo::StringData, mongo::TransientSSLParams const*) /data/mci/285b95a2e8a76dc69eb920a5ed416a5a/src/src/mongo/db/repl/tenant_migration_recipient_service.cpp:316:17
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.396+0000 d20020|     #11 0x56069b437cef in mongo::repl::TenantMigrationRecipientService::Instance::_createAndConnectClients()::$_4::operator()(mongo::HostAndPort const&) const /data/mci/285b95a2e8a76dc69eb920a5ed416a5a/src/src/mongo/db/repl/tenant_migration_recipient_service.cpp:384:27
      ...
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.492+0000 d20020| 0x607000574ad0 is located 16 bytes inside of 72-byte region [0x607000574ac0,0x607000574b08)
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.492+0000 d20020| freed by thread T370 (TenantM.vice-56) here:
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.492+0000 d20020|     #0 0x560699533792 in operator delete(void*) /data/mci/a0a92963d3836bdfa3c974d486c10ee5/toolchain-builder/tmp/build-llvm.sh-DWL/llvm/projects/compiler-rt/lib/asan/asan_new_delete.cc:167:3
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.492+0000 d20020|     #1 0x5606a228a46a in __gnu_cxx::new_allocator<mongo::SSLX509Name::Entry>::deallocate(mongo::SSLX509Name::Entry*, unsigned long) /opt/mongodbtoolchain/revisions/39699409944dd532c7cbdce2f62da11361ac220a/stow/gcc-v3.Me0/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/ext/new_allocator.h:125:2
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.492+0000 d20020|     #2 0x5606a228a46a in std::allocator_traits<std::allocator<mongo::SSLX509Name::Entry> >::deallocate(std::allocator<mongo::SSLX509Name::Entry>&, mongo::SSLX509Name::Entry*, unsigned long) /opt/mongodbtoolchain/revisions/39699409944dd532c7cbdce2f62da11361ac220a/stow/gcc-v3.Me0/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/alloc_traits.h:462
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.492+0000 d20020|     #3 0x5606a228a46a in std::_Vector_base<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >::_M_deallocate(mongo::SSLX509Name::Entry*, unsigned long) /opt/mongodbtoolchain/revisions/39699409944dd532c7cbdce2f62da11361ac220a/stow/gcc-v3.Me0/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/stl_vector.h:304
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.492+0000 d20020|     #4 0x5606a228a46a in std::_Vector_base<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >::~_Vector_base() /opt/mongodbtoolchain/revisions/39699409944dd532c7cbdce2f62da11361ac220a/stow/gcc-v3.Me0/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/stl_vector.h:285
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.493+0000 d20020|     #5 0x5606a228a46a in std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >::~vector() /opt/mongodbtoolchain/revisions/39699409944dd532c7cbdce2f62da11361ac220a/stow/gcc-v3.Me0/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/stl_vector.h:570
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.493+0000 d20020|     #6 0x5606a228a46a in void std::_Destroy<std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> > >(std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >*) /opt/mongodbtoolchain/revisions/39699409944dd532c7cbdce2f62da11361ac220a/stow/gcc-v3.Me0/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/stl_construct.h:98
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.493+0000 d20020|     #7 0x5606a228a46a in void std::_Destroy_aux<false>::__destroy<std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >*>(std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >*, std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >*) /opt/mongodbtoolchain/revisions/39699409944dd532c7cbdce2f62da11361ac220a/stow/gcc-v3.Me0/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/stl_construct.h:108
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.493+0000 d20020|     #8 0x5606a228a46a in void std::_Destroy<std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >*>(std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >*, std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >*) /opt/mongodbtoolchain/revisions/39699409944dd532c7cbdce2f62da11361ac220a/stow/gcc-v3.Me0/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/stl_construct.h:136
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.493+0000 d20020|     #9 0x5606a228a46a in void std::_Destroy<std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >*, std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> > >(std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >*, std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >*, std::allocator<std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> > >&) /opt/mongodbtoolchain/revisions/39699409944dd532c7cbdce2f62da11361ac220a/stow/gcc-v3.Me0/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/stl_construct.h:206
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.493+0000 d20020|     #10 0x5606a228a46a in std::vector<std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >, std::allocator<std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> > > >::~vector() /opt/mongodbtoolchain/revisions/39699409944dd532c7cbdce2f62da11361ac220a/stow/gcc-v3.Me0/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/stl_vector.h:567
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.493+0000 d20020|     #11 0x5606a228a46a in std::vector<std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >, std::allocator<std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> > > >::_M_move_assign(std::vector<std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >, std::allocator<std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> > > >&&, std::integral_constant<bool, true>) /opt/mongodbtoolchain/revisions/39699409944dd532c7cbdce2f62da11361ac220a/stow/gcc-v3.Me0/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/stl_vector.h:1683
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.563+0000 d20020|     #12 0x5606a22f99f8 in std::vector<std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >, std::allocator<std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> > > >::operator=(std::vector<std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> >, std::allocator<std::vector<mongo::SSLX509Name::Entry, std::allocator<mongo::SSLX509Name::Entry> > > >&&) /opt/mongodbtoolchain/revisions/39699409944dd532c7cbdce2f62da11361ac220a/stow/gcc-v3.Me0/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/stl_vector.h:601:2
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.563+0000 d20020|     #13 0x5606a22f99f8 in mongo::SSLX509Name::operator=(mongo::SSLX509Name&&) /data/mci/00a3370018bfacd9240233748dd9166f/src/src/mongo/util/net/ssl_types.h:51
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.563+0000 d20020|     #14 0x5606a22f99f8 in mongo::(anonymous namespace)::SSLManagerOpenSSL::_parseAndValidateCertificateFromBIO(std::unique_ptr<bio_st, mongo::OpenSSLDeleter<int (bio_st*), &(BIO_free)> >, mongo::(anonymous namespace)::SSLManagerOpenSSL::PasswordFetcher*, mongo::StringData, mongo::SSLX509Name*, mongo::Date_t*) /data/mci/00a3370018bfacd9240233748dd9166f/src/src/mongo/util/net/ssl_manager_openssl.cpp:2393
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.563+0000 d20020|     #15 0x5606a22d7003 in mongo::(anonymous namespace)::SSLManagerOpenSSL::_parseAndValidateCertificateFromMemory(mongo::StringData, mongo::(anonymous namespace)::SSLManagerOpenSSL::PasswordFetcher*, mongo::SSLX509Name*, mongo::Date_t*) /data/mci/00a3370018bfacd9240233748dd9166f/src/src/mongo/util/net/ssl_manager_openssl.cpp:2370:12
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.563+0000 d20020|     #16 0x5606a22d7003 in mongo::(anonymous namespace)::SSLManagerOpenSSL::initSSLContext(ssl_ctx_st*, mongo::SSLParams const&, mongo::TransientSSLParams const&, mongo::SSLManagerInterface::ConnectionDirection) /data/mci/00a3370018bfacd9240233748dd9166f/src/src/mongo/util/net/ssl_manager_openssl.cpp:2214
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.637+0000 d20020|     #17 0x5606a15f5e50 in mongo::transport::TransportLayerASIO::_createSSLContext(std::shared_ptr<mongo::SSLManagerInterface>&, mongo::SSLParams::SSLModes, mongo::TransientSSLParams, bool) const /data/mci/3da64909ec0a2680321efc4ca3c244ba/src/src/mongo/transport/transport_layer_asio.cpp:1283:49
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.637+0000 d20020|     #18 0x5606a15f8046 in mongo::transport::TransportLayerASIO::createTransientSSLContext(mongo::TransientSSLParams const&) /data/mci/3da64909ec0a2680321efc4ca3c244ba/src/src/mongo/transport/transport_layer_asio.cpp:1308:12
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.637+0000 d20020|     #19 0x5606a16107f4 in mongo::StatusWith<std::shared_ptr<mongo::transport::TransportLayerASIO::ASIOSession> > mongo::transport::TransportLayerASIO::_doSyncConnect<mongo::transport::WrappedEndpoint>(mongo::transport::WrappedEndpoint, mongo::HostAndPort const&, mongo::Duration<std::ratio<1l, 1000l> > const&, boost::optional<mongo::TransientSSLParams>) /data/mci/3da64909ec0a2680321efc4ca3c244ba/src/src/mongo/transport/transport_layer_asio.cpp:582:36
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.637+0000 d20020|     #20 0x5606a15d8255 in mongo::transport::TransportLayerASIO::connect(mongo::HostAndPort, mongo::transport::ConnectSSLMode, mongo::Duration<std::ratio<1l, 1000l> >, boost::optional<mongo::TransientSSLParams>) /data/mci/3da64909ec0a2680321efc4ca3c244ba/src/src/mongo/transport/transport_layer_asio.cpp:486:16
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.640+0000 d20020|     #21 0x5606a159e5ba in mongo::transport::TransportLayerManager::connect(mongo::HostAndPort, mongo::transport::ConnectSSLMode, mongo::Duration<std::ratio<1l, 1000l> >, boost::optional<mongo::TransientSSLParams>) /data/mci/3da64909ec0a2680321efc4ca3c244ba/src/src/mongo/transport/transport_layer_manager.cpp:69:26
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.640+0000 d20020|     #22 0x5606a12840d1 in mongo::DBClientConnection::connectSocketOnly(mongo::HostAndPort const&, boost::optional<mongo::TransientSSLParams>) /data/mci/3da64909ec0a2680321efc4ca3c244ba/src/src/mongo/client/dbclient_connection.cpp:422:64
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.641+0000 d20020|     #23 0x5606a127742e in mongo::DBClientConnection::connect(mongo::HostAndPort const&, mongo::StringData, boost::optional<mongo::TransientSSLParams>) /data/mci/3da64909ec0a2680321efc4ca3c244ba/src/src/mongo/client/dbclient_connection.cpp:291:26
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.644+0000 d20020|     #24 0x5606a12514f1 in mongo::ConnectionString::connect(mongo::StringData, double, mongo::MongoURI const*, mongo::ClientAPIVersionParameters const*, mongo::TransientSSLParams const*) const /data/mci/3da64909ec0a2680321efc4ca3c244ba/src/src/mongo/client/connection_string_connect.cpp:75:32
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.644+0000 d20020|     #25 0x56069b3f53a0 in mongo::repl::TenantMigrationRecipientService::Instance::_connectAndAuth(mongo::HostAndPort const&, mongo::StringData, mongo::TransientSSLParams const*) /data/mci/285b95a2e8a76dc69eb920a5ed416a5a/src/src/mongo/db/repl/tenant_migration_recipient_service.cpp:294:30
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.644+0000 d20020|     #26 0x56069b437fc8 in mongo::repl::TenantMigrationRecipientService::Instance::_createAndConnectClients()::$_4::operator()(mongo::HostAndPort const&) const /data/mci/285b95a2e8a76dc69eb920a5ed416a5a/src/src/mongo/db/repl/tenant_migration_recipient_service.cpp:392:17
      [js_test:tenant_migration_concurrent_migrations_recipient] 2021-01-27T20:05:49.644+0000 d20020|     #27 0x56069b437fc8 in std::pair<std::unique_ptr<mongo::DBClientConnection, std::default_delete<mongo::DBClientConnection> >, std::unique_ptr<mongo::DBClientConnection, std::default_delete<mongo::DBClientConnection> > > mongo::unique_function<std::pair<std::unique_ptr<mongo::DBClientConnection, std::default_delete<mongo::DBClientConnection> >, std::unique_ptr<mongo::DBClientConnection, std::default_delete<mongo::DBClientConnection> > > (mongo::HostAndPort)>::callRegularVoid<mongo::repl::TenantMigrationRecipientService::Instance::_createAndConnectClients()::$_4>(std::integral_constant<bool, false>, mongo::repl::TenantMigrationRecipientService::Instance::_createAndConnectClients()::$_4&, mongo::HostAndPort&&) /data/mci/285b95a2e8a76dc69eb920a5ed416a5a/src/src/mongo/util/functional.h:150
      

            Assignee:
            andrew.shuvalov@mongodb.com Andrew Shuvalov (Inactive)
            Reporter:
            lingzhi.deng@mongodb.com Lingzhi Deng
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: