Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-47412

Make the database cloner ensure collections are actually unsharded before cloning

XMLWordPrintableJSON

    • Type: Icon: Task Task
    • Resolution: Won't Fix
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Sharding
    • Sharding 2020-04-20

      +underlined text+The operation observers are being used, among other reasons, to maintain internal statistics on the collection and database usage, which can be used, for example, to detect hotspots.

      Changing getCollectionDescription_DEPRECATED for the intended getCollectionDescription that now throws on SSV on the sharding operation observers might cause a backtrace similar to the one below:

       src/mongo/util/stacktrace_posix.cpp:253:21: LibunwindStepIteration
 src/mongo/util/stacktrace_posix.cpp:424:36: mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) (.constprop.587)
 src/mongo/util/stacktrace_posix.cpp:477:44: mongo::printStackTrace()
 src/mongo/util/assert_util.cpp:81:24: mongo::DBException::traceIfNeeded(mongo::DBException const&)
 src/mongo/util/assert_util.h:134:22: mongo::DBException::DBException(mongo::Status const&)
 src/mongo/util/assert_util.h:151:66: _ZN5mongo18AssertionExceptionC4ERKNS_6StatusE
 src/mongo/util/assert_util.h:182:71: mongo::error_details::ExceptionForImpl<(mongo::ErrorCodes::Error)13388, mongo::ExceptionForCat<(mongo::ErrorCategory)3>, mongo::ExceptionForCat<(mongo::ErrorCategory)4> >::ExceptionForImpl(mongo::Status const&)
 build/optdebug/mongo/base/error_codes.cpp:2393:63: mongo::error_details::throwExceptionForStatus(mongo::Status const&)
 src/mongo/util/assert_util.cpp:257:43: mongo::uassertedWithLocation(mongo::Status const&, char const*, unsigned int)
 src/mongo/db/s/collection_sharding_runtime.cpp:151:5: operator()
 src/mongo/db/s/collection_sharding_runtime.cpp:151:5: mongo::CollectionShardingRuntime::getCollectionDescription() (.cold.960)
 src/mongo/db/s/shard_server_op_observer.cpp:241:57: mongo::ShardServerOpObserver::onInserts(mongo::OperationContext*, mongo::NamespaceString const&, boost::optional<mongo::UUID>, __gnu_cxx::__normal_iterator<mongo::InsertStatement const*, std::vector<mongo::InsertStatement, std::allocator<mongo::InsertStatement> > >, __gnu_cxx::__normal_iterator<mongo::InsertStatement const*, std::vector<mongo::InsertStatement, std::allocator<mongo::InsertStatement> > >, bool)
 src/mongo/s/server.cpp:370:52: mongo::(anonymous namespace)::initializeSharding(mongo::OperationContext*)
 src/mongo/db/catalog/collection_impl.cpp:491:58: mongo::CollectionImpl::insertDocuments(mongo::OperationContext*, __gnu_cxx::__normal_iterator<mongo::InsertStatement const*, std::vector<mongo::InsertStatement, std::allocator<mongo::InsertStatement> > >, __gnu_cxx::__normal_iterator<mongo::InsertStatement const*, std::vector<mongo::InsertStatement, std::allocator<mongo::InsertStatement> > >, mongo::OpDebug*, bool)
 src/mongo/db/catalog/collection_impl.cpp:531:81: mongo::CollectionImpl::insertDocument(mongo::OperationContext*, mongo::InsertStatement const&, mongo::OpDebug*, bool)
 src/mongo/db/cloner.cpp:240:94: mongo::Cloner::Fun::operator()(mongo::DBClientCursorBatchIterator&)::'lambda8'()::operator()() const
 src/mongo/db/concurrency/write_conflict_exception.h:94:21: auto mongo::writeConflictRetry<mongo::Cloner::Fun::operator()(mongo::DBClientCursorBatchIterator&)::'lambda8'()>(mongo::OperationContext*, mongo::StringData, mongo::StringData, mongo::Cloner::Fun::operator()(mongo::DBClientCursorBatchIterator&)::'lambda8'()&&)
 src/mongo/db/cloner.cpp:232:31: mongo::Cloner::Fun::operator()(mongo::DBClientCursorBatchIterator&)
 /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:687:14: std::function<void (mongo::DBClientCursorBatchIterator&)>::operator()(mongo::DBClientCursorBatchIterator&) const
 src/mongo/client/dbclient_base.cpp:824:10: mongo::DBClientBase::query(std::function<void (mongo::DBClientCursorBatchIterator&)>, mongo::NamespaceStringOrUUID const&, mongo::Query, mongo::BSONObj const*, int, int, boost::optional<mongo::BSONObj>)
 src/mongo/db/cloner.cpp:311:21: mongo::Cloner::copy(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::NamespaceString const&, mongo::BSONObj const&, mongo::BSONObj const&, mongo::NamespaceString const&, mongo::CloneOptions const&, mongo::Query)
 src/mongo/db/cloner.cpp:839:17: mongo::Cloner::copyDb(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::CloneOptions const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*, std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> >)
 src/mongo/db/s/clone_catalog_data_command.cpp:136:9: mongo::(anonymous namespace)::CloneCatalogDataCommand::run(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, mongo::BSONObjBuilder&)
 src/mongo/db/commands.h:799:19: mongo::BasicCommand::runWithReplyBuilder(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, mongo::rpc::ReplyBuilderInterface*)
 src/mongo/db/commands.cpp:756:48: mongo::BasicCommandWithReplyBuilderInterface::Invocation::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*)
 src/mongo/db/commands.cpp:184:20: mongo::CommandHelpers::runCommandInvocation(mongo::OperationContext*, mongo::OpMsgRequest const&, mongo::CommandInvocation*, mongo::rpc::ReplyBuilderInterface*)
 src/mongo/db/service_entry_point_common.cpp:829:53: mongo::(anonymous namespace)::runCommandImpl(mongo::OperationContext*, mongo::CommandInvocation*, mongo::OpMsgRequest const&, mongo::rpc::ReplyBuilderInterface*, mongo::LogicalTime, mongo::ServiceEntryPointCommon::Hooks const&, mongo::BSONObjBuilder*, mongo::OperationSessionInfoFromClient const&)
 src/mongo/db/service_entry_point_common.cpp:1186:32: mongo::(anonymous namespace)::execCommandDatabase(mongo::OperationContext*, mongo::Command*, mongo::OpMsgRequest const&, mongo::rpc::ReplyBuilderInterface*, mongo::ServiceEntryPointCommon::Hooks const&)
 src/mongo/db/service_entry_point_common.cpp:1362:32: operator()
 src/mongo/db/service_entry_point_common.cpp:1382:6: mongo::(anonymous namespace)::receivedCommands(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)
 src/mongo/db/service_entry_point_common.cpp:1686:38: mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)
 src/mongo/db/service_entry_point_mongod.cpp:277:68: mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)
 src/mongo/transport/service_state_machine.cpp:470:72: mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard)
 src/mongo/transport/service_state_machine.cpp:548:32: mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)
 src/mongo/transport/service_state_machine.cpp:592:29: operator()
 /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:297:37: std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::'lambda'()>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:687:14: std::function<void ()>::operator()() const
 src/mongo/transport/service_executor_synchronous.cpp:107:17: mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)
 src/mongo/transport/service_state_machine.cpp:595:80: mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)
 src/mongo/transport/service_state_machine.cpp:373:38: mongo::ServiceStateMachine::_sourceCallback(mongo::Status)
 src/mongo/transport/service_state_machine.cpp:329:24: operator()
 src/mongo/util/future_impl.h:237:16: auto mongo::future_details::call<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::'lambda0'(mongo::StatusWith<mongo::Message>)&, mongo::StatusWith<mongo::Message> >(mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::'lambda0'(mongo::StatusWith<mongo::Message>)&, mongo::StatusWith<mongo::Message>&&)
 src/mongo/util/future_impl.h:851:32: operator()
 src/mongo/util/future_impl.h:1163:27: generalImpl<mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda(mongo::Message&&)>, mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda(mongo::Status&&)>, mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda()> >
 src/mongo/util/future_impl.h:865:14: getAsync<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)> >
 src/mongo/util/future.h:348:9: getAsync<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)> >
 src/mongo/transport/service_state_machine.cpp:324:29: mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)
 src/mongo/transport/service_state_machine.cpp:545:31: mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)
 src/mongo/transport/service_state_machine.cpp:592:29: operator()
 /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:297:37: std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::'lambda'()>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:687:14: std::function<void ()>::operator()() const
 src/mongo/transport/service_executor_synchronous.cpp:124:36: operator()
 /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:297:37: std::_Function_handler<void (), mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)::'lambda0'()>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:687:14: std::function<void ()>::operator()() const
 src/mongo/transport/service_entry_point_utils.cpp:100:14: operator()
 /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:297:37: std::_Function_handler<void (), mongo::launchServiceWorkerThread(std::function<void ()>)::'lambda1'()>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/stow/gcc-v3.DR0/include/c++/8.2.0/bits/std_function.h:687:14: std::function<void ()>::operator()() const
 src/mongo/transport/service_entry_point_utils.cpp:58:15: mongo::(anonymous namespace)::runFunc(void*)
 ??:0:0: start_thread
 /build/glibc-OTsEL5/glibc-2.27/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95:0: clone

      This is caused to the fact that on the observer it is not expected a SSV when inserting data to a collection. This form part of the new assurances of the CollectionShardingState API and is being enforced as part of the project PM-1645.

      The cloner should refresh the filtering metadata and make sure all the collections are unsharded before doing the copy operation.

            Assignee:
            marcos.grillo@mongodb.com Marcos José Grillo Ramirez
            Reporter:
            marcos.grillo@mongodb.com Marcos José Grillo Ramirez
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: