Chunk migration attempts to avoid waiting for replication when persisting the range deletion task document. However it may still end up using a write concern which requires waiting for replication due to the presence of getLastErrorDefaults in the replica set config. This leads to an invariant failure in ReplicationCoordinatorImpl::awaitReplication() due to the OperationContext having a logical session checked out.
// It is illegal to wait for write concern with a session checked out, so persist the // range deletion task with an immediately satsifiable write concern and then wait for // majority after yielding the session. migrationutil::persistRangeDeletionTaskLocally( outerOpCtx, recipientDeletionTask, WriteConcernOptions());
[ShardedClusterFixture:job0:shard0:primary] {"t":{"$date":"2023-01-19T20:54:21.510+00:00"},"s":"D2", "c":"REPL", "id":22549, "ctx":"migrateThread","msg":"Waiting for write concern. OpTime: {replOpTime}, write concern: {writeConcern}","attr":{"replOpTime":{"ts":{"$timestamp":{"t":1674161661,"i":576}},"t":1},"writeConcern":{"w":2,"wtimeout":0,"provenance":"getLastErrorDefaults"}}} [ShardedClusterFixture:job0:shard0:primary] {"t":{"$date":"2023-01-19T20:54:21.510+00:00"},"s":"F", "c":"-", "id":23079, "ctx":"migrateThread","msg":"Invariant failure","attr":{"expr":"OperationContextSession::get(opCtx) == nullptr","file":"src/mongo/db/repl/replication_coordinator_impl.cpp","line":1897}}
#0 0x00007f465f176e87 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f465f1787f1 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00005577fd59e947 in mongo::invariantFailed (expr=<optimized out>, expr@entry=0x5577ff7147b8 "OperationContextSession::get(opCtx) == nullptr", file=<optimized out>, file@entry=0x5577ff710b68 "src/mongo/db/repl/replication_coordinator_impl.cpp", line=<optimized out>, line@entry=1897) at src/mongo/util/assert_util.cpp:117 #3 0x00005577fd2b9a07 in mongo::invariantWithLocation<bool> (testOK=<optimized out>, line=1897, file=0x5577ff710b68 "src/mongo/db/repl/replication_coordinator_impl.cpp", expr=0x5577ff7147b8 "OperationContextSession::get(opCtx) == nullptr") at src/mongo/util/invariant.h:69 #4 mongo::repl::ReplicationCoordinatorImpl::awaitReplication (this=0x557802323000, opCtx=0x55780cd00fc0, opTime=..., writeConcern=...) at src/mongo/db/repl/replication_coordinator_impl.cpp:1897 #5 0x00005577fddded87 in mongo::waitForWriteConcern (opCtx=0x55780cd00fc0, replOpTime=..., writeConcern=..., result=result@entry=0x7f4635baa810) at src/mongo/db/write_concern.cpp:354 #6 0x00005577fdaafb86 in mongo::ServiceEntryPointMongod::Hooks::waitForWriteConcern(mongo::OperationContext*, mongo::CommandInvocation const*, mongo::repl::OpTime const&, mongo::BSONObjBuilder&) const::{lambda()#1}::operator()() const (__closure=__closure@entry=0x7f4635baa900) at src/mongo/db/service_entry_point_mongod.cpp:120 #7 0x00005577fdaaff5b in mongo::ServiceEntryPointMongod::Hooks::waitForWriteConcern (this=<optimized out>, opCtx=<optimized out>, invocation=<optimized out>, lastOpBeforeRun=..., commandResponseBuilder=...) at src/mongo/db/service_entry_point_mongod.cpp:127 #8 0x00005577fdabcbb4 in <lambda(auto:45&&)>::operator()<mongo::BSONObjBuilder> (bb=..., __closure=<synthetic pointer>) at src/mongo/db/service_entry_point_common.cpp:776 #9 mongo::(anonymous namespace)::runCommandImpl (opCtx=<optimized out>, invocation=<optimized out>, request=..., replyBuilder=0x5578077a35e0, startOperationTime=..., behaviors=..., extraFieldsBuilder=0x7f4635bab050, sessionOptions=...) at src/mongo/db/service_entry_point_common.cpp:810 #10 0x00005577fdabfeff in mongo::(anonymous namespace)::execCommandDatabase (behaviors=..., replyBuilder=<optimized out>, request=..., command=<optimized out>, opCtx=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:1203 #11 <lambda()>::operator()(void) const (__closure=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:1391 #12 0x00005577fdac0ecb in mongo::(anonymous namespace)::receivedCommands (behaviors=..., message=..., opCtx=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:1316 #13 mongo::ServiceEntryPointCommon::handleRequest (opCtx=0x55780cd00fc0, m=..., behaviors=...) at src/mongo/db/service_entry_point_common.cpp:1740 #14 0x00005577fdaae95c in mongo::ServiceEntryPointMongod::handleRequest (this=<optimized out>, opCtx=<optimized out>, m=...) at src/mongo/db/service_entry_point_mongod.cpp:291 #15 0x00005577fe18e509 in mongo::(anonymous namespace)::loopbackBuildResponse (opCtx=0x55780cd00fc0, lastError=0x7f4635bace90, toSend=...) at src/mongo/db/dbdirectclient.cpp:146 #16 0x00005577fe18ea68 in mongo::DBDirectClient::call (this=<optimized out>, toSend=..., response=..., assertOk=<optimized out>, actualServer=<optimized out>) at src/mongo/db/dbdirectclient.cpp:151 #17 0x00005577fef04252 in mongo::DBClientBase::runCommandWithTarget (this=0x7f4635bace00, request=...) at src/mongo/client/dbclient_base.cpp:221 #18 0x00005577fdbe67ef in mongo::DBClientBase::runCommand (this=this@entry=0x7f4635bace00, request=...) at src/mongo/client/dbclient_base.h:256 #19 0x00005577fdbe9c71 in mongo::PersistentTaskStore<mongo::RangeDeletionTask>::add (this=this@entry=0x7f4635bacf70, opCtx=opCtx@entry=0x55780cd00fc0, task=..., writeConcern=...) at src/mongo/db/s/persistent_task_store.h:64 #20 0x00005577fdbdfdc3 in mongo::migrationutil::persistRangeDeletionTaskLocally (opCtx=0x55780cd00fc0, deletionTask=..., writeConcern=...) at src/mongo/db/s/migration_util.cpp:604 #21 0x00005577fdbcb16a in mongo::MigrationDestinationManager::_migrateDriver (this=0x55780245ada8, outerOpCtx=<optimized out>) at src/mongo/db/s/migration_destination_manager.cpp:1098 #22 0x00005577fdbccf41 in mongo::MigrationDestinationManager::_migrateThread (this=0x55780245ada8) at src/mongo/db/s/migration_destination_manager.cpp:898
- causes
-
SERVER-78116 [v4.4] Chunk migration still attempts to wait for replication with session checked out when getLastErrorDefaults are used in replica set config
- Closed
- is related to
-
SERVER-48641 Deadlock due to the MigrationDestinationManager waiting for write concern with the session checked-out
- Closed
-
SERVER-48689 MigrationDestinationManager waits for thread to join with session checked out
- Closed
- related to
-
SERVER-73107 [v4.4] Audit usage of default constructed WriteConcernOptions() in sharding catalog manipulation
- Closed