-
Type: Question
-
Resolution: Duplicate
-
Priority: Major - P3
-
None
-
Affects Version/s: 4.0.2
-
Component/s: None
-
None
Hi, Mongodb Teams !
When I add some index on the primary node shard, like below:
db.mycollection.createIndex({"colunm1":1,"colunm2":1,"colunm3":1},{background: true})
when primary excuted completed,and SECONDARY excuted completed too , then SECONDARY carsh , I collect the log, what the problem caused the crashed ?
Laster I start the SECONDARY is Ok .
2019-11-07T09:41:37.738+0800 F - [conn26083] Invariant failure params.descriptor src/mongo/db/query/stage_builder.cpp 100 2019-11-07T09:41:37.738+0800 F - [conn26083] ***aborting after invariant() failure 2019-11-07T09:41:37.767+0800 F - [conn26082] Invariant failure params.descriptor src/mongo/db/query/stage_builder.cpp 100 2019-11-07T09:41:37.767+0800 F - [conn26082] ***aborting after invariant() failure 2019-11-07T09:41:37.874+0800 F - [conn26083] Got signal: 6 (Aborted). 0x55a7dc251fe1 0x55a7dc2511f9 0x55a7dc2516dd 0x7f9bacfa85d0 0x7f9bacc02207 0x7f9bacc038f8 0x55a7da89391a 0x55a7db0d2875 0x55a7db0d1357 0x55a7db0d084f 0x55a7db0d161f 0x55a7db0d31c7 0x55a7db03ebf2 0x55a7db040387 0x55a7db0c5cc2 0x55a7db0c9544 0x55a7db0ca5f4 0x55a7db0bf117 0x55a7db0bf6c6 0x55a7db0bff15 0x55a7dae8d42a 0x55a7dbcc0e99 0x55a7da93e0cf 0x55a7da940312 0x55a7da942689 0x55a7da9435d1 0x55a7da930bba 0x55a7da93b95a 0x55a7da936617 0x55a7da939e21 0x55a7dbaf3432 0x55a7da93482f 0x55a7da9379c5 0x55a7da935d57 0x55a7da93669d 0x55a7da939e21 0x55a7dbaf3995 0x55a7dc1aafb4 0x7f9bacfa0dd5 0x7f9baccc9ead ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"55A7D9E8F000","o":"23C2FE1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55A7D9E8F000","o":"23C21F9"},{"b":"55A7D9E8F000","o":"23C26DD"},{"b":"7F9BACF99000","o":"F5D0"},{"b":"7F9BACBCC000","o ":"36207","s":"gsignal"},{"b":"7F9BACBCC000","o":"378F8","s":"abort"},{"b":"55A7D9E8F000","o":"A0491A","s":"_ZN5mongo22invariantFailedWithMsgEPKcRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_j"},{"b" :"55A7D9E8F000","o":"1243875","s":"_ZN5mongo11buildStagesEPNS_16OperationContextEPNS_10CollectionERKNS_14CanonicalQueryERKNS_13QuerySolutionEPKNS_17QuerySolutionNodeEPNS_10WorkingSetE"},{"b":"55A7D9E8F000","o": "1242357","s":"_ZN5mongo11buildStagesEPNS_16OperationContextEPNS_10CollectionERKNS_14CanonicalQueryERKNS_13QuerySolutionEPKNS_17QuerySolutionNodeEPNS_10WorkingSetE"},{"b":"55A7D9E8F000","o":"124184F","s":"_ZN5m ongo11buildStagesEPNS_16OperationContextEPNS_10CollectionERKNS_14CanonicalQueryERKNS_13QuerySolutionEPKNS_17QuerySolutionNodeEPNS_10WorkingSetE"},{"b":"55A7D9E8F000","o":"124261F","s":"_ZN5mongo11buildStagesEPN S_16OperationContextEPNS_10CollectionERKNS_14CanonicalQueryERKNS_13QuerySolutionEPKNS_17QuerySolutionNodeEPNS_10WorkingSetE"},{"b":"55A7D9E8F000","o":"12441C7","s":"_ZN5mongo12StageBuilder5buildEPNS_16Operation ContextEPNS_10CollectionERKNS_14CanonicalQueryERKNS_13QuerySolutionEPNS_10WorkingSetEPPNS_9PlanStageE"},{"b":"55A7D9E8F000","o":"11AFBF2","s":"_ZN5mongo15CachedPlanStage6replanEPNS_15PlanYieldPolicyEb"},{"b":"5 5A7D9E8F000","o":"11B1387","s":"_ZN5mongo15CachedPlanStage12pickBestPlanEPNS_15PlanYieldPolicyE"},{"b":"55A7D9E8F000","o":"1236CC2","s":"_ZN5mongo12PlanExecutor12pickBestPlanEPKNS_10CollectionE"},{"b":"55A7D9E8 F000","o":"123A544","s":"_ZN5mongo12PlanExecutor4makeEPNS_16OperationContextESt10unique_ptrINS_10WorkingSetESt14default_deleteIS4_EES3_INS_9PlanStageES5_IS8_EES3_INS_13QuerySolutionES5_ISB_EES3_INS_14CanonicalQ ueryES5_ISE_EEPKNS_10CollectionENS_15NamespaceStringENS0_11YieldPolicyE"},{"b":"55A7D9E8F000","o":"123B5F4","s":"_ZN5mongo12PlanExecutor4makeEPNS_16OperationContextESt10unique_ptrINS_10WorkingSetESt14default_de leteIS4_EES3_INS_9PlanStageES5_IS8_EES3_INS_13QuerySolutionES5_ISB_EES3_INS_14CanonicalQueryES5_ISE_EEPKNS_10CollectionENS0_11YieldPolicyE"},{"b":"55A7D9E8F000","o":"1230117","s":"_ZN5mongo11getExecutorEPNS_16O perationContextEPNS_10CollectionESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS5_EENS_12PlanExecutor11YieldPolicyEm"},{"b":"55A7D9E8F000","o":"12306C6"},{"b":"55A7D9E8F000","o":"1230F15","s":"_ZN5mongo 15getExecutorFindEPNS_16OperationContextEPNS_10CollectionERKNS_15NamespaceStringESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS8_EEm"},{"b":"55A7D9E8F000","o":"FFE42A"},{"b":"55A7D9E8F000","o":"1E31E99 ","s":"_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_19CommandReplyBuilderE"},{"b":"55A7D9E8F000","o":"AAF0CF"},{"b":"55A7D9E8F000","o":"AB1312"},{"b":"55A7D9E8F000","o":"AB3689"},{"b":"55A 7D9E8F000","o":"AB45D1","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE"},{"b":"55A7D9E8F000","o":"AA1BBA","s":"_ZN5mongo23ServiceEntryPointMongod13hand leRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"55A7D9E8F000","o":"AAC95A","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"55A7D9E8F000","o":"AA7617","s":"_ZN5mongo19Serv iceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55A7D9E8F000","o":"AAAE21"},{"b":"55A7D9E8F000","o":"1C64432","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15Ser viceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"55A7D9E8F000","o":"AA582F","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13Schedu leFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"55A7D9E8F000","o":"AA89C5","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"55A7D9E8F000","o":"AA6D57","s":"_ZN5mongo19Servi ceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"55A7D9E8F000","o":"AA769D","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55A7D9E8F000","o":"AAAE21"},{"b":"55A7D9E8F00 0","o":"1C64995"},{"b":"55A7D9E8F000","o":"231BFB4"},{"b":"7F9BACF99000","o":"7DD5"},{"b":"7F9BACBCC000","o":"FDEAD","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.0.2", "gitVersion" : "fc1573ba18aee42f97a 3bb13b67af7d837826b47", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-957.12.2.el7.x86_64", "version" : "#1 SMP Tue May 14 21:24:32 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "55A7D9E8F000", "elfType" : 3, "buildId" : "86DF86886A5B030E7D7D4639DB20F4B807A3DAA4" }, { "b" : "7FFF83EAC000", "elfType" : 3, "buildId" : "D149FABD4DF969958863221F946E26C130432B47" }, { "b" : "7F9BAE3 C6000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "9114859D3C4BEC47A03CA321EE367DCA799638CD" }, { "b" : "7F9BAE1AD000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "C444AE61E7C BB716FD9C18A0B46A7FE8F4FCF3E5" }, { "b" : "7F9BADD4B000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "3593FA778645A59EA272DBBB59D318C60940E792" }, { "b" : "7F9BADAD9000", "path" : "/lib64/lib ssl.so.10", "elfType" : 3, "buildId" : "AEF5E6F2240B55F90E9DF76CFBB8B9D9F5286583" }, { "b" : "7F9BAD8D5000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "357693C8F1F49D93010C4E31529C07CDD2BD3D08" } , { "b" : "7F9BAD6CD000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "EFDE2029C9A4A20BE5B8D8AE7E6551FF9B5755D2" }, { "b" : "7F9BAD3CB000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : " 5B14BE4D749631673523A61074C10959D50F5455" }, { "b" : "7F9BAD1B5000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "179F202998E429AA1215907F6D4C5C1BB9C90136" }, { "b" : "7F9BACF99000", "path" : "/ lib64/libpthread.so.0", "elfType" : 3, "buildId" : "96900CB0FF25B26F2BBDF247DE1408242E4773D8" }, { "b" : "7F9BACBCC000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "EB9F22A3891E5FD3494DFD9ED199E20A E71BB08D" }, { "b" : "7F9BAE62F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "A527FE72908703C5972AE384E78D1850D1881EE7" }, { "b" : "7F9BAC999000", "path" : "/lib64/libidn.so.11", "elf Type" : 3, "buildId" : "2B77BBEFFF65E94F3E0B71A4E89BEB68C4B476C5" }, { "b" : "7F9BAC76F000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "689404B6B895EACFAD65BA16E07E5BF0004F5E0C" }, { "b" : "7F9 BAC51D000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "2E28F6A705F2ECEA8460D4716D5D1C24B5DDA5E4" }, { "b" : "7F9BAC2F6000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "8D0B4010959 C321022DF9CE239277A9D7B34A76A" }, { "b" : "7F9BABFC9000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "F5A64BB37FA3972E545EF459A51310F0AB56FA56" }, { "b" : "7F9BABD99000", "path" : "/lib64/libnssut il3.so", "elfType" : 3, "buildId" : "E0705772325A52C3372FFFB8BDE5F786E2E200D6" }, { "b" : "7F9BABB95000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "084D2194302908913F68B9DCD27DE46FA5B50522" }, { "b" : "7F9BAB990000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "799B28AD9A5460D78376E2C11260F2E858B95DE3" }, { "b" : "7F9BAB752000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : " DE762A28174110911B273E175D54F222B313CFE0" }, { "b" : "7F9BAB505000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "BCC30853830CD911E58700591830DF51ABCBD7BA" }, { "b" : "7F9BAB21C000", "path " : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "45BAB0BB455BDFA960FDA22E4124CF17B67CC930" }, { "b" : "7F9BAAFE9000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "A9B3906192687CC45D483A E3C58C8AF745A6726A" }, { "b" : "7F9BAADE5000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "B4BE1023D9606A88169DF411BF94AF417D7BA1A0" }, { "b" : "7F9BAABD6000", "path" : "/lib64/liblber-2.4.so .2", "elfType" : 3, "buildId" : "3192C56CD451E18EB9F29CB045432BA9C738DD29" }, { "b" : "7F9BAA981000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "F1FADDDE0D21D5F4E2DCADEDD3B85B6E7AAC9883" }, { "b" : "7F9BAA76B000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "B9D5F73428BD6AD68C96986B57BEA3B7CEDB9745" }, { "b" : "7F9BAA55B000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buil dId" : "94B3BCB669126166B77CDCE6092679A6AA2004C8" }, { "b" : "7F9BAA357000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F9BAA13A000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "E2F2017F821DD1B9D307DA1A9B8014F2941AEB7B" }, { "b" : "7F9BA9F13000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "D2DD4DA3FDE1477D2 5BFFF80F3A25FDB541A8179" }, { "b" : "7F9BA9CDC000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "740CAD898E29E1F3B73A323CCEC4A7C88911647F" }, { "b" : "7F9BA9A7A000", "path" : "/lib64/libpcre.so. 1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" }, { "b" : "7F9BA9877000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "B758881F4B6AF6C28C07A1A57713CBD2144628D4" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55a7dc251fe1] mongod(+0x23C21F9) [0x55a7dc2511f9] mongod(+0x23C26DD) [0x55a7dc2516dd] libpthread.so.0(+0xF5D0) [0x7f9bacfa85d0] libc.so.6(gsignal+0x37) [0x7f9bacc02207] libc.so.6(abort+0x148) [0x7f9bacc038f8] mongod(_ZN5mongo22invariantFailedWithMsgEPKcRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES1_j+0x0) [0x55a7da89391a] mongod(_ZN5mongo11buildStagesEPNS_16OperationContextEPNS_10CollectionERKNS_14CanonicalQueryERKNS_13QuerySolutionEPKNS_17QuerySolutionNodeEPNS_10WorkingSetE+0x26A5) [0x55a7db0d2875] mongod(_ZN5mongo11buildStagesEPNS_16OperationContextEPNS_10CollectionERKNS_14CanonicalQueryERKNS_13QuerySolutionEPKNS_17QuerySolutionNodeEPNS_10WorkingSetE+0x1187) [0x55a7db0d1357] mongod(_ZN5mongo11buildStagesEPNS_16OperationContextEPNS_10CollectionERKNS_14CanonicalQueryERKNS_13QuerySolutionEPKNS_17QuerySolutionNodeEPNS_10WorkingSetE+0x67F) [0x55a7db0d084f] mongod(_ZN5mongo11buildStagesEPNS_16OperationContextEPNS_10CollectionERKNS_14CanonicalQueryERKNS_13QuerySolutionEPKNS_17QuerySolutionNodeEPNS_10WorkingSetE+0x144F) [0x55a7db0d161f] mongod(_ZN5mongo12StageBuilder5buildEPNS_16OperationContextEPNS_10CollectionERKNS_14CanonicalQueryERKNS_13QuerySolutionEPNS_10WorkingSetEPPNS_9PlanStageE+0x37) [0x55a7db0d31c7] mongod(_ZN5mongo15CachedPlanStage6replanEPNS_15PlanYieldPolicyEb+0x322) [0x55a7db03ebf2] mongod(_ZN5mongo15CachedPlanStage12pickBestPlanEPNS_15PlanYieldPolicyE+0x6D7) [0x55a7db040387] mongod(_ZN5mongo12PlanExecutor12pickBestPlanEPKNS_10CollectionE+0xD2) [0x55a7db0c5cc2] mongod(_ZN5mongo12PlanExecutor4makeEPNS_16OperationContextESt10unique_ptrINS_10WorkingSetESt14default_deleteIS4_EES3_INS_9PlanStageES5_IS8_EES3_INS_13QuerySolutionES5_ISB_EES3_INS_14CanonicalQueryES5_ISE_EEPKN S_10CollectionENS_15NamespaceStringENS0_11YieldPolicyE+0x404) [0x55a7db0c9544] mongod(_ZN5mongo12PlanExecutor4makeEPNS_16OperationContextESt10unique_ptrINS_10WorkingSetESt14default_deleteIS4_EES3_INS_9PlanStageES5_IS8_EES3_INS_13QuerySolutionES5_ISB_EES3_INS_14CanonicalQueryES5_ISE_EEPKN S_10CollectionENS0_11YieldPolicyE+0xA4) [0x55a7db0ca5f4] mongod(_ZN5mongo11getExecutorEPNS_16OperationContextEPNS_10CollectionESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS5_EENS_12PlanExecutor11YieldPolicyEm+0x117) [0x55a7db0bf117] mongod(+0x12306C6) [0x55a7db0bf6c6] mongod(_ZN5mongo15getExecutorFindEPNS_16OperationContextEPNS_10CollectionERKNS_15NamespaceStringESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS8_EEm+0x105) [0x55a7db0bff15] mongod(+0xFFE42A) [0x55a7dae8d42a] mongod(_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_19CommandReplyBuilderE+0xD9) [0x55a7dbcc0e99] mongod(+0xAAF0CF) [0x55a7da93e0cf] mongod(+0xAB1312) [0x55a7da940312] mongod(+0xAB3689) [0x55a7da942689] mongod(_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE+0x3D1) [0x55a7da9435d1] mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3A) [0x55a7da930bba] mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x55a7da93b95a] mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x55a7da936617] mongod(+0xAAAE21) [0x55a7da939e21] mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x55a7dbaf3432] mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x15F) [0x55a7da93482f] mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF5) [0x55a7da9379c5] mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x357) [0x55a7da935d57] mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x55a7da93669d] mongod(+0xAAAE21) [0x55a7da939e21] mongod(+0x1C64995) [0x55a7dbaf3995] mongod(+0x231BFB4) [0x55a7dc1aafb4] libpthread.so.0(+0x7DD5) [0x7f9bacfa0dd5] libc.so.6(clone+0x6D) [0x7f9baccc9ead] ----- END BACKTRACE -----
- duplicates
-
SERVER-41361 Do not read at lastApplied while already holding the PBWM lock on secondaries
- Closed