-
Type: Task
-
Resolution: Done
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: Replication
-
Fully Compatible
-
Repl 2016-11-21
Steps to repro :
1. start a replica set on 3.2 (PSS)
2. upgrade to 3.4
3. try to downgrade to 3.2
The secondaries fail in the downgrade with an assertion error :
2016-10-20T13:56:55.920-0400 I - [initandlisten] Fatal assertion 28771 NoSuchKey: Missing expected field "ts"
Full trace here :
2016-10-20T13:56:55.919-0400 I STORAGE [initandlisten] Removing path-level multikey information from index { v: 1, key: { _id: 1 }, name: "_id_", ns: "local.replset.minvalid" } and any other indexes on collection 'local.replset.minvalid' 2016-10-20T13:56:55.919-0400 I STORAGE [initandlisten] Done checking collection 'local.replset.minvalid' for indexes with path-level multikey information 2016-10-20T13:56:55.919-0400 I STORAGE [initandlisten] Checking collection 'local.startup_log' for indexes with path-level multikey information... 2016-10-20T13:56:55.919-0400 I STORAGE [initandlisten] Done checking collection 'local.startup_log' for indexes with path-level multikey information 2016-10-20T13:56:55.920-0400 I STORAGE [initandlisten] Checking collection 'local.system.replset' for indexes with path-level multikey information... 2016-10-20T13:56:55.920-0400 I STORAGE [initandlisten] Done checking collection 'local.system.replset' for indexes with path-level multikey information 2016-10-20T13:56:55.920-0400 I - [initandlisten] Fatal assertion 28771 NoSuchKey: Missing expected field "ts" 2016-10-20T13:56:55.928-0400 I CONTROL [initandlisten] 0x10f72493a 0x10f6d6101 0x10f6c4bcc 0x10f30ee5f 0x10f34056c 0x10f3441e5 0x10f346c75 0x10ee19fb6 0x10ee17a13 0x10ee1c0fc 0x10ee179f4 0x3 ----- BEGIN BACKTRACE —— {"backtrace":[{"b":"10EE17000","o":"90D93A","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"},{"b":"10EE17000","o":"8BF101","s":"_ZN5mongo10logContextEPKc"},{"b":"10EE17000","o":"8ADBCC","s":"_ZN5mongo23fassertFailedWithStatusEiRKNS_6StatusE"},{"b":"10EE17000","o":"4F7E5F","s":"_ZN5mongo4repl11getMinValidEPNS_16OperationContextE"},{"b":"10EE17000","o":"52956C","s":"_ZN5mongo4repl39ReplicationCoordinatorExternalStateImpl21cleanUpLastApplyBatchEPNS_16OperationContextE"},{"b":"10EE17000","o":"52D1E5","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl21_startLoadLocalConfigEPNS_16OperationContextE"},{"b":"10EE17000","o":"52FC75","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl16startReplicationEPNS_16OperationContextE"},{"b":"10EE17000","o":"2FB6","s":"_ZN5mongoL14_initAndListenEi"},{"b":"10EE17000","o":"A13","s":"_ZN5mongo13initAndListenEi"},{"b":"10EE17000","o":"50FC","s":"main"},{"b":"10EE17000","o":"9F4","s":"start"},{"b":"0","o":"3"}],"processInfo":{ "mongodbVersion" : "3.2.10", "gitVersion" : "79d9b3ab5ce20f51c272b4411202710a082d0317", "compiledModules" : [], "uname" : { "sysname" : "Darwin", "release" : "14.5.0", "version" : "Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/tmp/mms-automation/test/versions/mongodb-osx-x86_64-3.2.10/bin/mongod", "machType" : 2, "b" : "10EE17000", "vmaddr" : "100000000", "buildId" : "787C57070B50369E8157A670D010EF48" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF8D7A6000", "vmaddr" : "7FFF8D4F3000", "buildId" : "1866C519C5F33D098C17A8F703664521" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF8299B000", "vmaddr" : "7FFF826E8000", "buildId" : "1B9530FD989B3174BB1CBDC159501710" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF875D0000", "vmaddr" : "7FFF8731D000", "buildId" : "45E9A2E799C436B2BEE30C4E11614AD1" }, { "path" : "/usr/lib/system/libcommonCrypto.dyl mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x3A) [0x10f72493a] mongod(_ZN5mongo10logContextEPKc+0x171) [0x10f6d6101] mongod(_ZN5mongo23fassertFailedWithStatusEiRKNS_6StatusE+0xEC) [0x10f6c4bcc] mongod(_ZN5mongo4repl11getMinValidEPNS_16OperationContextE+0x65F) [0x10f30ee5f] mongod(_ZN5mongo4repl39ReplicationCoordinatorExternalStateImpl21cleanUpLastApplyBatchEPNS_16OperationContextE+0x1C) [0x10f34056c] mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl21_startLoadLocalConfigEPNS_16OperationContextE+0x745) [0x10f3441e5] mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl16startReplicationEPNS_16OperationContextE+0x145) [0x10f346c75] mongod(_ZN5mongoL14_initAndListenEi+0x1F66) [0x10ee19fb6] mongod(_ZN5mongo13initAndListenEi+0x13) [0x10ee17a13] mongod(main+0x3FC) [0x10ee1c0fc] mongod(start+0x34) [0x10ee179f4] ??? [0x3] ----- END BACKTRACE ----- 2016-10-20T13:56:55.928-0400 I - [initandlisten]
- is duplicated by
-
SERVER-27006 Arbiter crashes with "Fatal assertion 28771 NoSuchKey" when upgrading a sharded cluster from 3.2.10 -> 3.2.11-rc0 and then downgrading back to 3.2.10
- Closed