-
Type: Bug
-
Resolution: Incomplete
-
Priority: Major - P3
-
None
-
Affects Version/s: 2.6.4
-
Component/s: Index Maintenance, Stability
-
None
-
ALL
I have a 2.6.4 2+1 replica set . Once of my servers crashed with the following assert
2014-12-28T04:09:01.075+0000 [rsBackgroundSync] replset setting syncSourceFeedback to 10.0.2.188:27017 2014-12-28T04:09:01.098+0000 [repl writer worker 1] CMD: drop cove.product_active 2014-12-28T04:09:01.098+0000 [repl writer worker 1] ERROR: About to fassert - numIndexesTotal(): 0 numSystemIndexesEntries: 1 _entries.size(): 0 indexNamesToDrop: 1 haveIdIndex: 0 2014-12-28T04:09:01.098+0000 [repl writer worker 1] cove Fatal Assertion 17328 2014-12-28T04:09:01.105+0000 [repl writer worker 1] cove 0xf53ad6 0xf01462 0xeea693 0x8c5c55 0x8b3ab6 0x9b242e 0x9ab804 0x9acfec 0x9ade6e 0xcbaf45 0xcfa45f 0xcfa090 0xef827a 0xf8c4cc 0x7f2da12c1f18 0x7f2d9ff86b9d /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xf53ad6] /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x1a2) [0xf01462] /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xeea693] /usr/bin/mongod(_ZN5mongo12IndexCatalog14dropAllIndexesEb+0xb95) [0x8c5c55] /usr/bin/mongod(_ZN5mongo8Database14dropCollectionERKNS_10StringDataE+0x326) [0x8b3ab6] /usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x38e) [0x9b242e] /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9ab804] /usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xcdc) [0x9acfec] /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x29e) [0x9ade6e] /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0xe65) [0xcbaf45] /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x2bf) [0xcfa45f] /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x50) [0xcfa090] /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x43a) [0xef827a] /usr/bin/mongod() [0xf8c4cc] /lib64/libpthread.so.0(+0x7f18) [0x7f2da12c1f18] /lib64/libc.so.6(clone+0x6d) [0x7f2d9ff86b9d] 2014-12-28T04:09:01.105+0000 [repl writer worker 1] ***aborting after fassert() failure 2014-12-28T04:09:01.111+0000 [repl writer worker 1] SEVERE: Got signal: 6 (Aborted). Backtrace:0xf53ad6 0xf538b0 0x7f2d9fed7c60 0x7f2d9fed7be9 0x7f2d9fed8fe8 0xeea72c 0x8c5c55 0x8b3ab6 0x9b242e 0x9ab804 0x9acfec 0x9ade6e 0xcbaf45 0xcfa45f 0xcfa090 0xef827a 0xf8c4cc 0x7f2da12c1f18 0x7f2d9ff86b9d /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xf53ad6] /usr/bin/mongod() [0xf538b0] /lib64/libc.so.6(+0x33c60) [0x7f2d9fed7c60] /lib64/libc.so.6(gsignal+0x39) [0x7f2d9fed7be9] /lib64/libc.so.6(abort+0x148) [0x7f2d9fed8fe8] /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x13c) [0xeea72c] /usr/bin/mongod(_ZN5mongo12IndexCatalog14dropAllIndexesEb+0xb95) [0x8c5c55] /usr/bin/mongod(_ZN5mongo8Database14dropCollectionERKNS_10StringDataE+0x326) [0x8b3ab6] /usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x38e) [0x9b242e] /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9ab804] /usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xcdc) [0x9acfec] /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x29e) [0x9ade6e] /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0xe65) [0xcbaf45] /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x2bf) [0xcfa45f] /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x50) [0xcfa090] /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x43a) [0xef827a] /usr/bin/mongod() [0xf8c4cc] /lib64/libpthread.so.0(+0x7f18) [0x7f2da12c1f18] /lib64/libc.so.6(clone+0x6d) [0x7f2d9ff86b9d]
The full logs are attached. I also receive the following errors messages on the shell
2014-12-26T21:59:26.482+0000 [initandlisten] WARNING: the collection 'cove.tmp.mr.product_11239_inc' lacks a unique index on _id. This index is needed for replication to function properly 2014-12-26T21:59:26.482+0000 [initandlisten] To fix this, you need to create a unique index on _id. See http://dochub.mongodb.org/core/build-replica-set-indexes 2014-12-26T21:59:26.488+0000 [initandlisten] WARNING: the collection 'cove.tmp.mr.product_11238_inc' lacks a unique index on _id. This index is needed for replication to function properly 2014-12-26T21:59:26.488+0000 [initandlisten] To fix this, you need to create a unique index on _id. See http://dochub.mongodb.org/core/build-replica-set-indexes 2014-12-26T21:59:26.489+0000 [initandlisten] WARNING: the collection 'cove.tmp.mr.product_11237_inc' lacks a unique index on _id. This index is needed for replication to function properly 2014-12-26T21:59:26.489+0000 [initandlisten] To fix this, you need to create a unique index on _id. See http://dochub.mongodb.org/core/build-replica-set-indexes 2014-12-26T21:59:26.489+0000 [initandlisten] WARNING: the collection 'cove.tmp.mr.product_11240_inc' lacks a unique index on _id. This index is needed for replication to function properly 2014-12-26T21:59:26.489+0000 [initandlisten] To fix this, you need to create a unique index on _id. See http://dochub.mongodb.org/core/build-replica-set-indexes 2014-12-26T21:59:26.494+0000 [initandlisten] WARNING: the collection 'cove.tmp.mr.product_11236_inc' lacks a unique index on _id. This index is needed for replication to function properly 2014-12-26T21:59:26.494+0000 [initandlisten] To fix this, you need to create a unique index on _id. See http://dochub.mongodb.org/core/build-replica-set-indexes
Looks like the two of them are related. Can you suggest how I can proceed?