Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16678

MongoDB crash due to assert

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.6.4
    • None
    • ALL

      I have a 2.6.4 2+1 replica set . Once of my servers crashed with the following assert

      2014-12-28T04:09:01.075+0000 [rsBackgroundSync] replset setting syncSourceFeedback to 10.0.2.188:27017
      2014-12-28T04:09:01.098+0000 [repl writer worker 1] CMD: drop cove.product_active
      2014-12-28T04:09:01.098+0000 [repl writer worker 1] ERROR: About to fassert -  numIndexesTotal(): 0 numSystemIndexesEntries: 1 _entries.size(): 0 indexNamesToDrop: 1 haveIdIndex: 0
      2014-12-28T04:09:01.098+0000 [repl writer worker 1] cove Fatal Assertion 17328
      2014-12-28T04:09:01.105+0000 [repl writer worker 1] cove 0xf53ad6 0xf01462 0xeea693 0x8c5c55 0x8b3ab6 0x9b242e 0x9ab804 0x9acfec 0x9ade6e 0xcbaf45 0xcfa45f 0xcfa090 0xef827a 0xf8c4cc 0x7f2da12c1f18 0x7f2d9ff86b9d 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xf53ad6]
       /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x1a2) [0xf01462]
       /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xeea693]
       /usr/bin/mongod(_ZN5mongo12IndexCatalog14dropAllIndexesEb+0xb95) [0x8c5c55]
       /usr/bin/mongod(_ZN5mongo8Database14dropCollectionERKNS_10StringDataE+0x326) [0x8b3ab6]
       /usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x38e) [0x9b242e]
       /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9ab804]
       /usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xcdc) [0x9acfec]
       /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x29e) [0x9ade6e]
       /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0xe65) [0xcbaf45]
       /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x2bf) [0xcfa45f]
       /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x50) [0xcfa090]
       /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x43a) [0xef827a]
       /usr/bin/mongod() [0xf8c4cc]
       /lib64/libpthread.so.0(+0x7f18) [0x7f2da12c1f18]
       /lib64/libc.so.6(clone+0x6d) [0x7f2d9ff86b9d]
      2014-12-28T04:09:01.105+0000 [repl writer worker 1] 
      
      ***aborting after fassert() failure
      
      
      2014-12-28T04:09:01.111+0000 [repl writer worker 1] SEVERE: Got signal: 6 (Aborted).
      Backtrace:0xf53ad6 0xf538b0 0x7f2d9fed7c60 0x7f2d9fed7be9 0x7f2d9fed8fe8 0xeea72c 0x8c5c55 0x8b3ab6 0x9b242e 0x9ab804 0x9acfec 0x9ade6e 0xcbaf45 0xcfa45f 0xcfa090 0xef827a 0xf8c4cc 0x7f2da12c1f18 0x7f2d9ff86b9d 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xf53ad6]
       /usr/bin/mongod() [0xf538b0]
       /lib64/libc.so.6(+0x33c60) [0x7f2d9fed7c60]
       /lib64/libc.so.6(gsignal+0x39) [0x7f2d9fed7be9]
       /lib64/libc.so.6(abort+0x148) [0x7f2d9fed8fe8]
       /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x13c) [0xeea72c]
       /usr/bin/mongod(_ZN5mongo12IndexCatalog14dropAllIndexesEb+0xb95) [0x8c5c55]
       /usr/bin/mongod(_ZN5mongo8Database14dropCollectionERKNS_10StringDataE+0x326) [0x8b3ab6]
       /usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x38e) [0x9b242e]
       /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9ab804]
       /usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xcdc) [0x9acfec]
       /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x29e) [0x9ade6e]
       /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0xe65) [0xcbaf45]
       /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x2bf) [0xcfa45f]
       /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x50) [0xcfa090]
       /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x43a) [0xef827a]
       /usr/bin/mongod() [0xf8c4cc]
       /lib64/libpthread.so.0(+0x7f18) [0x7f2da12c1f18]
       /lib64/libc.so.6(clone+0x6d) [0x7f2d9ff86b9d]
      

      The full logs are attached. I also receive the following errors messages on the shell

      2014-12-26T21:59:26.482+0000 [initandlisten] WARNING: the collection 'cove.tmp.mr.product_11239_inc' lacks a unique index on _id. This index is needed for replication to function properly
      2014-12-26T21:59:26.482+0000 [initandlisten] 	 To fix this, you need to create a unique index on _id. See http://dochub.mongodb.org/core/build-replica-set-indexes
      2014-12-26T21:59:26.488+0000 [initandlisten] WARNING: the collection 'cove.tmp.mr.product_11238_inc' lacks a unique index on _id. This index is needed for replication to function properly
      2014-12-26T21:59:26.488+0000 [initandlisten] 	 To fix this, you need to create a unique index on _id. See http://dochub.mongodb.org/core/build-replica-set-indexes
      2014-12-26T21:59:26.489+0000 [initandlisten] WARNING: the collection 'cove.tmp.mr.product_11237_inc' lacks a unique index on _id. This index is needed for replication to function properly
      2014-12-26T21:59:26.489+0000 [initandlisten] 	 To fix this, you need to create a unique index on _id. See http://dochub.mongodb.org/core/build-replica-set-indexes
      2014-12-26T21:59:26.489+0000 [initandlisten] WARNING: the collection 'cove.tmp.mr.product_11240_inc' lacks a unique index on _id. This index is needed for replication to function properly
      2014-12-26T21:59:26.489+0000 [initandlisten] 	 To fix this, you need to create a unique index on _id. See http://dochub.mongodb.org/core/build-replica-set-indexes
      2014-12-26T21:59:26.494+0000 [initandlisten] WARNING: the collection 'cove.tmp.mr.product_11236_inc' lacks a unique index on _id. This index is needed for replication to function properly
      2014-12-26T21:59:26.494+0000 [initandlisten] 	 To fix this, you need to create a unique index on _id. See http://dochub.mongodb.org/core/build-replica-set-indexes
      

      Looks like the two of them are related. Can you suggest how I can proceed?

        1. mongod.log
          26.33 MB

            Assignee:
            ramon.fernandez@mongodb.com Ramon Fernandez Marina
            Reporter:
            dharshanr@scalegrid.net Dharshan Rangegowda
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: