Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-14238

Mongodb server crash after update of replica set member to 2.6

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.6.1
    • Component/s: Storage
    • None
    • ALL

      As per http://docs.mongodb.org/manual/release-notes/2.6-upgrade/ i have followed the instructions for upgradming a replica set to 2.6. I have shut down the server with
      service stop mongodb
      upgraded the mongod binary with apt-get
      started the mongodb
      this is the output:

      2014-06-12T14:22:37.897+0000 ***** SERVER RESTARTED *****
      2014-06-12T14:22:37.902+0000 [initandlisten] MongoDB starting : pid=39502 port=27017 dbpath=/mnt/datadrive/mongodb 64-bit host=vt-mongo-4
      2014-06-12T14:22:37.902+0000 [initandlisten] db version v2.6.1
      2014-06-12T14:22:37.902+0000 [initandlisten] git version: 4b95b086d2374bdcfcdf2249272fb552c9c726e8
      2014-06-12T14:22:37.902+0000 [initandlisten] build info: Linux build14.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
      2014-06-12T14:22:37.902+0000 [initandlisten] allocator: tcmalloc
      2014-06-12T14:22:37.903+0000 [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017 }, replication: { replSet: "vt_replicaset2" }, storage: { dbPath: "/mnt/datadrive/mongodb" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
      2014-06-12T14:22:37.998+0000 [initandlisten] journal dir=/mnt/datadrive/mongodb/journal
      2014-06-12T14:22:37.998+0000 [initandlisten] recover begin
      2014-06-12T14:22:37.998+0000 [initandlisten] info no lsn file in journal/ directory
      2014-06-12T14:22:37.998+0000 [initandlisten] recover lsn: 0
      2014-06-12T14:22:37.998+0000 [initandlisten] recover /mnt/datadrive/mongodb/journal/j._0
      2014-06-12T14:22:37.999+0000 [initandlisten] recover cleaning up
      2014-06-12T14:22:37.999+0000 [initandlisten] removeJournalFiles
      2014-06-12T14:22:38.473+0000 [initandlisten] recover done
      2014-06-12T14:22:38.473+0000 [initandlisten] preallocating a journal file /mnt/datadrive/mongodb/journal/prealloc.0
      2014-06-12T14:23:54.880+0000 [initandlisten] vt Invariant failure save == _entries.find( descriptor->indexName() ) src/mongo/db/catalog/index_catalog.cpp 156
      2014-06-12T14:23:54.901+0000 [initandlisten] vt 0x11c0e91 0x1163109 0x1145978 0x8d2640 0x8d2d5c 0x8aed8c 0x8ba88d 0x761c3e 0x76255f 0x766b1f 0x7673eb 0x767985 0x767ba9 0x7f11f600b76d 0x75eae9 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11c0e91]
       /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x1163109]
       /usr/bin/mongod(_ZN5mongo15invariantFailedEPKcS1_j+0x148) [0x1145978]
       /usr/bin/mongod(_ZN5mongo12IndexCatalog24_setupInMemoryStructuresEPNS_15IndexDescriptorE+0x400) [0x8d2640]
       /usr/bin/mongod(_ZN5mongo12IndexCatalog4initEv+0x58c) [0x8d2d5c]
       /usr/bin/mongod(_ZN5mongo10CollectionC1ERKNS_10StringDataEPNS_16NamespaceDetailsEPNS_8DatabaseE+0x30c) [0x8aed8c]
       /usr/bin/mongod(_ZN5mongo8Database13getCollectionERKNS_10StringDataE+0x16d) [0x8ba88d]
       /usr/bin/mongod(_ZN5mongo17checkForIdIndexesEPNS_8DatabaseE+0x19e) [0x761c3e]
       /usr/bin/mongod() [0x76255f]
       /usr/bin/mongod(_ZN5mongo14_initAndListenEi+0x5df) [0x766b1f]
       /usr/bin/mongod(_ZN5mongo13initAndListenEi+0x1b) [0x7673eb]
       /usr/bin/mongod() [0x767985]
       /usr/bin/mongod(main+0x9) [0x767ba9]
       /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f11f600b76d]
       /usr/bin/mongod() [0x75eae9]
      2014-06-12T14:23:54.902+0000 [initandlisten] 
      
      ***aborting after invariant() failure
      
      
      2014-06-12T14:23:54.922+0000 [initandlisten] SEVERE: Got signal: 6 (Aborted).
      Backtrace:0x11c0e91 0x11c026e 0x7f11f60204a0 0x7f11f6020425 0x7f11f6023b8b 0x11459e5 0x8d2640 0x8d2d5c 0x8aed8c 0x8ba88d 0x761c3e 0x76255f 0x766b1f 0x7673eb 0x767985 0x767ba9 0x7f11f600b76d 0x75eae9 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11c0e91]
       /usr/bin/mongod() [0x11c026e]
       /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f11f60204a0]
       /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f11f6020425]
       /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f11f6023b8b]
       /usr/bin/mongod(_ZN5mongo15invariantFailedEPKcS1_j+0x1b5) [0x11459e5]
       /usr/bin/mongod(_ZN5mongo12IndexCatalog24_setupInMemoryStructuresEPNS_15IndexDescriptorE+0x400) [0x8d2640]
       /usr/bin/mongod(_ZN5mongo12IndexCatalog4initEv+0x58c) [0x8d2d5c]
       /usr/bin/mongod(_ZN5mongo10CollectionC1ERKNS_10StringDataEPNS_16NamespaceDetailsEPNS_8DatabaseE+0x30c) [0x8aed8c]
       /usr/bin/mongod(_ZN5mongo8Database13getCollectionERKNS_10StringDataE+0x16d) [0x8ba88d]
       /usr/bin/mongod(_ZN5mongo17checkForIdIndexesEPNS_8DatabaseE+0x19e) [0x761c3e]
       /usr/bin/mongod() [0x76255f]
       /usr/bin/mongod(_ZN5mongo14_initAndListenEi+0x5df) [0x766b1f]
       /usr/bin/mongod(_ZN5mongo13initAndListenEi+0x1b) [0x7673eb]
       /usr/bin/mongod() [0x767985]
       /usr/bin/mongod(main+0x9) [0x767ba9]
       /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f11f600b76d]
       /usr/bin/mongod() [0x75eae9]
      ^C
      

      After reading this
      http://t193327.db-mongodb-user.dbtalk.us/mongodb-upgrade-problem-t193327.html
      we have rolled back to 2.4 and we have no problems.

            Assignee:
            kaloian.manassiev@mongodb.com Kaloian Manassiev
            Reporter:
            tudor_alexandru Tudor Alexandru
            Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: