Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20304

"Invariant failure details" – unable to start or repair after unclean shutdown

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.0.5
    • Component/s: Storage
    • None
    • Environment:
      See attachment.

      db version v3.0.5
      git version: 8bc4ae20708dbb493cb09338d9e7be6698e4a3a3
    • Windows

      Apparently there was an unclean shutdown in a production system, after which the MongoDB server does not start.

      Trying to run with --repair gives:

      2015-09-06T11:49:15.856+0300 I CONTROL  Hotfix KB2731284 or later update is not installed, will zero-out data files
      2015-09-06T11:49:15.872+0300 W -        [initandlisten] Detected unclean shutdown - C:\Program Files\MongoDB\Server\3.0\db\mongod.lock is not empty.
      2015-09-06T11:49:15.872+0300 I STORAGE  [initandlisten] ************** 
      You specified --repair but there are dirty journal files. Please
      restart without --repair to allow the journal files to be replayed.
      If you wish to repair all databases, please shutdown cleanly and
      run with --repair again.
      **************
      2015-09-06T11:49:15.872+0300 I STORAGE  [initandlisten] exception in initAndListen: 12596 old lock file, terminating
      2015-09-06T11:49:15.872+0300 I CONTROL  [initandlisten] dbexit:  rc: 100
      

      Running without --repair gives:

      2015-09-06T11:49:54.375+0300 I CONTROL  Hotfix KB2731284 or later update is not installed, will zero-out data files
      2015-09-06T11:49:54.375+0300 W -        [initandlisten] Detected unclean shutdown - C:\Program Files\MongoDB\Server\3.0\db\mongod.lock is not empty.
      2015-09-06T11:49:54.390+0300 I JOURNAL  [initandlisten] journal dir=C:\Program Files\MongoDB\Server\3.0\db\journal
      2015-09-06T11:49:54.390+0300 I JOURNAL  [initandlisten] recover begin
      2015-09-06T11:49:54.390+0300 I JOURNAL  [initandlisten] info no lsn file in journal/ directory
      2015-09-06T11:49:54.390+0300 I JOURNAL  [initandlisten] recover lsn: 0
      2015-09-06T11:49:54.390+0300 I JOURNAL  [initandlisten] recover C:\Program Files\MongoDB\Server\3.0\db\journal\j._0
      2015-09-06T11:49:54.390+0300 I JOURNAL  [initandlisten] recover cleaning up
      2015-09-06T11:49:54.390+0300 I JOURNAL  [initandlisten] removeJournalFiles
      2015-09-06T11:49:54.406+0300 I JOURNAL  [initandlisten] recover done
      2015-09-06T11:49:54.432+0300 I JOURNAL  [durability] Durability thread started
      2015-09-06T11:49:54.432+0300 I JOURNAL  [journal writer] Journal writer thread started
      2015-09-06T11:49:54.445+0300 I CONTROL  [initandlisten] MongoDB starting : pid=5952 port=27017 dbpath=C:\Program Files\MongoDB\Server\3.0\db 64-bit host=RockSense
      2015-09-06T11:49:54.445+0300 I CONTROL  [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
      2015-09-06T11:49:54.445+0300 I CONTROL  [initandlisten] db version v3.0.5
      2015-09-06T11:49:54.445+0300 I CONTROL  [initandlisten] git version: 8bc4ae20708dbb493cb09338d9e7be6698e4a3a3
      2015-09-06T11:49:54.445+0300 I CONTROL  [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49
      2015-09-06T11:49:54.445+0300 I CONTROL  [initandlisten] allocator: tcmalloc
      2015-09-06T11:49:54.445+0300 I CONTROL  [initandlisten] options: { config: "C:\Program Files\MongoDB\Server\3.0\bin\mongod.cfg", storage: { dbPath: "C:\Program Files\MongoDB\Server\3.0\db" }, systemLog: { destination: "file", path: "C:\Program Files\MongoDB\Server\3.0\log\mongod.log" } }
      2015-09-06T11:49:54.452+0300 I -        [initandlisten] Invariant failure details src\mongo\db\storage\mmap_v1\mmap_v1_database_catalog_entry.cpp 732
      2015-09-06T11:49:54.965+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\stacktrace_win.cpp(175)                                mongo::printStackTrace+0x43
      2015-09-06T11:49:54.965+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\log.cpp(135)                                           mongo::logContext+0x97
      2015-09-06T11:49:54.965+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\util\assert_util.cpp(147)                                   mongo::invariantFailed+0xf0
      2015-09-06T11:49:54.965+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\storage\mmap_v1\mmap_v1_database_catalog_entry.cpp(732)  mongo::MMAPV1DatabaseCatalogEntry::_insertInCache+0x69
      2015-09-06T11:49:54.965+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\storage\mmap_v1\mmap_v1_database_catalog_entry.cpp(626)  mongo::MMAPV1DatabaseCatalogEntry::_init+0xae0
      2015-09-06T11:49:54.965+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\storage\mmap_v1\mmap_v1_database_catalog_entry.cpp(185)  mongo::MMAPV1DatabaseCatalogEntry::MMAPV1DatabaseCatalogEntry+0x262
      2015-09-06T11:49:54.965+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\storage\mmap_v1\mmap_v1_engine.cpp(272)                  mongo::MMAPV1Engine::getDatabaseCatalogEntry+0x1a4
      2015-09-06T11:49:54.965+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\catalog\database_holder.cpp(128)                         mongo::DatabaseHolder::openDb+0x242
      2015-09-06T11:49:54.965+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(373)                                              mongo::repairDatabasesAndCheckVersion+0x42f
      2015-09-06T11:49:54.965+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(549)                                              mongo::_initAndListen+0xc26
      2015-09-06T11:49:54.965+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(651)                                              mongo::initAndListen+0x27
      2015-09-06T11:49:54.965+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(896)                                              mongoDbMain+0x206
      2015-09-06T11:49:54.965+0300 I CONTROL  [initandlisten] mongod.exe    ...\src\mongo\db\db.cpp(694)                                              wmain+0x35
      2015-09-06T11:49:54.965+0300 I CONTROL  [initandlisten] mongod.exe    f:\dd\vctools\crt\crtw32\dllstuff\crtexe.c(623)                           __tmainCRTStartup+0x10f
      2015-09-06T11:49:54.965+0300 I CONTROL  [initandlisten] kernel32.dll                                                                            BaseThreadInitThunk+0xd
      2015-09-06T11:49:54.965+0300 I CONTROL  [initandlisten] 
      2015-09-06T11:49:54.965+0300 I -        [initandlisten] 
      
      ***aborting after invariant() failure
      

      In addition to bug fixes, any ideas (workarounds) how to fix a production server would be appreciated.

        1. system-information.png
          system-information.png
          26 kB
        2. mongod.cfg
          0.2 kB

            Assignee:
            ramon.fernandez@mongodb.com Ramon Fernandez Marina
            Reporter:
            reunanen Juha Reunanen
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: