Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-27648

Mongo repair failing . Not able to connect to mongo

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.9
    • Component/s: WiredTiger
    • Linux

      2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten] MongoDB starting : pid=31896 port=27017 dbpath=/var/lib/mongodb 64-bit host=vps306768
      2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten] db version v3.2.9
      2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten] git version: 22ec9e93b40c85fc7cae7d56e7d6a02fd811088c
      2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g-fips  1 Mar 2016
      2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten] allocator: tcmalloc
      2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten] modules: none
      2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten] build environment:
      2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten]     distmod: ubuntu1604
      2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten]     distarch: x86_64
      2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten]     target_arch: x86_64
      2017-01-12T06:21:26.938+0100 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb", engine: "wiredTiger" } }
      2017-01-12T06:21:26.966+0100 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
      2017-01-12T06:21:26.966+0100 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=3G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2017-01-12T06:21:26.986+0100 E STORAGE  [initandlisten] WiredTiger (0) [1484198486:986082][31896:0x7f0049b4ec80], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 53248: block header checksum of 2750094563 doesn't match expected checksum of 1661035712
      2017-01-12T06:21:26.986+0100 E STORAGE  [initandlisten] WiredTiger (0) [1484198486:986144][31896:0x7f0049b4ec80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2017-01-12T06:21:26.986+0100 E STORAGE  [initandlisten] WiredTiger (-31804) [1484198486:986158][31896:0x7f0049b4ec80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2017-01-12T06:21:26.986+0100 I -        [initandlisten] Fatal Assertion 28558
      2017-01-12T06:21:26.986+0100 I -        [initandlisten]
      
      ***aborting after fassert() failure
      
      
      2017-01-12T06:21:27.005+0100 F -        [initandlisten] Got signal: 6 (Aborted).
      
       0x1521902 0x15208a9 0x1521112 0x7f004873d3d0 0x7f0048398418 0x7f004839a01a 0x14a3713 0x12480f2 0x1c371b7 0x1c375ae 0x1c379b4 0x1b6421f 0x1b6474b 0x1b613e8 0x1b65827 0x1b83008 0x1bb77d0 0x1c35fcb 0x1c36560 0x1c366dc 0x1bc6bb8 0x1c33208 0x1bfc56f 0x1bfc62c 0x1bb3f2a 0x122cbae 0x1225f85 0x1121898 0x9c0070 0x975802 0x7f0048383830 0x9ba699
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"1121902","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"11208A9"},{"b":"400000","o":"1121112"},{"b":"7F004872C000","o":"113D0"},{"b":"7F0048363000","o":"35418","s":"gsignal"},{"b":"7F0048363000","o":"3701A","s":"abort"},{"b":"400000","o":"10A3713","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"E480F2"},{"b":"400000","o":"18371B7","s":"__wt_eventv"},{"b":"400000","o":"18375AE","s":"__wt_err"},{"b":"400000","o":"18379B4","s":"__wt_panic"},{"b":"400000","o":"176421F","s":"__wt_block_extlist_read"},{"b":"400000","o":"176474B","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"17613E8","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"1765827"},{"b":"400000","o":"1783008","s":"__wt_btree_open"},{"b":"400000","o":"17B77D0","s":"__wt_conn_btree_open"},{"b":"400000","o":"1835FCB","s":"__wt_session_get_btree"},{"b":"400000","o":"1836560","s":"__wt_session_get_btree"},{"b":"400000","o":"18366DC","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"17C6BB8","s":"__wt_curfile_open"},{"b":"400000","o":"1833208"},{"b":"400000","o":"17FC56F","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"17FC62C","s":"__wt_metadata_cursor"},{"b":"400000","o":"17B3F2A","s":"wiredtiger_open"},{"b":"400000","o":"E2CBAE","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"400000","o":"E25F85"},{"b":"400000","o":"D21898","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5C0070","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"575802","s":"main"},{"b":"7F0048363000","o":"20830","s":"__libc_start_main"},{"b":"400000","o":"5BA699","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.2.9", "gitVersion" : "22ec9e93b40c85fc7cae7d56e7d6a02fd811088c", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-22-generic", "version" : "#40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "1F47133C759EFF79F90A6EBF640DC97B744D70C1" }, { "b" : "7FFD695E0000", "elfType" : 3, "buildId" : "0BFAB3B46E2221A329F60F59D05CBFBED1FECA7C" }, { "b" : "7F00496CF000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "822754695E4B31AE82937258BDFF3D52EFA0BA36" }, { "b" : "7F0049274000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "B75A96C59BE1B5B54FBF1A91ED722BEC9406288E" }, { "b" : "7F004906C000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A779DBCB3A477DC0C8D09B60FAC7335D396C19DF" }, { "b" : "7F0048E68000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "A2ADF3615338D49C702C41EB83A99AB743D2B574" }, { "b" : "7F0048B5F000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "5C4078C04888A418F3DB0868702ECFDB35B3AD8B" }, { "b" : "7F0048949000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F004872C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B77847CC9CACBCA3B5753D0D25A32E5795AFE75B" }, { "b" : "7F0048363000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "369DE0E1D833CAA693AF17F17C83BA937F0A4DAD" }, { "b" : "7F0049938000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "EDFA6D46E00CA97F349FDD3333D88493D442932C" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1521902]
       mongod(+0x11208A9) [0x15208a9]
       mongod(+0x1121112) [0x1521112]
       libpthread.so.0(+0x113D0) [0x7f004873d3d0]
       libc.so.6(gsignal+0x38) [0x7f0048398418]
       libc.so.6(abort+0x16A) [0x7f004839a01a]
       mongod(_ZN5mongo13fassertFailedEi+0x93) [0x14a3713]
       mongod(+0xE480F2) [0x12480f2]
       mongod(__wt_eventv+0x4F7) [0x1c371b7]
       mongod(__wt_err+0x8E) [0x1c375ae]
       mongod(__wt_panic+0x24) [0x1c379b4]
       mongod(__wt_block_extlist_read+0x6F) [0x1b6421f]
       mongod(__wt_block_extlist_read_avail+0x2B) [0x1b6474b]
       mongod(__wt_block_checkpoint_load+0x1F8) [0x1b613e8]
       mongod(+0x1765827) [0x1b65827]
       mongod(__wt_btree_open+0xBA8) [0x1b83008]
       mongod(__wt_conn_btree_open+0x140) [0x1bb77d0]
       mongod(__wt_session_get_btree+0xEB) [0x1c35fcb]
       mongod(__wt_session_get_btree+0x680) [0x1c36560]
       mongod(__wt_session_get_btree_ckpt+0x12C) [0x1c366dc]
       mongod(__wt_curfile_open+0x218) [0x1bc6bb8]
       mongod(+0x1833208) [0x1c33208]
       mongod(__wt_metadata_cursor_open+0x5F) [0x1bfc56f]
       mongod(__wt_metadata_cursor+0x3C) [0x1bfc62c]
       mongod(wiredtiger_open+0x158A) [0x1bb3f2a]
       mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0xBCE) [0x122cbae]
       mongod(+0xE25F85) [0x1225f85]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3E8) [0x1121898]
       mongod(_ZN5mongo13initAndListenEi+0x480) [0x9c0070]
       mongod(main+0x712) [0x975802]
       libc.so.6(__libc_start_main+0xF0) [0x7f0048383830]
       mongod(_start+0x29) [0x9ba699]
      -----  END BACKTRACE  -----
      mongorepair.sh: line 5: 31896 Aborted                 mongod --dbpath /var/lib/mongodb --repair --storageEngine wiredTiger
      Mongo Repair Complete
      Starting Mongo Server
      

      I tried to revover data using
      http://www.alexbevi.com/blog/2016/02/10/recovering-a-wiredtiger-collection-from-a-corrupt-mongodb-installation/

      But this is also giving me error while recovery

      Unable to find source-code formatter for language: shell. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      
      root@vps306768:~/wiredtiger-2.9.1# ./wt -v -h ../backup/mongodb -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R salvage collection-8--2945606112955784657.wt
      [1484199812:302917][32152:0x7f757c003700], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 53248: block header checksum of 2750094563 doesn't match expected checksum of 1661035712
      [1484199812:303046][32152:0x7f757c003700], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      [1484199812:303094][32152:0x7f757c003700], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      lt-wt: WT_PANIC: WiredTiger library panic
      

      Attached : WiredTiger.wt and WiredTiger.turtle

        1. WiredTiger.tar.gz
          5 kB
        2. WiredTiger.turtle
          0.9 kB
        3. WiredTiger.wt
          132 kB

            Assignee:
            mark.agarunov Mark Agarunov
            Reporter:
            saurabh73 Saurabh Dutta
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: