Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-27330

WiredTiger.wt corrupt - unable to repair to start mongo

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.8
    • Component/s: WiredTiger
    • Environment:
      linux 14.04 lts
    • Linux

      Mongo server was running fine for months and after a casual command line `reboot` of the linux box I began getting this error and unable to repair or salvage the database. I noticed others have had the same with in their cases they say "unusual shutdowns" and they uploaded their WiredTiger.wt - I've attached files. Thx in advance for any help how to resolve and recover my db!

      admin@jenkins:/data/db$ sudo mongod
      2016-12-07T18:38:56.511-0800 I CONTROL  [initandlisten] MongoDB starting : pid=22869 port=27017 dbpath=/data/db 64-bit host=jenkins
      2016-12-07T18:38:56.512-0800 I CONTROL  [initandlisten] db version v3.2.8
      2016-12-07T18:38:56.512-0800 I CONTROL  [initandlisten] git version: ed70e33130c977bda0024c125b56d159573dbaf0
      2016-12-07T18:38:56.512-0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
      2016-12-07T18:38:56.512-0800 I CONTROL  [initandlisten] allocator: tcmalloc
      2016-12-07T18:38:56.512-0800 I CONTROL  [initandlisten] modules: none
      2016-12-07T18:38:56.512-0800 I CONTROL  [initandlisten] build environment:
      2016-12-07T18:38:56.512-0800 I CONTROL  [initandlisten]     distmod: ubuntu1404
      2016-12-07T18:38:56.512-0800 I CONTROL  [initandlisten]     distarch: x86_64
      2016-12-07T18:38:56.512-0800 I CONTROL  [initandlisten]     target_arch: x86_64
      2016-12-07T18:38:56.512-0800 I CONTROL  [initandlisten] options: {}
      2016-12-07T18:38:56.542-0800 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2016-12-07T18:38:56.542-0800 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2016-12-07T18:38:56.568-0800 E STORAGE  [initandlisten] WiredTiger (0) [1481164736:568070][22869:0x7f760b5c1cc0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 12288: block header checksum of 3539301520 doesn't match expected checksum of 1790700336
      2016-12-07T18:38:56.568-0800 E STORAGE  [initandlisten] WiredTiger (0) [1481164736:568241][22869:0x7f760b5c1cc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2016-12-07T18:38:56.568-0800 E STORAGE  [initandlisten] WiredTiger (-31804) [1481164736:568317][22869:0x7f760b5c1cc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2016-12-07T18:38:56.568-0800 I -        [initandlisten] Fatal Assertion 28558
      2016-12-07T18:38:56.568-0800 I -        [initandlisten] 
      
      ***aborting after fassert() failure
      
      
      2016-12-07T18:38:56.592-0800 F -        [initandlisten] Got signal: 6 (Aborted).
      
       0x131ce72 0x131bfc9 0x131c7d2 0x7f760a23d340 0x7f7609e9ecc9 0x7f7609ea20d8 0x12a6772 0x10a0923 0x1a7e31c 0x1a7e7dd 0x1a7ebc4 0x19affac 0x19b0523 0x19ad4b5 0x19b1569 0x19cdffc 0x1a04270 0x1a7d047 0x1a7d589 0x1a7d6ab 0x1a12858 0x1a7a1d5 0x1a44ccf 0x1a44dce 0x1a00e11 0x10889cf 0x1084c93 0xfadc48 0x9b48fd 0x96e04d 0x7f7609e89ec5 0x9b1037
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"F1CE72","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F1BFC9"},{"b":"400000","o":"F1C7D2"},{"b":"7F760A22D000","o":"10340"},{"b":"7F7609E68000","o":"36CC9","s":"gsignal"},{"b":"7F7609E68000","o":"3A0D8","s":"abort"},{"b":"400000","o":"EA6772","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"CA0923"},{"b":"400000","o":"167E31C","s":"__wt_eventv"},{"b":"400000","o":"167E7DD","s":"__wt_err"},{"b":"400000","o":"167EBC4","s":"__wt_panic"},{"b":"400000","o":"15AFFAC","s":"__wt_block_extlist_read"},{"b":"400000","o":"15B0523","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15AD4B5","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15B1569"},{"b":"400000","o":"15CDFFC","s":"__wt_btree_open"},{"b":"400000","o":"1604270","s":"__wt_conn_btree_open"},{"b":"400000","o":"167D047","s":"__wt_session_get_btree"},{"b":"400000","o":"167D589","s":"__wt_session_get_btree"},{"b":"400000","o":"167D6AB","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"1612858","s":"__wt_curfile_open"},{"b":"400000","o":"167A1D5"},{"b":"400000","o":"1644CCF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"1644DCE","s":"__wt_metadata_cursor"},{"b":"400000","o":"1600E11","s":"wiredtiger_open"},{"b":"400000","o":"C889CF","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C84C93"},{"b":"400000","o":"BADC48","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5B48FD","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"56E04D","s":"main"},{"b":"7F7609E68000","o":"21EC5","s":"__libc_start_main"},{"b":"400000","o":"5B1037"}],"processInfo":{ "mongodbVersion" : "3.2.8", "gitVersion" : "ed70e33130c977bda0024c125b56d159573dbaf0", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-71-generic", "version" : "#114-Ubuntu SMP Tue Dec 1 02:34:22 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "A53FF676E1D627BD1D9B1BF524DEFA13B667EE83" }, { "b" : "7FFF3E09C000", "elfType" : 3, "buildId" : "C2B4C020D20EE489936A4B11AF2C695FD0ECD484" }, { "b" : "7F760B14E000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "D08DD65F97859C71BB2CBBF1043BD968EFE18AAD" }, { "b" : "7F760AD73000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F86FA9FB4ECEB4E06B40DBDF761A4172B70A4229" }, { "b" : "7F760AB6B000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7F760A967000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7F760A661000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7F760A44B000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }, { "b" : "7F760A22D000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7F7609E68000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7F760B3AD000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x131ce72]
       mongod(+0xF1BFC9) [0x131bfc9]
       mongod(+0xF1C7D2) [0x131c7d2]
       libpthread.so.0(+0x10340) [0x7f760a23d340]
       libc.so.6(gsignal+0x39) [0x7f7609e9ecc9]
       libc.so.6(abort+0x148) [0x7f7609ea20d8]
       mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12a6772]
       mongod(+0xCA0923) [0x10a0923]
       mongod(__wt_eventv+0x42C) [0x1a7e31c]
       mongod(__wt_err+0x8D) [0x1a7e7dd]
       mongod(__wt_panic+0x24) [0x1a7ebc4]
       mongod(__wt_block_extlist_read+0x6C) [0x19affac]
       mongod(__wt_block_extlist_read_avail+0x33) [0x19b0523]
       mongod(__wt_block_checkpoint_load+0x3C5) [0x19ad4b5]
       mongod(+0x15B1569) [0x19b1569]
       mongod(__wt_btree_open+0xC7C) [0x19cdffc]
       mongod(__wt_conn_btree_open+0x140) [0x1a04270]
       mongod(__wt_session_get_btree+0xE7) [0x1a7d047]
       mongod(__wt_session_get_btree+0x629) [0x1a7d589]
       mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a7d6ab]
       mongod(__wt_curfile_open+0x218) [0x1a12858]
       mongod(+0x167A1D5) [0x1a7a1d5]
       mongod(__wt_metadata_cursor_open+0x5F) [0x1a44ccf]
       mongod(__wt_metadata_cursor+0x7E) [0x1a44dce]
       mongod(wiredtiger_open+0x1541) [0x1a00e11]
       mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x77F) [0x10889cf]
       mongod(+0xC84C93) [0x1084c93]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xfadc48]
       mongod(_ZN5mongo13initAndListenEi+0x3DD) [0x9b48fd]
       mongod(main+0x15D) [0x96e04d]
       libc.so.6(__libc_start_main+0xF5) [0x7f7609e89ec5]
       mongod(+0x5B1037) [0x9b1037]
      -----  END BACKTRACE  -----
      

        1. repair_attempt.tar.gz
          7 kB
        2. WiredTiger.turtle
          0.9 kB
        3. WiredTiger.wt
          144 kB

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            coryrobinson42@gmail.com Cory Robinson
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: