Hi,
I have a problem.
One day, my mongodb environment encountered suddenly power off.
After that, I can't start mongod, and following messages displayed.
Could you help me to recover my database, please?
2018-03-19T09:41:44.979+0900 I CONTROL [initandlisten] MongoDB starting : pid=32098 port=27017 dbpath=/data/db 64-bit host=mangodb-xiao 2018-03-19T09:41:44.983+0900 I CONTROL [initandlisten] db version v3.4.6 2018-03-19T09:41:44.983+0900 I CONTROL [initandlisten] git version: c55eb86ef46ee7aede3b1e2a5d184a7df4bfb5b5 2018-03-19T09:41:44.983+0900 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016 2018-03-19T09:41:44.983+0900 I CONTROL [initandlisten] allocator: tcmalloc 2018-03-19T09:41:44.983+0900 I CONTROL [initandlisten] modules: enterprise 2018-03-19T09:41:44.983+0900 I CONTROL [initandlisten] build environment: 2018-03-19T09:41:44.983+0900 I CONTROL [initandlisten] distmod: ubuntu1604 2018-03-19T09:41:44.983+0900 I CONTROL [initandlisten] distarch: x86_64 2018-03-19T09:41:44.983+0900 I CONTROL [initandlisten] target_arch: x86_64 2018-03-19T09:41:44.983+0900 I CONTROL [initandlisten] options: {} 2018-03-19T09:41:45.036+0900 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2018-03-19T09:41:45.038+0900 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7511M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2018-03-19T09:41:45.052+0900 E STORAGE [initandlisten] WiredTiger error (0) [1521420105:52354][32098:0x7ff0afbdcd40], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 49152: block header checksum of 3126403452 doesn't match expected checksum of 71792573 2018-03-19T09:41:45.052+0900 E STORAGE [initandlisten] WiredTiger error (0) [1521420105:52407][32098:0x7ff0afbdcd40], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value 2018-03-19T09:41:45.052+0900 E STORAGE [initandlisten] WiredTiger error (-31804) [1521420105:52415][32098:0x7ff0afbdcd40], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic 2018-03-19T09:41:45.052+0900 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361 2018-03-19T09:41:45.052+0900 I - [initandlisten] ***aborting after fassert() failure 2018-03-19T09:41:45.090+0900 F - [initandlisten] Got signal: 6 (Aborted). 0x55b0bb686df1 0x55b0bb686009 0x55b0bb6864ed 0x7ff0acd06390 0x7ff0ac961428 0x7ff0ac96302a 0x55b0ba8844a9 0x55b0bb39b016 0x55b0ba88e742 0x55b0ba88e967 0x55b0ba88ebbf 0x55b0bbf8c47f 0x55b0bbf8c9cb 0x55b0bbf88fbd 0x55b0bbf8db97 0x55b0bbfa778b 0x55b0bbfe0b03 0x55b0bc06b36b 0x55b0bc06b8ad 0x55b0bc06bb6c 0x55b0bbfef201 0x55b0bc061718 0x55b0bc02b6be 0x55b0bc02b79b 0x55b0bbfdcd6d 0x55b0bb37fedf 0x55b0bb37875c 0x55b0bb26c590 0x55b0ba87050e 0x55b0ba88ff66 0x7ff0ac94c830 0x55b0ba8f5a49 ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"55B0BA023000","o":"1663DF1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55B0BA023000","o":"1663009"},{"b":"55B0BA023000","o":"16634ED"},{"b":"7FF0ACCF5000","o":"11390"},{"b":"7FF0AC92C000","o":"35428","s":"gsignal"},{"b":"7FF0AC92C000","o":"3702A","s":"abort"},{"b":"55B0BA023000","o":"8614A9","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55B0BA023000","o":"1378016"},{"b":"55B0BA023000","o":"86B742","s":"__wt_eventv"},{"b":"55B0BA023000","o":"86B967","s":"__wt_err"},{"b":"55B0BA023000","o":"86BBBF","s":"__wt_panic"},{"b":"55B0BA023000","o":"1F6947F","s":"__wt_block_extlist_read"},{"b":"55B0BA023000","o":"1F699CB","s":"__wt_block_extlist_read_avail"},{"b":"55B0BA023000","o":"1F65FBD","s":"__wt_block_checkpoint_load"},{"b":"55B0BA023000","o":"1F6AB97"},{"b":"55B0BA023000","o":"1F8478B","s":"__wt_btree_open"},{"b":"55B0BA023000","o":"1FBDB03","s":"__wt_conn_btree_open"},{"b":"55B0BA023000","o":"204836B","s":"__wt_session_get_btree"},{"b":"55B0BA023000","o":"20488AD","s":"__wt_session_get_btree"},{"b":"55B0BA023000","o":"2048B6C","s":"__wt_session_get_btree_ckpt"},{"b":"55B0BA023000","o":"1FCC201","s":"__wt_curfile_open"},{"b":"55B0BA023000","o":"203E718"},{"b":"55B0BA023000","o":"20086BE","s":"__wt_metadata_cursor_open"},{"b":"55B0BA023000","o":"200879B","s":"__wt_metadata_cursor"},{"b":"55B0BA023000","o":"1FB9D6D","s":"wiredtiger_open"},{"b":"55B0BA023000","o":"135CEDF","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55B0BA023000","o":"135575C"},{"b":"55B0BA023000","o":"1249590","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55B0BA023000","o":"84D50E"},{"b":"55B0BA023000","o":"86CF66","s":"main"},{"b":"7FF0AC92C000","o":"20830","s":"__libc_start_main"},{"b":"55B0BA023000","o":"8D2A49","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.6", "gitVersion" : "c55eb86ef46ee7aede3b1e2a5d184a7df4bfb5b5", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "4.8.0-36-generic", "version" : "#36~16.04.1-Ubuntu SMP Sun Feb 5 09:39:57 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "55B0BA023000", "elfType" : 3, "buildId" : "EAF1102647F381CAB3D8EC0A23EB4B023E96A5EC" }, { "b" : "7FFC45F76000", "elfType" : 3, "buildId" : "FE95293FF090A087E09855AADBE77230311AFFB1" }, { "b" : "7FF0AF7EB000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "96BCC7EB28D81B1469EED6F24FC083CBD58577BC" }, { "b" : "7FF0AF5A1000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "DB5180B568097E2A4690A5B40D36BD134C893FEE" }, { "b" : "7FF0AF332000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "DED90CB11A7648E334E068A2585DB44E8CC07686" }, { "b" : "7FF0AEEB6000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmpmibs.so.30", "elfType" : 3, "buildId" : "8FE272AED3883FEB05AC784E50D9D8F0FAC4E899" }, { "b" : "7FF0AECA7000", "path" : "/usr/lib/x86_64-linux-gnu/libsensors.so.4", "elfType" : 3, "buildId" : "1A855916EA4E082827D0599C11C0E82D98284118" }, { "b" : "7FF0AEA9A000", "path" : "/lib/x86_64-linux-gnu/libpci.so.3", "elfType" : 3, "buildId" : "3F90C8FDF5FC50256DD803E6F5EBED276A8193A6" }, { "b" : "7FF0AE896000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "AFDCB9B3774F5AFE6EB3EC9A76A8BA159A17D5B6" }, { "b" : "7FF0AE62D000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmpagent.so.30", "elfType" : 3, "buildId" : "901744249A5926FD226B85CBE594271ADFB80E52" }, { "b" : "7FF0AE423000", "path" : "/lib/x86_64-linux-gnu/libwrap.so.0", "elfType" : 3, "buildId" : "54FCBC5B0F994A13A9B0EAD46F23E7DA7F7FE75B" }, { "b" : "7FF0AE146000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmp.so.30", "elfType" : 3, "buildId" : "0C029B406758FAFD5EC0E9B2C1F81D8D5FFC9D26" }, { "b" : "7FF0ADD02000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "E6D4D2E4A048992CD5501E5985094E6CEC6C5D4F" }, { "b" : "7FF0ADAB1000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "ADF6F41F223D42193165FA0C55871F02D915FB19" }, { "b" : "7FF0AD8A2000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "6B9F4061A1D44813A54DA4DBB0088F529D8D78EA" }, { "b" : "7FF0AD599000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "4DBE0469DACBA00586428C9C7F4BBCEC8B6B1489" }, { "b" : "7FF0AD330000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "7F514146540382F59AD705BA8C913A75204C6858" }, { "b" : "7FF0AD128000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A656E8087D6226AE46A173477B8B8E0767D559D9" }, { "b" : "7FF0ACF12000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FF0ACCF5000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "4D7F52F335DC9665C2DCF308CE6514A6AE86DEDE" }, { "b" : "7FF0AC92C000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "60131540DADC6796CAB33388349E6E4E68692053" }, { "b" : "7FF0AFA06000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "EA8652B03788783033C3AF093DFB71509C1378F8" }, { "b" : "7FF0AC65A000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "16E3DBC6D048145939BB43BBFD7954D27421B00F" }, { "b" : "7FF0AC42B000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "AEB4C08FC47F86C475E9D3996DFE5E9B403ACEBF" }, { "b" : "7FF0AC227000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "1E16CB57F699E215A2A8D4EFEF90883BC749B12D" }, { "b" : "7FF0AC01C000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "DF3219B89E86920E901BAC4A80AA60F2B6134588" }, { "b" : "7FF0ABDE9000", "path" : "/usr/lib/x86_64-linux-gnu/libidn.so.11", "elfType" : 3, "buildId" : "F3945D210B52CC973C619CCD08C4394BBC33E07F" }, { "b" : "7FF0ABBCD000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "B15D78D75542A4CAEC1DB7F7FB7723817E58D536" }, { "b" : "7FF0AB9B3000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "340B7B463F981B8A0FB3451751F881DF1B0C2F74" }, { "b" : "7FF0AB798000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "1C39D66F4A377C4E71DA0BF23E7221B45DEF9E7F" }, { "b" : "7FF0AFBEA000", "path" : "/lib/x86_64-linux-gnu/libudev.so.1", "elfType" : 3, "buildId" : "677E00F6578E7FD54267F859E744E43667001142" }, { "b" : "7FF0AB3BD000", "path" : "/usr/lib/x86_64-linux-gnu/libperl.so.5.22", "elfType" : 3, "buildId" : "9F15B66FC5A0C22FB46B2B2E91F3CE705857A05E" }, { "b" : "7FF0AB1A4000", "path" : "/lib/x86_64-linux-gnu/libnsl.so.1", "elfType" : 3, "buildId" : "A393386743793EDF5A6DE8E5BF11C315E0ADD6FB" }, { "b" : "7FF0AAF63000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "40315538B8F467FA9379C26806C2C2E5F75DF62A" }, { "b" : "7FF0AAC33000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "B081530BA0EE8C7628C535957E776208328C3AD8" }, { "b" : "7FF0AAA2F000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "3364D4BF2113C4E8D17EF533867ECC99A53413D6" }, { "b" : "7FF0AA7FC000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "B11678F560199547DCF726384EA39153EE0DFABF" }, { "b" : "7FF0AA5C6000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "D6B36C5A463EE0FA84FDD6D5FD3F7726EDB90D54" }, { "b" : "7FF0AA346000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "7B3533D5998D20EE1A1BE3F87789B69041E7F620" }, { "b" : "7FF0AA10E000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "C60ECF42664BABCC4F6D735D70051131C9FD28C9" }, { "b" : "7FF0A9F05000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "D72147C9E81DC821D2C5CA3A93757EE00DAC67EA" }, { "b" : "7FF0A9C7B000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "889221E5B7DFAA3D74CD8D55DAEF89DD962DAEB7" }, { "b" : "7FF0A99D9000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "B7A0E46FB5107A9566AB0D448C865DB7BB1390BF" }, { "b" : "7FF0A97A6000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "54027CA597C9AFC9D32A3D677857E5C8512C9FBF" }, { "b" : "7FF0A9590000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "AE98B3C2FEC7AEDB3064E03BD8C1883FD12898D1" }, { "b" : "7FF0A932C000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "A0E2D03FF5CF65937F4425D4EFD4D655243809EB" }, { "b" : "7FF0A9119000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "F5B37E248C55FD582ECD9B0240F96E8458F8E6F0" }, { "b" : "7FF0A8EF0000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "FEA2A08A986C1C954F7DED5A27616F0B4B18C6A2" }, { "b" : "7FF0A8CE1000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "E5BC16E04FC04459D5AB8F045B5F7C7838B86027" }, { "b" : "7FF0A8A96000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "6B6EF07C8D0FA1CE7CCEE58043BDB6C347F25544" }, { "b" : "7FF0A87C1000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "D9782BA023CAEC26B15D8676E3A5D07B55E121EF" }, { "b" : "7FF0A85B9000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "9D9C958F1F4894AFEF6AECD90D1C430EA29AC34F" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55b0bb686df1] mongod(+0x1663009) [0x55b0bb686009] mongod(+0x16634ED) [0x55b0bb6864ed] libpthread.so.0(+0x11390) [0x7ff0acd06390] libc.so.6(gsignal+0x38) [0x7ff0ac961428] libc.so.6(abort+0x16A) [0x7ff0ac96302a] mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55b0ba8844a9] mongod(+0x1378016) [0x55b0bb39b016] mongod(__wt_eventv+0x3D7) [0x55b0ba88e742] mongod(__wt_err+0x9D) [0x55b0ba88e967] mongod(__wt_panic+0x24) [0x55b0ba88ebbf] mongod(__wt_block_extlist_read+0x8F) [0x55b0bbf8c47f] mongod(__wt_block_extlist_read_avail+0x2B) [0x55b0bbf8c9cb] mongod(__wt_block_checkpoint_load+0x26D) [0x55b0bbf88fbd] mongod(+0x1F6AB97) [0x55b0bbf8db97] mongod(__wt_btree_open+0xB3B) [0x55b0bbfa778b] mongod(__wt_conn_btree_open+0x163) [0x55b0bbfe0b03] mongod(__wt_session_get_btree+0xFB) [0x55b0bc06b36b] mongod(__wt_session_get_btree+0x63D) [0x55b0bc06b8ad] mongod(__wt_session_get_btree_ckpt+0x14C) [0x55b0bc06bb6c] mongod(__wt_curfile_open+0x161) [0x55b0bbfef201] mongod(+0x203E718) [0x55b0bc061718] mongod(__wt_metadata_cursor_open+0x6E) [0x55b0bc02b6be] mongod(__wt_metadata_cursor+0x4B) [0x55b0bc02b79b] mongod(wiredtiger_open+0x164D) [0x55b0bbfdcd6d] mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x55b0bb37fedf] mongod(+0x135575C) [0x55b0bb37875c] mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x55b0bb26c590] mongod(+0x84D50E) [0x55b0ba87050e] mongod(main+0x966) [0x55b0ba88ff66] libc.so.6(__libc_start_main+0xF0) [0x7ff0ac94c830] mongod(_start+0x29) [0x55b0ba8f5a49] ----- END BACKTRACE -----