Major - P3 Hi,
I have accidentally reboot the mongodb server (AWS EC2) and the mongod now won't be able to boot. I have Googled for similar issues and end up here. I have tried using `--repair` on the ./data directory but no luck. Seems to me the only way is to seek help from the development team here and fix the checksum of the WiredTiger.wt. I have attached the related files and please kindly help to repair the file and I will be forever grateful.
2018-04-04T02:41:16.911+0000 I CONTROL [initandlisten] MongoDB starting : pid=1713 port=27017 dbpath=/home/datafeed/luxify-scrap/./data 64-bit host=ip-172-31-24-153 2018-04-04T02:41:16.911+0000 I CONTROL [initandlisten] db version v3.0.12 2018-04-04T02:41:16.911+0000 I CONTROL [initandlisten] git version: 33934938e0e95d534cebbaff656cde916b9c3573 2018-04-04T02:41:16.911+0000 I CONTROL [initandlisten] build info: Linux ip-10-229-66-2 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49 2018-04-04T02:41:16.911+0000 I CONTROL [initandlisten] allocator: tcmalloc 2018-04-04T02:41:16.911+0000 I CONTROL [initandlisten] options: { processManagement: { fork: true }, storage: { dbPath: "./data", engine: "wiredTiger" }, systemLog: { destination: "file", path: "/var/log/mongodb/mongod.log" } } 2018-04-04T02:41:16.933+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2018-04-04T02:41:16.938+0000 E STORAGE [initandlisten] WiredTiger (0) [1522809676:938331][1713:0x7f41675eabc0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 45056: block header checksum of 1936028255 doesn't match expected checksum of 3000991456 2018-04-04T02:41:16.938+0000 E STORAGE [initandlisten] WiredTiger (0) [1522809676:938356][1713:0x7f41675eabc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value 2018-04-04T02:41:16.938+0000 E STORAGE [initandlisten] WiredTiger (-31804) [1522809676:938368][1713:0x7f41675eabc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic 2018-04-04T02:41:16.938+0000 I - [initandlisten] Fatal Assertion 28558 2018-04-04T02:41:16.950+0000 I CONTROL [initandlisten] 0xfa8242 0xf454d9 0xf29706 0xdca081 0x13ff32c 0x13ff4dd 0x13ff954 0x134832c 0x1348883 0x13454a2 0x13495ce 0x1365204 0x1390a5b 0x13fe6d9 0x13ca183 0x138dfc9 0xdb429b 0xdb1e70 0xab1dca 0x82f9b1 0x7fd249 0x7f4165eb7f45 0x82d6d7 ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"BA8242","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"B454D9","s":"_ZN5mongo10logContextEPKc"},{"b":"400000","o":"B29706","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"9CA081"},{"b":"400000","o":"FFF32C","s":"__wt_eventv"},{"b":"400000","o":"FFF4DD","s":"__wt_err"},{"b":"400000","o":"FFF954","s":"__wt_panic"},{"b":"400000","o":"F4832C","s":"__wt_block_extlist_read"},{"b":"400000","o":"F48883","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"F454A2","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"F495CE"},{"b":"400000","o":"F65204","s":"__wt_btree_open"},{"b":"400000","o":"F90A5B","s":"__wt_conn_btree_get"},{"b":"400000","o":"FFE6D9","s":"__wt_session_get_btree"},{"b":"400000","o":"FCA183","s":"__wt_metadata_open"},{"b":"400000","o":"F8DFC9","s":"wiredtiger_open"},{"b":"400000","o":"9B429B","s":"_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb"},{"b":"400000","o":"9B1E70"},{"b":"400000","o":"6B1DCA","s":"_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs"},{"b":"400000","o":"42F9B1","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"3FD249","s":"main"},{"b":"7F4165E96000","o":"21F45","s":"__libc_start_main"},{"b":"400000","o":"42D6D7"}],"processInfo":{ "mongodbVersion" : "3.0.12", "gitVersion" : "33934938e0e95d534cebbaff656cde916b9c3573", "uname" : { "sysname" : "Linux", "release" : "3.13.0-92-generic", "version" : "#139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "EBD85547D0C90260CB4C429A2A005349F1C70CB5" }, { "b" : "7FFD2E351000", "elfType" : 3, "buildId" : "7F14688F101A2ACE5CAD23DFBFBC918616651576" }, { "b" : "7F41671BE000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7F4166F5F000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "74864DB9D5F69D39A67E4755012FB6573C469B3D" }, { "b" : "7F4166B83000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "AAE7CFF8351B730830BDBCE0DCABBE06574B7144" }, { "b" : "7F416697B000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7F4166777000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7F4166471000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7F416625B000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F4165E96000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7F41673DC000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x32) [0xfa8242] mongod(_ZN5mongo10logContextEPKc+0xE9) [0xf454d9] mongod(_ZN5mongo13fassertFailedEi+0x66) [0xf29706] mongod(+0x9CA081) [0xdca081] mongod(__wt_eventv+0x49C) [0x13ff32c] mongod(__wt_err+0x8D) [0x13ff4dd] mongod(__wt_panic+0x24) [0x13ff954] mongod(__wt_block_extlist_read+0x6C) [0x134832c] mongod(__wt_block_extlist_read_avail+0x33) [0x1348883] mongod(__wt_block_checkpoint_load+0x202) [0x13454a2] mongod(+0xF495CE) [0x13495ce] mongod(__wt_btree_open+0xB54) [0x1365204] mongod(__wt_conn_btree_get+0x1AB) [0x1390a5b] mongod(__wt_session_get_btree+0x449) [0x13fe6d9] mongod(__wt_metadata_open+0x33) [0x13ca183] mongod(wiredtiger_open+0xD99) [0x138dfc9] mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x53B) [0xdb429b] mongod(+0x9B1E70) [0xdb1e70] mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x32A) [0xab1dca] mongod(_ZN5mongo13initAndListenEi+0x421) [0x82f9b1] mongod(main+0x159) [0x7fd249] libc.so.6(__libc_start_main+0xF5) [0x7f4165eb7f45] mongod(+0x42D6D7) [0x82d6d7] ----- END BACKTRACE ----- 2018-04-04T02:41:16.950+0000 I - [initandlisten] ***aborting after fassert() failure