After the mongodb server suddenly loses power, it restarts with the Got signal: 6 (Aborted).

The mongodb engineer helped me fix the WiredTiger.wt and WiredTiger.turtle files, but the problem still exists.

What else can I do to save my data? Thank goodness.

Most recent stacktrace is here:

---

2018-12-27T20:32:33.069+0800 I CONTROL [initandlisten] options: { config: "/usr/local/iGET/SRE/software/mongo/bin/../conf/mongodb_single.conf", cpu: false, net: { bindIp: "127.0.0.1,3.1.204.206", http:

{ RESTInterfaceEnabled: false }

, maxIncomingConnections: 2048, port: 8888, ssl: { CAFile: "/usr/local/iGET/SRE/ssl/mongodb_cert/root-ca.pem", PEMKeyFile: "/usr/local/iGET/SRE/ssl/mongodb_cert/VCM-0-6S.pem", PEMKeyPassword: "<password>", allowInvalidCertificates: true, allowInvalidHostnames: true, clusterFile: "/usr/local/iGET/SRE/ssl/mongodb_cert/VCM-0-6S.pem", clusterPassword: "<password>", disabledProtocols: "TLS1_0,TLS1_1", mode: "requireSSL", sslCipherConfig: "TLSv1.2+AES", weakCertificateValidation: false }, unixDomainSocket: { enabled: false }, wireObjectCheck: true }, notablescan: false, operationProfiling: { mode: "off", slowOpThresholdMs: 100 }, processManagement: { fork: true, pidFilePath: "/sre_storage/data1/mongo/mongo.pid" }, repair: true, replication: { oplogSizeMB: 81920 }, security: { authorization: "enabled", clusterAuthMode: "x509", javascriptEnabled: false, keyFile: "/usr/local/iGET/SRE/software/mongo/bin/key" }, setParameter: { enableLocalhostAuthBypass: "0", enableTestCommands: "0", logUserIds: "1" }, storage: { dbPath: "/sre_storage/data1/mongo/data/", directoryPerDB: true, engine: "wiredTiger", mmapv1: { nsSize: 64, preallocDataFiles: true, quota:

{ enforced: false }

, smallFiles: false }, syncPeriodSecs: 5.0, wiredTiger: { collectionConfig:

{ blockCompressor: "snappy" }

, engineConfig: { cacheSizeGB: 4, journalCompressor: "snappy" }, indexConfig: { prefixCompression: true } } }, systemLog: { destination: "file", logAppend: true, path: "/sre_storage/data1/mongo/logs/mongostatus.log", verbosity: 0 } }
2018-12-27T20:32:33.070+0800 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2018-12-27T20:32:33.070+0800 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=5,log_size=2GB),statistics_log=(wait=0),
2018-12-27T20:32:33.083+0800 E STORAGE [initandlisten] WiredTiger (0) [1545913953:83030][5671:0x7fcf046ccdc0], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2018-12-27T20:32:33.083+0800 E STORAGE [initandlisten] WiredTiger (-31804) [1545913953:83063][5671:0x7fcf046ccdc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-12-27T20:32:33.083+0800 I - [initandlisten] Fatal Assertion 28558
2018-12-27T20:32:33.083+0800 I - [initandlisten]

***aborting after fassert() failure

2018-12-27T20:32:33.099+0800 F - [initandlisten] Got signal: 6 (Aborted).

0x130ab72 0x1309cc9 0x130a4d2 0x7fcf02fb5100 0x7fcf02c1a597 0x7fcf02c1bc88 0x1291192 0x108b693 0x94dc9e 0x94dd73 0x94df37 0x1a3d606 0x1a3bf9a 0x19f30e1 0x1a79dfe 0x1a7a485 0x1a7a8fb 0x1a01d06 0x1a7001d 0x1a3b78f 0x1a3b89e 0x19ef62e 0x1072ab1 0x106ee93 0xf98788 0x99613e 0x9986f0 0x94e79d 0x7fcf02c06bb5 0x992aad
----- BEGIN BACKTRACE -----

{"backtrace":[\{"b":"400000","o":"F0AB72","s":"_ZN5mongo15printStackTraceERSo"}

,{"b":"400000","o":"F09CC9"},{"b":"400000","o":"F0A4D2"},{"b":"7FCF02FA6000","o":"F100"},{"b":"7FCF02BE5000","o":"35597","s":"gsignal"},{"b":"7FCF02BE5000","o":"36C88","s":"abort"},{"b":"400000","o":"E91192","s":"ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C8B693"},{"b":"400000","o":"54DC9E"},{"b":"400000","o":"54DD73","s":"wt_err"},{"b":"400000","o":"54DF37","s":"wt_panic"},{"b":"400000","o":"163D606"},{"b":"400000","o":"163BF9A"},{"b":"400000","o":"15F30E1"},{"b":"400000","o":"1679DFE"},{"b":"400000","o":"167A485"},{"b":"400000","o":"167A8FB"},{"b":"400000","o":"1601D06"},{"b":"400000","o":"167001D"},{"b":"400000","o":"163B78F"},{"b":"400000","o":"163B89E"},{"b":"400000","o":"15EF62E","s":"wiredtiger_open"},{"b":"400000","o":"C72AB1","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C6EE93"},{"b":"400000","o":"B98788","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"59613E"},{"b":"400000","o":"5986F0","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"54E79D","s":"main"},{"b":"7FCF02BE5000","o":"21BB5","s":"_libc_start_main"},{"b":"400000","o":"592AAD"}],"processInfo":{ "mongodbVersion" : "3.2.12", "gitVersion" : "ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14", "compiledModules" : [], "uname" :

{ "sysname" : "Linux", "release" : "3.10.0-327.59.59.46.h24.x86_64", "version" : "#1 SMP Wed Oct 11 06:59:50 UTC 2017", "machine" : "x86_64" }

, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "ADE07DD69BB7FD3B7000A0A0FE3C8178E7C83809" }, { "b" : "7FFD1FEA3000", "elfType" : 3, "buildId" : "385332720BD3BAAB7719A3D57F3FA52B9E70D20E" }, { "b" : "7FCF04252000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "AAB14F6D12976911FDFDA468E4F6213595F05798" }, { "b" : "7FCF03E68000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "8B2012EB69FF14394DDC8321F468022AD82B8482" }, { "b" : "7FCF03C60000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "2D227847AB521134592AEC7BA0F89025137449ED" }, { "b" : "7FCF03A5C000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "0D169E3DB55D4C66617903F940E412803298DA80" }, { "b" : "7FCF036DA000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "3BCAF1CF6BBCC1C79A0A64B3E996BD365F11C71A" }, { "b" : "7FCF033D8000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "0AFFB8028C0403515173F63B973AFA45D8A6E7DD" }, { "b" : "7FCF031C2000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "802D73D3C6B7203CF2CACBA3AF94A463E536FA57" }, { "b" : "7FCF02FA6000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "2DB99C0B19EB7547970E44D450444332D06EBAA8" }, { "b" : "7FCF02BE5000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "28D908BB0F165C9EA3B2501734B6E6CA619BDA7F" }, { "b" : "7FCF044C0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5460CA1B85820630FE5171A049178C9774B1BB12" }, { "b" : "7FCF02997000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "A875D496A04DEECBE1B99645B2AB3AF9D57EAFDC" }, { "b" : "7FCF026B0000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "83AC4B0EA72D0C262414C60A23841CA30FD4FAEB" }, { "b" : "7FCF024AC000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "F91C59010F3EADFB0F37BD4A3D9F9DA7F2A5449E" }, { "b" : "7FCF0227A000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "5A49BC797F1927AB1091AB74D2FCD5394DFEC678" }, { "b" : "7FCF02064000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "AAC85EE763F961681951A9D68D9F8A608178FEF2" }, { "b" : "7FCF01E55000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "55AAF7A0C4220AD1E5D4A73D2AC33BB78279539F" }, { "b" : "7FCF01C51000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "E7FD292753065CB083D21C743F998F9ECC824B1C" }, { "b" : "7FCF01A37000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "55A18352E2DB168A3EE70CFB1C553C138B8F7F5A" }, { "b" : "7FCF01812000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "DA49E0D5C8E9150C3860617509A0CD6106E487A0" }, { "b" : "7FCF015B1000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "7D76D546405803D934E009061E6DECD4874EA63E" }, { "b" : "7FCF0138C000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "0815E89D27D798AB2F45CA887DF9B3085E3BAB52" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x130ab72]
mongod(+0xF09CC9) [0x1309cc9]
mongod(+0xF0A4D2) [0x130a4d2]
libpthread.so.0(+0xF100) [0x7fcf02fb5100]
libc.so.6(gsignal+0x37) [0x7fcf02c1a597]
libc.so.6(abort+0x148) [0x7fcf02c1bc88]
mongod(_ZN5mongo13fassertFailedEi+0x82) [0x1291192]
mongod(+0xC8B693) [0x108b693]
mongod(+0x54DC9E) [0x94dc9e]
mongod(__wt_err+0x8D) [0x94dd73]
mongod(__wt_panic+0x24) [0x94df37]
mongod(+0x163D606) [0x1a3d606]
mongod(+0x163BF9A) [0x1a3bf9a]
mongod(+0x15F30E1) [0x19f30e1]
mongod(+0x1679DFE) [0x1a79dfe]
mongod(+0x167A485) [0x1a7a485]
mongod(+0x167A8FB) [0x1a7a8fb]
mongod(+0x1601D06) [0x1a01d06]
mongod(+0x167001D) [0x1a7001d]
mongod(+0x163B78F) [0x1a3b78f]
mongod(+0x163B89E) [0x1a3b89e]
mongod(wiredtiger_open+0x148E) [0x19ef62e]
mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x681) [0x1072ab1]
mongod(+0xC6EE93) [0x106ee93]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf98788]
mongod(+0x59613E) [0x99613e]
mongod(_ZN5mongo13initAndListenEi+0x10) [0x9986f0]
mongod(main+0x15D) [0x94e79d]
libc.so.6(__libc_start_main+0xF5) [0x7fcf02c06bb5]
mongod(+0x592AAD) [0x992aad]
----- END BACKTRACE -----

---

 

Attachment data.rar is all files in the dbpath, except the original data file.

Attachment repair_attempt.tar.gz is the data that mongo engineers helped me to fix last time.