./mongod --dbpath /SFS/aep/prd/dc/mongodb_data/db --repair 2019-05-03T21:51:51.906-0400 I CONTROL [initandlisten] MongoDB starting : pid=19521 port=27017 dbpath=/SFS/aep/prd/dc/mongodb_data/db 64-bit host=mongo_prd1.merck.com 2019-05-03T21:51:51.906-0400 I CONTROL [initandlisten] db version v3.6.0 2019-05-03T21:51:51.906-0400 I CONTROL [initandlisten] git version: a57d8e71e6998a2d0afde7edc11bd23e5661c915 2019-05-03T21:51:51.906-0400 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013 2019-05-03T21:51:51.906-0400 I CONTROL [initandlisten] allocator: tcmalloc 2019-05-03T21:51:51.906-0400 I CONTROL [initandlisten] modules: none 2019-05-03T21:51:51.906-0400 I CONTROL [initandlisten] build environment: 2019-05-03T21:51:51.906-0400 I CONTROL [initandlisten] distmod: rhel70 2019-05-03T21:51:51.906-0400 I CONTROL [initandlisten] distarch: x86_64 2019-05-03T21:51:51.906-0400 I CONTROL [initandlisten] target_arch: x86_64 2019-05-03T21:51:51.906-0400 I CONTROL [initandlisten] options: { repair: true, storage: { dbPath: "/SFS/aep/prd/dc/mongodb_data/db" } } 2019-05-03T21:51:51.913-0400 W - [initandlisten] Detected unclean shutdown - /SFS/aep/prd/dc/mongodb_data/db/mongod.lock is not empty. 2019-05-03T21:51:51.914-0400 I - [initandlisten] Detected data files in /SFS/aep/prd/dc/mongodb_data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2019-05-03T21:51:51.914-0400 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2019-05-03T21:51:51.915-0400 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint. 2019-05-03T21:51:51.915-0400 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=31621M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress), 2019-05-03T21:51:51.940-0400 E STORAGE [initandlisten] WiredTiger error (0) [1556934711:940439][19521:0x7f0551023b00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value: (__wt_turtle_read, 291) 2019-05-03T21:51:51.940-0400 E STORAGE [initandlisten] WiredTiger error (-31804) [1556934711:940503][19521:0x7f0551023b00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic 2019-05-03T21:51:51.940-0400 F - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361 2019-05-03T21:51:51.940-0400 F - [initandlisten] ***aborting after fassert() failure 2019-05-03T21:51:51.984-0400 F - [initandlisten] Got signal: 6 (Aborted). 0x7f055323ab51 0x7f0553239d69 0x7f055323a24d 0x7f054fa01370 0x7f054f6661d7 0x7f054f6678c8 0x7f05519cc4e3 0x7f0551a98c0e 0x7f0551969277 0x7f0551969493 0x7f05519697bc 0x7f0551ac46e9 0x7f0551ac2d69 0x7f0551aa884a 0x7f0551af56db 0x7f0551af5bfd 0x7f0551af5ebc 0x7f0551b5e622 0x7f0551aec468 0x7f0551ac243e 0x7f0551ac251b 0x7f0551aa75b4 0x7f0551a7d07a 0x7f0551a617c4 0x7f0551c4fe37 0x7f0551965027 0x7f0551a406ec 0x7f05519ce1a9 0x7f054f652b35 0x7f0551a3008f ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"7F0551041000","o":"21F9B51","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F0551041000","o":"21F8D69"},{"b":"7F0551041000","o":"21F924D"},{"b":"7F054F9F2000","o":"F370"},{"b":"7F054F631000","o":"351D7","s":"gsignal"},{"b":"7F054F631000","o":"368C8","s":"abort"},{"b":"7F0551041000","o":"98B4E3","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"7F0551041000","o":"A57C0E"},{"b":"7F0551041000","o":"928277","s":"__wt_eventv"},{"b":"7F0551041000","o":"928493","s":"__wt_err"},{"b":"7F0551041000","o":"9287BC","s":"__wt_panic"},{"b":"7F0551041000","o":"A836E9","s":"__wt_turtle_read"},{"b":"7F0551041000","o":"A81D69","s":"__wt_metadata_search"},{"b":"7F0551041000","o":"A6784A","s":"__wt_conn_dhandle_open"},{"b":"7F0551041000","o":"AB46DB","s":"__wt_session_get_dhandle"},{"b":"7F0551041000","o":"AB4BFD","s":"__wt_session_get_dhandle"},{"b":"7F0551041000","o":"AB4EBC","s":"__wt_session_get_btree_ckpt"},{"b":"7F0551041000","o":"B1D622","s":"__wt_curfile_open"},{"b":"7F0551041000","o":"AAB468"},{"b":"7F0551041000","o":"A8143E","s":"__wt_metadata_cursor_open"},{"b":"7F0551041000","o":"A8151B","s":"__wt_metadata_cursor"},{"b":"7F0551041000","o":"A665B4","s":"wiredtiger_open"},{"b":"7F0551041000","o":"A3C07A","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"7F0551041000","o":"A207C4"},{"b":"7F0551041000","o":"C0EE37","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"7F0551041000","o":"924027"},{"b":"7F0551041000","o":"9FF6EC","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"7F0551041000","o":"98D1A9","s":"main"},{"b":"7F054F631000","o":"21B35","s":"__libc_start_main"},{"b":"7F0551041000","o":"9EF08F"}],"processInfo":{ "mongodbVersion" : "3.6.0", "gitVersion" : "a57d8e71e6998a2d0afde7edc11bd23e5661c915", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-514.10.2.el7.x86_64", "version" : "#1 SMP Fri Mar 3 00:04:05 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "7F0551041000", "elfType" : 3, "buildId" : "DE8018F36C6B9F029DEF9254662EDB5AF914ED1A" }, { "b" : "7FFC96688000", "elfType" : 3, "buildId" : "37F4BC2378CEB52CED04E51DD874A4B847BA93DA" }, { "b" : "7F0550C05000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "FE7AE845A123A3DFC0FDC2408BCBC2BA8B61B158" }, { "b" : "7F0550993000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6468A4E28A19CDD885A3CBC30E009589CA4C2E92" }, { "b" : "7F0550532000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "67CEB4EDD36BFE0EB31CD92DA2694ACA5377A599" }, { "b" : "7F055032A000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "82E77ADE22BC9FFF8D3458BD37331E7EDF174C28" }, { "b" : "7F0550126000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "C5F560504E1AF52E29679C3B52FF11121015D6BB" }, { "b" : "7F054FE24000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "721C7CC9488EFA25F83B48AF713AB27DBE48EF3E" }, { "b" : "7F054FC0E000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "408B46E291B2D4C9612E27C0509D165D7E186D40" }, { "b" : "7F054F9F2000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "C3DEB1FA27CD0C1C3CC575B944ABACBA0698B0F2" }, { "b" : "7F054F631000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "8B2C421716985B927AA0CAF2A05D0B1F452367F7" }, { "b" : "7F0550E1F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "8F3E366E2DB73C330A3791DEAE31AE9579099B44" }, { "b" : "7F054F3E3000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "A2499C359AA179EE23324ED949C0E508E4434F10" }, { "b" : "7F054F0FC000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "E09A34D9083DC6FEAF7018C09D55631DEEE2836D" }, { "b" : "7F054EEF8000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "BF54B7C8932E450769FBBB8B18864D1DD70BBC67" }, { "b" : "7F054ECC6000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "BF8F00D7CB849ADB0B7A4703BC7B8D66AEE6A49C" }, { "b" : "7F054EAB0000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "EA8E45DC8E395CC5E26890470112D97A1F1E0B65" }, { "b" : "7F054E8A1000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "1E7A92FDD6FB3871DA97F4BCA2E147E72B6B6E1F" }, { "b" : "7F054E69D000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F054E476000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "76687CA31A406854DF3BCF8D03055656F56E6892" }, { "b" : "7F054E215000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "AE64AA461A26E01F60408013D361749D56DD0AE1" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7f055323ab51] mongod(+0x21F8D69) [0x7f0553239d69] mongod(+0x21F924D) [0x7f055323a24d] libpthread.so.0(+0xF370) [0x7f054fa01370] libc.so.6(gsignal+0x37) [0x7f054f6661d7] libc.so.6(abort+0x148) [0x7f054f6678c8] mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x7f05519cc4e3] mongod(+0xA57C0E) [0x7f0551a98c0e] mongod(__wt_eventv+0x3D7) [0x7f0551969277] mongod(__wt_err+0x9D) [0x7f0551969493] mongod(__wt_panic+0x33) [0x7f05519697bc] mongod(__wt_turtle_read+0x269) [0x7f0551ac46e9] mongod(__wt_metadata_search+0xA9) [0x7f0551ac2d69] mongod(__wt_conn_dhandle_open+0x8A) [0x7f0551aa884a] mongod(__wt_session_get_dhandle+0xFB) [0x7f0551af56db] mongod(__wt_session_get_dhandle+0x61D) [0x7f0551af5bfd] mongod(__wt_session_get_btree_ckpt+0x14C) [0x7f0551af5ebc] mongod(__wt_curfile_open+0x52) [0x7f0551b5e622] mongod(+0xAAB468) [0x7f0551aec468] mongod(__wt_metadata_cursor_open+0x6E) [0x7f0551ac243e] mongod(__wt_metadata_cursor+0x4B) [0x7f0551ac251b] mongod(wiredtiger_open+0x19D4) [0x7f0551aa75b4] mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xBEA) [0x7f0551a7d07a] mongod(+0xA207C4) [0x7f0551a617c4] mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x637) [0x7f0551c4fe37] mongod(+0x924027) [0x7f0551965027] mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x86C) [0x7f0551a406ec] mongod(main+0x9) [0x7f05519ce1a9] libc.so.6(__libc_start_main+0xF5) [0x7f054f652b35] mongod(+0x9EF08F) [0x7f0551a3008f] ----- END BACKTRACE ----- Aborted (core dumped)