Hi Team,
mongod is crashing continously with the below logs. This is happening repeatedly because of which mongod doesnt come up.
This is on linux platform.
2017-10-13T16:59:49.436+0000 I CONTROL [main] ***** SERVER RESTARTED ***** 2017-10-13T16:59:49.443+0000 I CONTROL [initandlisten] MongoDB starting : pid=7489 port=27017 dbpath=/var/opt/lamplighter/data 64-bit host=sojo-cfmcv-01.provocraft.com 2017-10-13T16:59:49.443+0000 I CONTROL [initandlisten] db version v3.2.12 2017-10-13T16:59:49.443+0000 I CONTROL [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14 2017-10-13T16:59:49.443+0000 I CONTROL [initandlisten] allocator: tcmalloc 2017-10-13T16:59:49.443+0000 I CONTROL [initandlisten] modules: none 2017-10-13T16:59:49.443+0000 I CONTROL [initandlisten] build environment: 2017-10-13T16:59:49.443+0000 I CONTROL [initandlisten] distarch: x86_64 2017-10-13T16:59:49.443+0000 I CONTROL [initandlisten] target_arch: x86_64 2017-10-13T16:59:49.443+0000 I CONTROL [initandlisten] options: { config: "/var/opt/lamplighter/etc/mongod.conf", storage: { dbPath: "/var/opt/lamplighter/data", engine: "wiredTiger", wiredTiger: { engineConfig: { configString: "cache_size=1536M" } } }, systemLog: { component: { accessControl: { verbosity: 0 }, command: { verbosity: 0 } }, destination: "file", logAppend: true, logRotate: "reopen", path: "/var/opt/lamplighter/log/mongodb.log", timeStampFormat: "iso8601-local", traceAllExceptions: false, verbosity: 0 } } 2017-10-13T16:59:49.495+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=8G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),cache_size=1536M 2017-10-13T16:59:49.741+0000 E STORAGE [initandlisten] WiredTiger (-31802) [1507913989:740996][7489:0x7f4165a00c80], file:collection-40--8734011645000920999.wt, WT_SESSION.open_cursor: unable to read root page from file:collection-40--8734011645000920999.wt: WT_ERROR: non-specific WiredTiger error 2017-10-13T16:59:49.741+0000 I - [initandlisten] Invariant failure: ret resulted in status UnknownError: -31802: WT_ERROR: non-specific WiredTiger error at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 79 2017-10-13T16:59:49.751+0000 I CONTROL [initandlisten] 0x130ff72 0x12bae28 0x12a314d 0x108d304 0x108bc00 0x1087c22 0x108662d 0x10771db 0xfd66d4 0xfdc472 0x1075a58 0xf996e8 0x95a91e 0x95cec0 0x9600cd 0x7f4164930635 0x9572d9 ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"F0FF72","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"EBAE28","s":"_ZN5mongo10logContextEPKc"},{"b":"400000","o":"EA314D","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"400000","o":"C8D304","s":"_ZN5mongo17WiredTigerSession9getCursorERKSsmb"},{"b":"400000","o":"C8BC00","s":"_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE"},{"b":"400000","o":"C87C22","s":"_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b"},{"b":"400000","o":"C8662D","s":"_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_SsbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE"},{"b":"400000","o":"C771DB","s":"_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE"},{"b":"400000","o":"BD66D4","s":"_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSsb"},{"b":"400000","o":"BDC472","s":"_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE"},{"b":"400000","o":"C75A58"},{"b":"400000","o":"B996E8","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"55A91E"},{"b":"400000","o":"55CEC0","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"5600CD","s":"main"},{"b":"7F416490F000","o":"21635","s":"__libc_start_main"},{"b":"400000","o":"5572D9"}],"processInfo":{ "mongodbVersion" : "3.2.12", "gitVersion" : "ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.53sf.virtual-150", "version" : "#1 SMP Thu Aug 31 19:10:36 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFEEEFE000", "path" : "linux-vdso.so.1", "elfType" : 3 }, { "b" : "7F41655E7000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7F41653E3000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7F41650E9000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7F4164ED4000", "path" : "/usr/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F4164CB7000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7F416490F000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7F41657F0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x130ff72] mongod(_ZN5mongo10logContextEPKc+0x138) [0x12bae28] mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xAD) [0x12a314d] mongod(_ZN5mongo17WiredTigerSession9getCursorERKSsmb+0xE4) [0x108d304] mongod(_ZN5mongo16WiredTigerCursorC1ERKSsmbPNS_16OperationContextE+0x50) [0x108bc00] mongod(_ZN5mongo21WiredTigerRecordStore6CursorC1EPNS_16OperationContextERKS0_b+0x92) [0x1087c22] mongod(_ZN5mongo21WiredTigerRecordStoreC1EPNS_16OperationContextENS_10StringDataES3_SsbbllPNS_14CappedCallbackEPNS_20WiredTigerSizeStorerE+0x3ED) [0x108662d] mongod(_ZN5mongo18WiredTigerKVEngine14getRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsE+0x1AB) [0x10771db] mongod(_ZN5mongo22KVDatabaseCatalogEntry14initCollectionEPNS_16OperationContextERKSsb+0x204) [0xfd66d4] mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsE+0x6D2) [0xfdc472] mongod(+0xC75A58) [0x1075a58] mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf996e8] mongod(+0x55A91E) [0x95a91e] mongod(_ZN5mongo13initAndListenEi+0x10) [0x95cec0] mongod(main+0x15D) [0x9600cd] libc.so.6(__libc_start_main+0xF5) [0x7f4164930635] mongod(+0x5572D9) [0x9572d9] ----- END BACKTRACE ----- 2017-10-13T16:59:49.751+0000 I - [initandlisten] ***aborting after invariant() failure
Please let us know if there is any workaround.
Can I run "mongod —repair" here? Would it help?
Thanks,
Sandesh