Loading...

Type: Bug
Resolution: Done
Priority: Major - P3
Fix Version/s: None
Affects Version/s: 3.4.7
Component/s: WiredTiger
Labels:
- envns
- rge
- rpu
- trct
- wtc

Operating System:
Linux

service mongod unexpectedly changed state from "active" to "failed". repair tool did not help.

log file output:

2018-04-19T05:24:30.688+0300 I CONTROL  [main] ***** SERVER RESTARTED *****
2018-04-19T05:24:30.696+0300 I CONTROL  [initandlisten] MongoDB starting : pid=2648 port=27017 dbpath=/var/lib/mongodb 64-bit host=ColeteElastic
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] db version v3.4.7
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] allocator: tcmalloc
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] modules: none
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] build environment:
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten]     distmod: ubuntu1604
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten]     distarch: x86_64
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten]     target_arch: x86_64
2018-04-19T05:24:30.697+0300 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2018-04-19T05:24:30.697+0300 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2018-04-19T05:24:30.751+0300 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-04-19T05:24:30.751+0300 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2018-04-19T05:24:30.751+0300 I STORAGE  [initandlisten] 
2018-04-19T05:24:30.751+0300 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-04-19T05:24:30.751+0300 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-04-19T05:24:30.751+0300 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3475M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-04-19T05:24:30.762+0300 E STORAGE  [initandlisten] WiredTiger error (0) [1524104670:761987][2648:0x7f15a3c44d00], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
2018-04-19T05:24:30.762+0300 E STORAGE  [initandlisten] WiredTiger error (-31804) [1524104670:762051][2648:0x7f15a3c44d00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-04-19T05:24:30.762+0300 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2018-04-19T05:24:30.762+0300 I -        [initandlisten] 

***aborting after fassert() failure


2018-04-19T05:24:30.794+0300 F -        [initandlisten] Got signal: 6 (Aborted).

 0x556257693ea1 0x5562576930b9 0x55625769359d 0x7f15a2849390 0x7f15a24a3428 0x7f15a24a502a 0x556256942733 0x5562573a7c26 0x55625694c9cc 0x55625694cbf1 0x55625694ce49 0x55625803aafc 0x556258039269 0x556257fedd03 0x55625807865b 0x556258078b9d 0x556258078e5c 0x556257ffc4f1 0x55625806ea08 0x5562580389ae 0x556258038a8b 0x556257fea05d 0x55625738ca6f 0x5562573852ec 0x556257278a00 0x55625692e95e 0x55625694e1f6 0x7f15a248e830 0x5562569ace09
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"556256128000","o":"156BEA1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"556256128000","o":"156B0B9"},{"b":"556256128000","o":"156B59D"},{"b":"7F15A2838000","o":"11390"},{"b":"7F15A246E000","o":"35428","s":"gsignal"},{"b":"7F15A246E000","o":"3702A","s":"abort"},{"b":"556256128000","o":"81A733","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"556256128000","o":"127FC26"},{"b":"556256128000","o":"8249CC","s":"__wt_eventv"},{"b":"556256128000","o":"824BF1","s":"__wt_err"},{"b":"556256128000","o":"824E49","s":"__wt_panic"},{"b":"556256128000","o":"1F12AFC","s":"__wt_turtle_read"},{"b":"556256128000","o":"1F11269","s":"__wt_metadata_search"},{"b":"556256128000","o":"1EC5D03","s":"__wt_conn_btree_open"},{"b":"556256128000","o":"1F5065B","s":"__wt_session_get_btree"},{"b":"556256128000","o":"1F50B9D","s":"__wt_session_get_btree"},{"b":"556256128000","o":"1F50E5C","s":"__wt_session_get_btree_ckpt"},{"b":"556256128000","o":"1ED44F1","s":"__wt_curfile_open"},{"b":"556256128000","o":"1F46A08"},{"b":"556256128000","o":"1F109AE","s":"__wt_metadata_cursor_open"},{"b":"556256128000","o":"1F10A8B","s":"__wt_metadata_cursor"},{"b":"556256128000","o":"1EC205D","s":"wiredtiger_open"},{"b":"556256128000","o":"1264A6F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"556256128000","o":"125D2EC"},{"b":"556256128000","o":"1150A00","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"556256128000","o":"80695E"},{"b":"556256128000","o":"8261F6","s":"main"},{"b":"7F15A246E000","o":"20830","s":"__libc_start_main"},{"b":"556256128000","o":"884E09","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-92-generic", "version" : "#115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "556256128000", "elfType" : 3, "buildId" : "7E08C88DF63C9DEE479A2B1C6C7E11D8651F1184" }, { "b" : "7FFD781FB000", "elfType" : 3, "buildId" : "F23E2C79BCC8E97B12E1BB62A1BF196F8423FC40" }, { "b" : "7F15A37C4000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7F15A3380000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7F15A3178000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7F15A2F74000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7F15A2C6B000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7F15A2A55000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F15A2838000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7F15A246E000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7F15A3A2D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x556257693ea1]
 mongod(+0x156B0B9) [0x5562576930b9]
 mongod(+0x156B59D) [0x55625769359d]
 libpthread.so.0(+0x11390) [0x7f15a2849390]
 libc.so.6(gsignal+0x38) [0x7f15a24a3428]
 libc.so.6(abort+0x16A) [0x7f15a24a502a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x556256942733]
 mongod(+0x127FC26) [0x5562573a7c26]
 mongod(__wt_eventv+0x3D7) [0x55625694c9cc]
 mongod(__wt_err+0x9D) [0x55625694cbf1]
 mongod(__wt_panic+0x24) [0x55625694ce49]
 mongod(__wt_turtle_read+0x26C) [0x55625803aafc]
 mongod(__wt_metadata_search+0x99) [0x556258039269]
 mongod(__wt_conn_btree_open+0x73) [0x556257fedd03]
 mongod(__wt_session_get_btree+0xFB) [0x55625807865b]
 mongod(__wt_session_get_btree+0x63D) [0x556258078b9d]
 mongod(__wt_session_get_btree_ckpt+0x14C) [0x556258078e5c]
 mongod(__wt_curfile_open+0x161) [0x556257ffc4f1]
 mongod(+0x1F46A08) [0x55625806ea08]
 mongod(__wt_metadata_cursor_open+0x6E) [0x5562580389ae]
 mongod(__wt_metadata_cursor+0x4B) [0x556258038a8b]
 mongod(wiredtiger_open+0x164D) [0x556257fea05d]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x55625738ca6f]
 mongod(+0x125D2EC) [0x5562573852ec]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x556257278a00]
 mongod(+0x80695E) [0x55625692e95e]
 mongod(main+0x966) [0x55625694e1f6]
 libc.so.6(__libc_start_main+0xF0) [0x7f15a248e830]
 mongod(_start+0x29) [0x5562569ace09]
-----  END BACKTRACE  -----

(sudo mongod --repair --dbpath /var/log/mongodb/ --storageEngine wiredTiger)
--reapir output:

2018-04-19T05:48:43.620+0300 I CONTROL  [initandlisten] MongoDB starting : pid=2675 port=27017 dbpath=/var/log/mongodb/ 64-bit host=ColeteElastic
2018-04-19T05:48:43.620+0300 I CONTROL  [initandlisten] db version v3.4.7
2018-04-19T05:48:43.620+0300 I CONTROL  [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
2018-04-19T05:48:43.620+0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-04-19T05:48:43.620+0300 I CONTROL  [initandlisten] allocator: tcmalloc
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten] modules: none
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten] build environment:
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten]     distmod: ubuntu1604
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten]     distarch: x86_64
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten]     target_arch: x86_64
2018-04-19T05:48:43.621+0300 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/log/mongodb/", engine: "wiredTiger" } }
2018-04-19T05:48:43.674+0300 I STORAGE  [initandlisten] 
2018-04-19T05:48:43.674+0300 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-04-19T05:48:43.674+0300 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-04-19T05:48:43.674+0300 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3475M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),
2018-04-19T05:48:43.748+0300 I STORAGE  [initandlisten] Repairing size cache
2018-04-19T05:48:43.749+0300 I STORAGE  [initandlisten] Verify succeeded on uri table:sizeStorer. Not salvaging.
2018-04-19T05:48:43.749+0300 I STORAGE  [initandlisten] Repairing catalog metadata
2018-04-19T05:48:43.750+0300 I STORAGE  [initandlisten] Verify succeeded on uri table:_mdb_catalog. Not salvaging.
2018-04-19T05:48:43.765+0300 I CONTROL  [initandlisten] 
2018-04-19T05:48:43.765+0300 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-04-19T05:48:43.765+0300 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-04-19T05:48:43.765+0300 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2018-04-19T05:48:43.765+0300 I CONTROL  [initandlisten] 
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] 
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] 
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-04-19T05:48:43.766+0300 I CONTROL  [initandlisten] 
2018-04-19T05:48:43.766+0300 I STORAGE  [initandlisten] finished checking dbs
2018-04-19T05:48:43.766+0300 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2018-04-19T05:48:43.766+0300 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2018-04-19T05:48:43.766+0300 I NETWORK  [initandlisten] shutdown: going to flush diaglog...
2018-04-19T05:48:43.766+0300 I STORAGE  [initandlisten] WiredTigerKVEngine shutting down
2018-04-19T05:48:43.793+0300 I STORAGE  [initandlisten] shutdown: removing fs lock...
2018-04-19T05:48:43.793+0300 I CONTROL  [initandlisten] now exiting
2018-04-19T05:48:43.793+0300 I CONTROL  [initandlisten] shutting down with code:0

more info:
What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?

storage device attached over network, HDD, RAID (NetApp native)

Would you please check the integrity of your disks?

yes

Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.
-yes, always this version of mongo

Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?

no

Have you ever restored this instance from backups?

no

What method do you use to create backups?

there was no back-ups at the moment.

When was the underlying filesystem last checked and is it currently marked clean?

never

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

WiredTigerLAS.wt
4 kB
Apr 19 2018 02:59:21 AM UTC
WiredTiger.wt
244 kB
Apr 19 2018 02:59:21 AM UTC
WiredTiger.turtle
1.0 kB
Apr 19 2018 02:59:21 AM UTC
WiredTiger.lock
0.0 kB
Apr 19 2018 02:59:21 AM UTC
WiredTiger
0.0 kB
Apr 19 2018 02:59:21 AM UTC
repair_attempt.tar.gz
15 kB
Apr 19 2018 07:40:09 PM UTC

Details

Description

Attachments

Attachments

Activity

People

Dates