ISSUE SUMMARY
After an ungraceful termination, a mongod running WiredTiger may fail to read the journal and fail to start, showing the following messages in the logs:
2015-03-16T14:49:24.992-0400 W - [initandlisten] Detected unclean shutdown - /data/wiredTiger/mongod.lock is not empty. 2015-03-16T14:49:24.992-0400 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. ... 2015-03-16T14:49:25.075-0400 F - [initandlisten] Invalid access at address: 0
WORKAROUNDS
Please upgrade to to 3.0.2.
Alternatively, a mongod from 3.0.0-rc8 can be used to recover the data files, but this is not recommended.
AFFECTED VERSIONS
3.0.0-rc8+ and 3.0.1
FIX VERSION
The fix is included in the 3.0.2 production release.
Original description
As part of the Power cycle testing (see SERVER-17152), the following was discovered:
jonathan@CAP-JA-Ubuntu14:~$ sudo mongod --dbpath /data/wiredTiger --storageEngine wiredTiger 2015-03-16T14:49:24.992-0400 W - [initandlisten] Detected unclean shutdown - /data/wiredTiger/mongod.lock is not empty. 2015-03-16T14:49:24.992-0400 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2015-03-16T14:49:24.992-0400 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2015-03-16T14:49:25.075-0400 F - [initandlisten] Invalid access at address: 0 2015-03-16T14:49:25.083-0400 F - [initandlisten] Got signal: 11 (Segmentation fault). 0xf4fd39 0xf4f3b2 0xf4f70e 0x7f4bac390340 0x13576e8 0x132ad0d 0x132b31b 0x133a2ca 0x138c8a5 0x131b3c1 0x13150dc 0xd62cab 0xd60b28 0xa6f8bd 0x7e20c0 0x7e7704 0x7f4bab3b0ec5 0x7e02c9 ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"B4FD39"},{"b":"400000","o":"B4F3B2"},{"b":"400000","o":"B4F70E"},{"b":"7F4BAC380000","o":"10340"},{"b":"400000","o":"F576E8"},{"b":"400000","o":"F2AD0D"},{"b":"400000","o":"F2B31B"},{"b":"400000","o":"F3A2CA"},{"b":"400000","o":"F8C8A5"},{"b":"400000","o":"F1B3C1"},{"b":"400000","o":"F150DC"},{"b":"400000","o":"962CAB"},{"b":"400000","o":"960B28"},{"b":"400000","o":"66F8BD"},{"b":"400000","o":"3E20C0"},{"b":"400000","o":"3E7704"},{"b":"7F4BAB38F000","o":"21EC5"},{"b":"400000","o":"3E02C9"}],"processInfo":{ "mongodbVersion" : "3.0.1-rc0", "gitVersion" : "3f999d764cdd307f779b27e3a28230b21bd43e25", "uname" : { "sysname" : "Linux", "release" : "3.16.0-31-generic", "version" : "#41~14.04.1-Ubuntu SMP Wed Feb 11 19:30:13 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF64135000", "elfType" : 3 }, { "b" : "7F4BAC380000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F4BAC178000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F4BABF74000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F4BABC70000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F4BAB96A000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F4BAB754000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F4BAB38F000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F4BAC59E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf4fd39] mongod(+0xB4F3B2) [0xf4f3b2] mongod(+0xB4F70E) [0xf4f70e] libpthread.so.0(+0x10340) [0x7f4bac390340] mongod(__wt_struct_unpack+0x368) [0x13576e8] mongod(+0xF2AD0D) [0x132ad0d] mongod(+0xF2B31B) [0x132b31b] mongod(__wt_log_needs_recovery+0xCA) [0x133a2ca] mongod(__wt_txn_recover+0x425) [0x138c8a5] mongod(__wt_connection_workers+0x61) [0x131b3c1] mongod(wiredtiger_open+0x118C) [0x13150dc] mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x2EB) [0xd62cab] mongod(+0x960B28) [0xd60b28] mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x30D) [0xa6f8bd] mongod(_ZN5mongo13initAndListenEi+0x2F0) [0x7e20c0] mongod(main+0x134) [0x7e7704] libc.so.6(__libc_start_main+0xF5) [0x7f4bab3b0ec5] mongod(+0x3E02C9) [0x7e02c9] ----- END BACKTRACE -----
- is depended on by
-
WT-1901 Issues resolved in WiredTiger 2.5.3
- Closed
- is duplicated by
-
SERVER-17824 Mongo server seagfault
- Closed
-
SERVER-17569 Got signal: 11 (Segmentation fault).
- Closed
- is related to
-
SERVER-17152 WiredTiger file corrupted during power cycle test
- Closed