-
Type:
Bug
-
Resolution: Done
-
Priority:
Critical - P2
-
Affects Version/s: 3.0.0-rc9
-
Component/s: Storage, WiredTiger
-
None
-
Fully Compatible
-
ALL
-
ISSUE SUMMARY
A bug in the WiredTiger algorithm for page management may cause the server to abort with a message like the following:
2015-02-21T07:08:08.397-0800 E STORAGE [WiredTigerRecordStoreThread for local.oplog.$main] WiredTiger (22) [1424531288:395397][1514015:0x7f903a7ff700], session.truncate: the start cursor position is after the stop cursor position: Invalid argument 2015-02-21T07:08:08.399-0800 I - [WiredTigerRecordStoreThread for local.oplog.$main] Invariant failure: ret resulted in status BadValue 22: Invalid argument at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 518
USER IMPACT
The mongod process aborts after the invariant failure and needs to be restarted. This can only impact capped collections.
WORKAROUNDS
N/A
AFFECTED VERSIONS
MongoDB 3.0.0 running with the WiredTiger storage engine.
FIX VERSION
The fix is included in the 3.0.1 production release.
RESOLUTION DETAILS
WiredTiger no longer incorrectly positions the start cursor during the truncate operation.
Original description
I get this from iibench-mongodb with 10 insert threads, 1 query thread and zlib after about 300M docs have been inserted. It did not occur with snappy. Prior to this crash there were many stalls (slow inserts) and that was also only an issue with zlib. This appears to be different than the "session.truncate" crash that occurred prior to rc9 (see https://jira.mongodb.org/browse/SERVER-17141)
log has this content:
2015-02-21T07:08:08.397-0800 E STORAGE [WiredTigerRecordStoreThread for local.oplog.$main] WiredTiger (22) [1424531288:395397][1514015:0x7f903a7ff700], session.truncate: the start cursor position is after the stop cursor position: Invalid argument
2015-02-21T07:08:08.399-0800 I - [WiredTigerRecordStoreThread for local.oplog.$main] Invariant failure: ret resulted in status BadValue 22: Invalid argument at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 518
2015-02-21T07:08:08.409-0800 I CONTROL [WiredTigerRecordStoreThread for local.oplog.$main]
0x117b00b 0x1114881 0x10f7fb8 0xf267df 0xf2cbbf 0x10f941e 0x11c8358 0x7f904931e7c9 0x7f90490578ad 0
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"D7B00B"},{"b":"400000","o":"D14881"},{"b":"400000","o":"CF7FB8"},{"b":"400000","o":"B267DF"},{"b":"400000","o":"B2CBBF"},{"b":"400000","o":"CF941E"},{"b":"400000","o":"DC8358"},{"b":"7F9049317000","o":"77C9"},{"b":"7F9048F4B000","o":"10C8AD"},{"b":"0","o":"0"}],"processInfo":{ "mongodbVersion" : "3.0.0-rc9", "gitVersion" : "nogitversion", "uname" : { "sysname" : "Linux", "release" : "3.10.53-43_fbk14_02627_gb5c286c", "version" : "#1 SMP Mon Oct 20 15:06:57 PDT 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "9D8BCA7928778A629A6BD7DD44CF150933BC2E78" }, { "b" : "7FFFA18FE000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "50103D521098AC1A8DA637FE9F84CCFE48B62B25" }, { "b" : "7F9049F87000", "path" : "/usr/local/fbcode/gcc-4.9-glibc-2.20/lib/librt.so.1", "elfType" : 3, "buildId" : "36E417C497D7D99E323324B54709FFF95E21F209" }, { "b" : "7F9049D83000", "path" : "/usr/local/fbcode/gcc-4.9-glibc-2.20/lib/libdl.so.2", "elfType" : 3, "buildId" : "7258590522FF8998FF3DC9CD4800354FF634B049" }, { "b" : "7F9049A52000", "path" : "/usr/local/fbcode/gcc-4.9-glibc-2.20/lib/libstdc++.so.6", "elfType" : 3, "buildId" : "AADD96ABFCFED89F9C3343339F009D973AC5822F" }, { "b" : "7F904974B000", "path" : "/usr/local/fbcode/gcc-4.9-glibc-2.20/lib/libm.so.6", "elfType" : 3, "buildId" : "5180B0D6214EFC6BD4024BCC46459868BB9FF312" }, { "b" : "7F9049534000", "path" : "/usr/local/fbcode/gcc-4.9-glibc-2.20/lib/libgcc_s.so.1", "elfType" : 3, "buildId" : "CAE88F8F821F6A585CC52F17AA185621534F442C" }, { "b" : "7F9049317000", "path" : "/usr/local/fbcode/gcc-4.9-glibc-2.20/lib/libpthread.so.0", "elfType" : 3, "buildId" : "F3CF4B6D065A6FCF55D752C1B7FBF951C041B871" }, { "b" : "7F9048F4B000", "path" : "/usr/local/fbcode/gcc-4.9-glibc-2.20/lib/libc.so.6", "elfType" : 3, "buildId" : "C645843801A68480D80BF9D0FD4A20182CBD1D9C" }, { "b" : "7F904A190000", "path" : "/usr/local/fbcode/gcc-4.9-glibc-2.20/lib/ld.so", "elfType" : 3, "buildId" : "FAF6C327CBADE1769B09F51830B64480E9FFDC82" } ] }}
mongod.30rc9.jem.norox(_ZN5mongo15printStackTraceERSo+0x2B) [0x117b00b]
mongod.30rc9.jem.norox(_ZN5mongo10logContextEPKc+0xE1) [0x1114881]
mongod.30rc9.jem.norox(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x108) [0x10f7fb8]
mongod.30rc9.jem.norox(_ZN5mongo21WiredTigerRecordStore27cappedDeleteAsNeeded_inlockEPNS_16OperationContextERKNS_8RecordIdE+0x8EF) [0xf267df]
mongod.30rc9.jem.norox(+0xB2CBBF) [0xf2cbbf]
mongod.30rc9.jem.norox(_ZN5mongo13BackgroundJob7jobBodyEv+0x11E) [0x10f941e]
mongod.30rc9.jem.norox(+0xDC8358) [0x11c8358]
libpthread.so.0(+0x77C9) [0x7f904931e7c9]
libc.so.6(clone+0x6D) [0x7f90490578ad]
??? [0]
----- END BACKTRACE -----
2015-02-21T07:08:08.409-0800 I - [WiredTigerRecordStoreThread for local.oplog.$main]
***aborting after invariant() failure
- duplicates
-
-
- Closed
-
- is duplicated by
-
-
- Closed
-