-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
None
-
Affects Version/s: 3.1.6
-
Component/s: WiredTiger
-
None
-
ALL
-
-
0
While running the load phase of a YCSB test, wiredTiger encountered an error and panic. This has happened 3 out of 4 times so it seems fairly reproducible. The workload is doing a large number of inserts of documents that are 1K each. The mongod log reports some slow inserts (>100 msec) and then showed the following line about wiredTiger. This was seen on an EC2 instance running Enterprise version of mongod (3.1.7-pre, git version: 1b5b55cb48ff9daca74177b86d336613d207c5d1).
015-08-11T08:53:47.184+0000 E STORAGE [thread1] WiredTiger (-31802) [1439283227:184443][18776:0x7fdf70a65700], file:WiredTiger.wt, WT_SESSION.checkpoint: WiredTiger.wt read error: failed to read 4096 bytes at offset 32768: WT_ERROR: non-specific WiredTiger error 2015-08-11T08:53:47.184+0000 E STORAGE [thread1] WiredTiger (-31802) [1439283227:184629][18776:0x7fdf70a65700], checkpoint-server: checkpoint server error: WT_ERROR: non-specific WiredTiger error 2015-08-11T08:53:47.184+0000 E STORAGE [thread1] WiredTiger (-31804) [1439283227:184661][18776:0x7fdf70a65700], checkpoint-server: the process must exit and restart: WT_PANIC: WiredTiger library panic 2015-08-11T08:53:47.184+0000 I - [conn9] Fatal Assertion 28559 2015-08-11T08:53:47.184+0000 I - [conn16] Fatal Assertion 28559
One the console where the YCSB test were run, these error repeated many times around the problem and are mostly related. Part of this error message that is interesting is where it says "Caused by: java.io.EOFException), which could be related to the error in mongod log.
com.mongodb.MongoException$Network: Operation on server localhost:27017 failed
at com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:215)
at com.mongodb.DBCollectionImpl.writeWithCommandProtocol(DBCollectionImpl.java:461)
at com.mongodb.DBCollectionImpl.insertWithCommandProtocol(DBCollectionImpl.java:426)
at com.mongodb.DBCollectionImpl.insert(DBCollectionImpl.java:188)
at com.mongodb.DBCollectionImpl.insert(DBCollectionImpl.java:167)
at com.mongodb.DBCollection.insert(DBCollection.java:93)
at com.mongodb.DBCollection.insert(DBCollection.java:78)
at com.mongodb.DBCollection.insert(DBCollection.java:120)
at com.yahoo.ycsb.db.MongoDbClient.insert(MongoDbClient.java:246)
at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:148)
at com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:461)
at com.yahoo.ycsb.ClientThread.run(Client.java:269)
Caused by: java.io.EOFException
at org.bson.io.Bits.readFully(Bits.java:75)
at org.bson.io.Bits.readFully(Bits.java:50)
at org.bson.io.Bits.readFully(Bits.java:37)
at com.mongodb.Response.<init>(Response.java:42)
at com.mongodb.DBCollectionImpl.receiveWriteCommandMessage(DBCollectionImpl.java:532)
at com.mongodb.DBCollectionImpl.access$300(DBCollectionImpl.java:48)
at com.mongodb.DBCollectionImpl$2.execute(DBCollectionImpl.java:474)
at com.mongodb.DBCollectionImpl$2.execute(DBCollectionImpl.java:461)
at com.mongodb.DBPort.doOperation(DBPort.java:187)
at com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:208)
... 11 more