-
Type: Bug
-
Resolution: Done
-
Priority: Major - P3
-
Affects Version/s: 2.8.0-rc5
-
Component/s: Storage
-
Fully Compatible
-
ALL
During one 10-minute run of a heavy mixed workload a period of several seconds of very low throughput was seen:
- During the pause all statistics appear to be consistent with a large number of cache evictions, resulting in a lot of writes, system cpu time, and context switches. This heavy eviction load seems like a plausible cause for the low throughput during this period.
- Immediately before the pause the number of bytes in the cache jumped by about 4 GiB between two samples about 0.1 seconds apart. This is a plausible trigger for the heavy eviction load.
- During the pause bytes and pages in cache declined slowly, presumably a result of the page evictions.
- The pause ends with a sudden drop in the size of the cache by about 4 GiB between two samples about 0.1 seconds apart.
Here are the specific samples
- just before increase
"localTime":"2015-01-15T18:42:20.344Z"
"bytes currently in the cache":4315898330
"pages currently held in the cache":60756
- just after increase
"localTime":"2015-01-15T18:42:20.472Z"
"bytes currently in the cache":8594367123
"pages currently held in the cache":60598
- just before decrease
"localTime":"2015-01-15T18:42:25.058Z"
"bytes currently in the cache":7242022268
"pages currently held in the cache":35169
- just after decrease
"localTime":"2015-01-15T18:42:25.170Z"
"bytes currently in the cache":2941126278
"pages currently held in the cache":35137
The size of the sudden jump was: 7242022268 - 2941126278 = 3.98 GiB
The size of the sudden drop was: 8594367123 - 4315898330 = 4.01 GiB
Neither a sudden increase nor a sudden decrease of cache size by 4 GiB in 0.1 seconds due to real cached data is plausible.