When running with --queryableBackupMode, WiredTigerKVEngine::initRsOplogBackgroundThread() should not start a thread, and therefore not perform oplog sampling.
In addition, we should consider adding a separate configuration option to disable this oplog sampling without needing to be running with --queryableBackupMode.
Original Description
In version 3.1.8 mongo added a new feature: SERVER-19551
On startup WiredTiger now needs to read values from the oplog to figure out where to place the milestones:
2016-04-26T15:42:22.066+0000 I STORAGE [initandlisten] The size storer reports that the oplog contains 19279531 records totaling to 53948096450 bytes 2016-04-26T15:42:22.410+0000 I STORAGE [initandlisten] Sampling from the oplog between Apr 23 22:02:46:71 and Apr 26 14:10:01:d1 to determine where to place markers for truncation 2016-04-26T15:42:22.410+0000 I STORAGE [initandlisten] Taking 1004 samples and assuming that each section of oplog contains approximately 191863 records totaling to 536872169 bytes
This process is fine if all of the data is stored locally on disk. The problem is when you are using external storage (for us, Amazon EBS). The sampling from cold EBS is taking somewhere between 20 and 30 minutes leading to incredibly slow startup times.
Is there a good way to speed up this process / bring startup times back to 3.1.7 levels for EBS backed instances?