$queryStats is an aggregation query that does a read-only scan of the query stats cache, which is implemented as a collection of 16MB partitions. Even at a generous cache size of 300MB (the expected size on Atlas clusters), this results in 19 partitions.

If there are many $queryStats aggregation queries coming in to take a lock on each partition, the entire cache becomes a bottleneck for concurrent find commands, which perform short-running read-update-writes to one specific partition.

Currently, this is being mitigated by minimizing the lock time for the $queryStats aggregation queries by copying the entire partition out, effectively creating a read-only view of the partition. This still requires copying 16MB under a lock, however, and the impact can be very visible. (See attachment, which shows massive performance degradation with 32 concurrent $queryStats aggregations).

Although we do not expect many concurrent $queryStats aggregations at the moment, this can change in the future. And if it does, we will want to consider further improvements in the contention between the long-running read-only $queryStats aggregations and the short-running read-update-write find commands.

Some basic ideas to possibly consider:

1. Finer grained locking for hash tables (to allow for more partitions) can decrease the contention between readers and writers. Since $queryStats aggregation throughput is not particularly time-sensitive, but find command throughput is, it's reasonable to trade long-running read-only performance losses for short-running read-write-update performance gains.
2. Lock-free data structures can provide some help here. Lock-free queues can help with LRU implementation.