There is a race condition in ShardServerCatalogCacheLoader, where if a shard node is running as a primary and a step-down happens, it may read in-memory task queue and persisted cache state, which is not consistent.

Specifically, consider a node which is a primary and found some data reading from the config server here. It will then schedule this data to be persisted to the cache collections and then will proceed to do a merge of the task queue + what's already persisted in order to produce a list of the changed chunks.

In a stepdown-free case, this would work fine. However, if by the time it got to read what it persisted and what is on the queue, the node stepped down, neither the write to the cache collections could have happened, nor anything remained on the task queue because of the change in term. That way it could come back with incomplete data (which would be a data loss) or it could come back with an empty list, which will invariant.

In order to fix it, after reading from the task queue + persisted cache, we should check if the term has changed here and throw ConflictingOperationInProgress error so the load can be retried as secondary.