FCV 3.4->3.6 has no schema upgrade process for config.cache.collections.

config.cache.collections was added to shards as part of the safe secondary reads project, which didn't require an upgrade process, and safely started persisting routing metadata as soon as the v3.6 binary cluster started up. However, UUIDs were added to config.cache.collections to address replication change streams needs. I believe that a config config.collections schema upgrade process was added, but no one thought to add an upgrade process for shards.

So, ShardServerCatalogCacheLoader will happily persist config.cache.collections from the start. Then setFCV(3.6) happens, and config.cache.collections won't get the UUID until the shard refreshes NEW chunks for that collection.

The ShardServerCatalogCacheLoader won't schedule a persisted update unless new chunks are received from the config server, per this bit of code. That was added to improve secondary availability because every primary persistence task sets flags that block metadata reads, and might even prompt forcing the secondary to refresh if we got around to that (I forget).

Consider the in-memory UUID values in the upgrade process. ChunkManager is probably fine because it gets recreated/updated every CatalogCache refresh. Unsure if we have any other in-memory UUIDs fields that must get set on upgrade.