I was investigating an issue whereby creating a new collection was taking around 100 seconds. When I looked at the mongos logs I noticed that there was message of following type for every sharded collection in the database:

ChunkManager: time to load chunks for ddp_rel.timeseries.58c142b39340386c4d618779: 249ms sequenceNumber: 12117 version: 3|1||58c142b72810c82e0b145d5f based on: (empty)

(I have attached log).

After investigating I realised that my shard tag was incorrect because it only had the collection name and not the full namespace including the database name.

After that, it worked better as each ChunkManager statement was taking 1ms. However, I am concerned that if we have > 10,000 collections (which is quite feasible as this is a time series database with collection per time series) it will still take a long time.

Can someone explain to me why this is happening? It states "based on: (empty)" but I did testing by loading dummy data, and it still happens even if the sharded collection is not empty.

Any explanation would be greatly appreciated, and if possible a way to prevent this happening.

As FYI, we use sharded tags so that the data is pinned to location and shard.