A number of the unit tests in shard_server_catalog_cache_loader_test.cpp call get() on the notification returned by ShardServerCatalogCacheLoader::getChunksSince() and then finish the test. We set this notification immediately after scheduling a task to persist metadata to config.cache.collections and config.cache.chunks.db.collname, meaning we do not actually wait for the metadata to be persisted before setting the notification. When persisting metadata to config.cache.chunks.db.collname, we create this collection and then run createIndex() on it through DBDirectClient. However, in the destructor for the unit test fixture (ServiceContextMongoDTest), we call DatabaseHolderImpl::closeAll() which invariants that we are not currently building any indexes.

This started happening frequently once storage switched to the new index builds implementation because they are now on a new thread pool. When we shutdown the shard server catalog cache loader, we call join on all the threads in its thread pool. Previously index builds were synchronous, so when tearing down one of these unit tests we would join the shard loader's threads which would include any index builds. Now, if one of the tasks on the shard loader's thread pool creates a collection and thus creates an index on that collection, this index build will be scheduled on a thread pool on the index builds coordinator. So, when joining the threads on the shard loader we aren't joining any index builds as well. When tearing down the unit tests, we call shutdown on the shard loader first (which won't stop any index builds) and then attempt to close dbs before shutting down the index builds coordinator. We should actually shutdown the index builds coordinator before attempting to close dbs since we invariant that there are no index builds ongoing when closing dbs.