The consistency checker performs checks even on dropped collections (this is because dropped collections still appear in config.collections with dropped: true). This can lead to noisy NamespaceNotFound in logs on the config server.

original description:

 

cluster topology:

sharded cluster with 1 config node (primary replica set) , 2 different databearing nodes  (primary replica set) and 1 router.

 

We upgraded our mongo cluster  from 4.0.9 to 4.2.8 and a couple of days later to 4.4.0

In the config logs we see every 10 minutes like below (attached mongod log of config).

The databases mentioned are supposed to be dropped from the data bearing nodes.

I'm finding  entries in config.collections (config) or  config.cached.collections (data bearing nodes) that reference to already dropped db's.

Also initiated flushrouterconfig but that didn't help.

4.2.8

2020-08-09T03:11:01.807+0200 I SHARDING [PeriodicShardedIndexConsistencyChecker] Checking consistency of sharded collection indexes across the cluster
2020-08-09T03:11:01.824+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database acc took 0 ms and failed :: caused by :: NamespaceNotFound: database acc not found
2020-08-09T03:11:01.828+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database lukes-test took 0 ms and failed :: caused by :: NamespaceNotFound: database lukes-test not found
2020-08-09T03:11:01.830+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_1492084538520 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_1492084538520 not found
2020-08-09T03:11:01.831+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_1492085358785 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_1492085358785 not found
2020-08-09T03:11:01.831+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_1492093702966 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_1492093702966 not found
2020-08-09T03:11:01.831+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_1492094907720 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_1492094907720 not found
.
.
.
2020-08-09T03:11:02.874+0200 I  SH_REFR  [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_2020-08-07-00-00-46 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-00-46 not found
2020-08-09T03:11:02.874+0200 I  SH_REFR  [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_2020-08-07-00-02-08 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-02-08 not found
2020-08-09T03:11:02.874+0200 I  SH_REFR  [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_2020-08-07-00-02-08 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-02-08 not found
2020-08-09T03:11:02.874+0200 I  SH_REFR  [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_2020-08-07-00-02-08 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-02-08 not found
2020-08-09T03:11:02.875+0200 I  SH_REFR  [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_2020-08-07-00-04-07 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-04-07 not found
2020-08-09T03:11:02.875+0200 I  SH_REFR  [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_2020-08-07-00-04-07 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-04-07 not found
2020-08-09T03:11:02.875+0200 I  SH_REFR  [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_2020-08-07-00-04-07 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-04-07 not found
2020-08-09T03:11:02.875+0200 I  SHARDING [PeriodicShardedIndexConsistencyChecker] Found 0 collections with inconsistent indexes

 

 also seen after an upgrade to 4.4.0 but there is not end message from the Checker that 0 collections are found. 

The sharded cluster was succesfully upgraded around 10:40 CEST

faced an issue and use the workaround as mentioned in https://jira.mongodb.org/browse/SERVER-50137

 

{"t":{"$date":"2020-08-11T03:00:23.147+02:00"},"s":"I",  "c":"SHARDING", "id":22049,   "ctx":"PeriodicShardedIndexConsistencyChecker","msg":"Checking consistency of sharded collection indexes across the cluster"}
{"t":{"$date":"2020-08-11T03:00:23.163+02:00"},"s":"I",  "c":"SH_REFR",  "id":24100,   "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"acc","durationMillis":0,"error":"NamespaceNotFound: database acc not found"}}
{"t":{"$date":"2020-08-11T03:00:23.163+02:00"},"s":"I",  "c":"SH_REFR",  "id":24100,   "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"at-dev","durationMillis":0,"error":"NamespaceNotFound: database at-dev not found"}}
{"t":{"$date":"2020-08-11T03:00:23.166+02:00"},"s":"I",  "c":"SH_REFR",  "id":24100,   "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-test","durationMillis":0,"error":"NamespaceNotFound: database ctr-test not found"}}
{"t":{"$date":"2020-08-11T03:00:23.167+02:00"},"s":"I",  "c":"SH_REFR",  "id":24100,   "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_1492084538520","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_1492084538520 not found"}}
{"t":{"$date":"2020-08-11T03:00:23.167+02:00"},"s":"I",  "c":"SH_REFR",  "id":24100,   "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_1492085358785","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_1492085358785 not found"}}

.
.
.
{"t":{"$date":"2020-08-11T03:00:24.234+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-07-00-04-07","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-04-07 not found"}}
{"t":{"$date":"2020-08-11T03:00:24.234+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-07-00-04-07","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-04-07 not found"}}
{"t":{"$date":"2020-08-11T03:00:24.234+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-07-00-04-07","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-04-07 not found"}}
{"t":{"$date":"2020-08-11T03:00:24.235+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-10-11-50-03","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-10-11-50-03 not found"}}
{"t":{"$date":"2020-08-11T03:00:24.235+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-10-11-50-03","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-10-11-50-03 not found"}}
{"t":{"$date":"2020-08-11T03:00:24.235+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-10-11-50-03","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-10-11-50-03 not found"}}
{"t":{"$date":"2020-08-11T03:00:24.235+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-11-00-01-39","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-11-00-01-39 not found"}}
{"t":{"$date":"2020-08-11T03:00:24.235+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-11-00-01-39","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-11-00-01-39 not found"}}
{"t":{"$date":"2020-08-11T03:00:24.236+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-11-00-01-39","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-11-00-01-39 not found"}}
{"t":{"$date":"2020-08-11T03:00:24.292+02:00"},"s":"I", "c":"SHARDING", "id":20997, "ctx":"conn4315","msg":"Refreshed RWC defaults","attr":{"newDefaults":{}}}