-
Type: Bug
-
Resolution: Works as Designed
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Sharding
-
None
-
Fully Compatible
-
ALL
We ran sharded mongo cluster on Mongo 4.0.13 on Ubuntu 18.04 x64 .
A single (non-sharded) mongo was initially added to cluster (srv211), later 3 other mongo servers were added and collections were moved to 3 new shards. srv211 shard was removed, all mongos services were restarted (shard/cfg instances were not).
But suddenly after several months of uptime, the cluster was blocked and we got the following error messages in one of our shard logs:
{{2019-12-30T19:35:18.079+0000 I SHARDING [conn123235] received splitChunk request: { splitChunk: "project_prod_user.player_maps", from: "project-db-3", keyPattern:
{ pers_id: "hashed" }, epoch: ObjectId('5dafd8dc63ae2c18230f9dce'), shardVersion: [ Timestamp(10814, 0), ObjectId('5dafd8dc63ae2c18230f9dce') ], min: { pers_id: 4618768445950617740 }, max: { pers_id: 4620991399317296785 }, splitKeys: [ { pers_id: 4619791202844536983 }, { pers_id: 4620989830407096034 } ], $clusterTime: { clusterTime: Timestamp(1577734518, 5), signature:
{ hash: BinData(0, 4219F00DC9FE995B5CBBBC9B5E39EBFF35C51A10), keyId: 6750833113431015451 }}, $configServerState: { opTime:
{ ts: Timestamp(1577734517, 30), t: 5 } }, $db: "admin" }}}
2019-12-30T19:35:18.085+0000 I SHARDING [conn123235] distributed lock 'project_prod_user.player_maps' acquired for 'splitting chunk [{ pers_id: 4618768445950617740 }, { pers_id: 4620991399317296785 }) in project_prod_user.player_maps', ts : 5e0a5176dc25d0219069dd07
2019-12-30T19:35:18.136+0000 I SHARDING [conn123235] distributed lock with ts: 5e0a5176dc25d0219069dd07' unlocked.
2019-12-30T19:35:18.148+0000 I SHARDING [conn125292] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.148+0000 I SHARDING [conn125292] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.149+0000 I SHARDING [conn125258] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.150+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.150+0000 I SHARDING [conn125295] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.151+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.151+0000 I SHARDING [conn125295] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.151+0000 I SHARDING [conn125258] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.151+0000 I SHARDING [conn125258] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.152+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.152+0000 I SHARDING [conn125295] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.153+0000 I SHARDING [conn125258] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.153+0000 I SHARDING [conn125258] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.153+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.153+0000 I SHARDING [conn125295] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.154+0000 I SHARDING [conn125258] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.154+0000 I SHARDING [conn125258] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.154+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.154+0000 I SHARDING [conn125295] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.155+0000 I SHARDING [conn125258] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.155+0000 I SHARDING [conn125258] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.155+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.155+0000 I SHARDING [conn125295] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.156+0000 I SHARDING [conn125258] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.156+0000 I SHARDING [conn125258] Failed to handle stale version exception :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.168+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.171+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
2019-12-30T19:35:18.174+0000 I SHARDING [conn125295] Failed to refresh metadata for collectionproject_prod_user.player_maps :: caused by :: ShardNotFound: Shard srv211 not found
I've examined config database, but found no mentions of srv211 in db.chunks or db.shards.
Is it a known issue?