Currently removeShard removes the entry in config.shards for the shard being removed and then calls ShardRegistry::reload(). It's possible that this reload actually joins an already running reload however, and so will still read in the entry just removed. We should check to see if the reload joined, and if so reload again.
Also, addShard currently inserts an entry into config.shards and then calls ShardRegistry::getShard() which checks for an entry for this shard and if it does not find one, reloads. It's possible that there is an old entry for the same shard in the ShardRegistry though, so we may decide not reload and leave the ShardRegistry in a stale state until the next refresh. To be sure the ShardRegistry picks up the newest targeter and RSM, we should force a refresh after inserting the doc.