The current upgrade mechanism for the database relies on the following ordering of binary upgrades:

• We first upgrade the config shard binaries
• We then upgrade the data shard binaries
• We then upgrade mongos binaries

With the possibility of having an embedded config shard we've now broken this assumption as a data shard might contain the config shard. As a result, a data shard running a newer binary can contact the config shard running an older binary within the same ReplicaSet. This can be more or less be seen with a secondary node having a newer binary stepping up to become a primary and then contacting a secondary with the older binary.

We currently do not have a testing suite that checks the behavior of the server with mixed version deployments and stepdowns occurring. We should ideally address this so we can catch errors before going to the user.