Problem Description

The 2.2 release notes instructions for upgrading a replica set do not have any particular instructions for when using authentication (i.e. running with --keyfile). Given that the instructions for upgrading a shard cluster that immediately follow mention specific instructions for authentication, the strong impression is that there is nothing special to do for a replica set.

In a 2.0.x replica set, not in a sharded cluster and using authentication, when the secondary is dropped and then 2.2.0 started instead, the following error occurs (in the log file):

Thu Aug 30 14:47:37 [conn30] command denied: { replSetHeartbeat: "MyRS", v: 3, pv: 1, checkEmpty: false, from: "ian.local:30003" }
Thu Aug 30 14:47:37 [conn31] command denied: { replSetHeartbeat: "MyRS", v: 3, pv: 1, checkEmpty: false, from: "ian.local:30001" }

The entry for the secondary in rs.status() contains the following error:

"errmsg" : "unauthorized: no auth credentials provided for command and authenticated using internal user.  This is most likely because you are using an old version of mongos

The full rs.status() is:

MyRS:PRIMARY> rs.status()
{
	"set" : "MyRS",
	"date" : ISODate("2012-08-30T04:48:56Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "ian.local:30001",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"optime" : Timestamp(1346301568000, 1),
			"optimeDate" : ISODate("2012-08-30T04:39:28Z"),
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "ian.local:30002",
			"health" : 0,
			"state" : 8,
			"stateStr" : "(not reachable/healthy)",
			"uptime" : 0,
			"optime" : Timestamp(1346301568000, 1),
			"optimeDate" : ISODate("2012-08-30T04:39:28Z"),
			"lastHeartbeat" : ISODate("2012-08-30T04:48:55Z"),
			"pingMs" : 0,
			"errmsg" : "unauthorized: no auth credentials provided for command and authenticated using internal user.  This is most likely because you are using an old version of mongos"
		},
		{
			"_id" : 2,
			"name" : "ian.local:30003",
			"health" : 1,
			"state" : 7,
			"stateStr" : "ARBITER",
			"uptime" : 568,
			"optime" : Timestamp(0, 0),
			"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
			"lastHeartbeat" : ISODate("2012-08-30T04:48:55Z"),
			"pingMs" : 0
		}
	],
	"ok" : 1
}

Note that the error message in the secondary of the rs.status() mentions mongos, which implies a sharded cluster. There is no sharded cluster; just a single replica set.

Consequently if you proceed with dropping the other secondary nodes, as per our upgrade instructions, eventually the primary will demote itself to a secondary once there is no longer a majority. Thus there is downtime. You have to shutdown the primary then start it up again with the new version before the problem is resolved.

Workaround for 2.2.0

You cannot perform a rolling upgrade of a replica set using authentication from an earlier version to 2.2.0. You must shut down all nodes of the replica set first, then start each node with the 2.2.0 mongod binary.

Fix in 2.2.1

You will be able to perform a rolling upgrade of a replica set using authentication to 2.2.1 (and later versions). You will be able to shutdown each secondary one at a time and re-start it with the 2.2.1 (or later) binary. Then perform an rs.stepDown() on the primary, then shut it down and re-start it with the 2.2.1 (or later) binary.

Looking forward, it will be possible to do a rolling upgrade of a replica set using authentication from 2.0.x to 2.4, however only by first doing a rolling upgrade to 2.2.1 (or a later 2.2.x release) and then a rolling upgrade from 2.2.x to 2.4.x.