For some queries we use slaveOkay in the PHP driver. However, on shutting down the secondary, queries were still being routed to it even after the 5 second timeout (as per http://www.php.net/manual/en/mongo.queries.php). During this time rs3b was down and rs3a was up as the primary. rs3c and rs3d are also up but are hidden so are not used for slave queries.

Since reads are randomly distributed some queries were returning whilst others were erroring.

set3:PRIMARY> rs.status()
{
"set" : "set3",
"date" : ISODate("2011-01-30T13:31:43Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "rs3a:27018",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"optime" :

{ "t" : 1296394303000, "i" : 875 }

,
"optimeDate" : ISODate("2011-01-30T13:31:43Z"),
"self" : true
},
{
"_id" : 1,
"name" : "rs3b:27018",
"health" : 0,
"state" : 2,
"stateStr" : "(not reachable/healthy)",
"uptime" : 0,
"optime" :

{ "t" : 1296394194000, "i" : 959 }

,
"optimeDate" : ISODate("2011-01-30T13:29:54Z"),
"lastHeartbeat" : ISODate("2011-01-30T13:29:54Z"),
"errmsg" : "socket exception"
},
{
"_id" : 2,
"name" : "rs3c:27018",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 934,
"optime" :

{ "t" : 1296394302000, "i" : 323 }

,
"optimeDate" : ISODate("2011-01-30T13:31:42Z"),
"lastHeartbeat" : ISODate("2011-01-30T13:31:42Z")
},
{
"_id" : 3,
"name" : "rs3d:27018",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 934,
"optime" :

{ "t" : 1296387100000, "i" : 49 }

,
"optimeDate" : ISODate("2011-01-30T11:31:40Z"),
"lastHeartbeat" : ISODate("2011-01-30T13:31:42Z")
},
{
"_id" : 4,
"name" : "rs3arbiter:27018",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 934,
"optime" :

{ "t" : 0, "i" : 0 }

,
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2011-01-30T13:31:42Z")
}
],
"ok" : 1
}