Hi,

We've noticed that on one of our sharded collection we cannot do any non-specific queries. For example:

db.log_service.find() returns nothing, querying on anything other than only _id fails.
db.log_service.find(

{"_id":ObjectId("4eb3bf877961dc244d008235")}

) returns the document.

The environment is sharded over 3 replicaton sets, all of which currently only have a primary and an arbiter. The secondary has been taken down due to issues non-related to this. The configuration of the replicasets has been adjusted and rs.reconfig() has been run on all primaries. We have noticed however that mongos keeps complaining in the logs about not being able to reach these servers. Not sure if this is related to the issue.

Having previously discussed this on IRC, I've prepared a few pastes:

db.printShardingStatus();
http://pastie.org/private/pufkfflhtkwifmgcq8aq

mongos log extract:
http://pastie.org/private/93el01g9nocz9ocfiqzdw

rs.status() on replSetB primary (the only shard that has data for that collection as an automated script removed the data from the other shards, not from this one as aparently it couldn't find any):
http://pastie.org/private/an8g0gun7l6kevffusmytw

explain() on the working query:
db.log_service.find(

{"_id":ObjectId("4eb3bf877961dc244d008235")}

)
http://pastie.org/private/edfgbh9vkhlwf7o8tjmdq

explain() on the failing query:
db.log_service.find()
http://pastie.org/private/igeth69f84agwdjhmycyqa

We first noticed the issue when we couldn't query data with skip() set to a number higher than what was available on replSetA. At that time find() without limit and skip worked. Now, replSetA is empty and these stopped working.
Queries with skip() higher than item count on replSetA for different collections works without issues.

Best regards,
Koen