I'm creating this question at Ian Whalen's suggestion. It may be that some of this will become irrelevant if we decide to convert to replica sets, and I'm submitting it in part in case it might be of use to the MongoDB developers.

Below is the text of the relevant part of my email to Ian:

We are getting errors that indicate that our oplog has gone out of sync or something, the dates seem corrupted, and perhaps there is some issue with the slave. We haven't converted over to Replica set--we are using legacy master-slave replication.

Our question is: what can we do to fix this situation (see some detailed info below)? Can we do a repair? do we have to restart oplog from scratch? Do we need to just bite the bullet and learn how to upgrade to Replica sets? Notice the "upgradeNeeded": true from the db.slaves.find() command. I don't understand that--they are both now on 2.4.1.

Anyway, thank you so much for your time!

configured oplog size: 44792.567773437506MB
log length start to end: 5613.861999999965secs (1.56hrs)
oplog first event time: Fri Jan 16 1970 11:15:32 GMT-0600 (CST)
oplog last event time: Fri Jan 16 1970 12:49:06 GMT-0600 (CST)
now: Sun Mar 24 2013 18:04:52 GMT-0500 (CDT)

db.slaves.find()

{ "_id" : ObjectId("50f463aebc26d2d32eb20df1"), "host" : "72.233.16.182", "ns" : "local.oplog.$main", "syncedTo" :

{ "t" : 1363746775, "i" : 1 }

}

{ "_id" : ObjectId("514ea0359cf7fda50eceafa5"), "config" :

{ "host" : "72.233.16.182:37402", "upgradeNeeded" : true }

, "ns" : "local.oplog.$main", "syncedTo" :

{ "t" : 1363746785, "i" : 1 }

}

db.printSlaveReplicationInfo()

source: submongo.geekdo.com

syncedTo: Fri Jan 16 1970 12:49:06 GMT-0600 (CST)

= 1362799568 secs ago (378555.44hrs)

our oplog seems massively huge too

43 gb

In our slave log, we see things like:

Sun Mar 24 18:10:02.706 [replslave] repl: syncing from host:submongo.geekdo.com

so it seems to think it is syncing, but we don't really know whether to trust it.