Thu Jun 16 16:43:34 [conn35976] core.Core_Model_User Assertion failure ! "DR102 too much data written uncommitted" db/dur_commitjob.cpp 204
0x51fe32 0x5305cf 0x5e6a28 0x5da88a 0x6947c8 0x698477 0x6948c2 0x60fb65 0x61193f 0x6cb29d 0x6d1520 0x6d23c2 0x548582 0x5cc648 0x35c3855112 0x35c385738f 0x35c38557b3 0x35c384f794 0x35c3855112 0x35c385738f
/usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xb2) [0x51fe32]
/usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0xbf) [0x5305cf]
/usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x198) [0x5e6a28]
/usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x5da88a]
/usr/bin/mongod(ZN5mongo12append_O_ObjEPcRKNS_7BSONObjES3+0x48) [0x6947c8]
/usr/bin/mongod [0x698477]
/usr/bin/mongod(_ZN5mongo5logOpEPKcS1_RKNS_7BSONObjEPS2_Pb+0x42) [0x6948c2]
/usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugEPNS_11RemoveSaverE+0x1f05) [0x60fb65]
/usr/bin/mongod(_ZN5mongo13updateObjectsEPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugE+0x12f) [0x61193f]
/usr/bin/mongod(_ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE+0x4dd) [0x6cb29d]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x15a0) [0x6d1520]
/usr/bin/mongod(_ZN5mongo14DBDirectClient3sayERNS_7MessageE+0x62) [0x6d23c2]
/usr/bin/mongod(_ZN5mongo12DBClientBase6updateERKSsNS_5QueryENS_7BSONObjEbb+0x242) [0x548582]
/usr/bin/mongod(ZN5mongo12mongo_updateEP9JSContextP8JSObjectjPlS4+0x1c8) [0x5cc648]
/usr/lib64/libjs.so.1(js_Invoke+0x492) [0x35c3855112]
/usr/lib64/libjs.so.1(js_Interpret+0x14df) [0x35c385738f]
/usr/lib64/libjs.so.1(js_Invoke+0xb33) [0x35c38557b3]
/usr/lib64/libjs.so.1 [0x35c384f794]
/usr/lib64/libjs.so.1(js_Invoke+0x492) [0x35c3855112]
/usr/lib64/libjs.so.1(js_Interpret+0x14df) [0x35c385738f]
Thu Jun 16 16:43:34 [conn35976] local.oplog.rs Assertion failure ! "DR102 too much data written uncommitted" db/dur_commitjob.cpp 204

This message was repeating in the logs (unfortunately we failed to save the logs prior to restart so we don't have any more info from them.

Trying to connect to the master failed:

PROD root@docbase1-10-125-50-69 ~ $ mongo localhost:27018
MongoDB shell version: 1.8.1
connecting to: localhost:27018/test
Thu Jun 16 16:39:05 MessagingPort recv() errno:104 Connection reset by peer 127.0.0.1:27018
Thu Jun 16 16:39:05 SocketException: remote: error: 9001 socket exception [1]
Thu Jun 16 16:39:05 DBClientCursor::init call() failed
exception: DBClientBase::findOne: transport error: localhost:27018

But it was still sending heartbeats

query:

{ whatsmyuri: 1 }

"members" : [
{
"_id" : 0,
"name" : "ec2-50-16-30-183.compute-1.amazonaws.com:27018",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 35043,
"optime" :

{ "t" : 1308256847000, "i" : 28 }

,
"optimeDate" : ISODate("2011-06-16T20:40:47Z"),
"lastHeartbeat" : ISODate("2011-06-16T20:40:47Z")
},
{
"_id" : 1,
"name" : "ec2-184-72-213-192.compute-1.amazonaws.com:27018",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"optime" :

{ "t" : 1308254621000, "i" : 7 }

,
"optimeDate" : ISODate("2011-06-16T20:03:41Z"),
"self" : true
},