Sat Nov 17 09:42:17.743 [initandlisten] MongoDB starting : pid=30585 port=27999 dbpath=/data/db/sconsTests/ 64-bit host=domU-12-31-39-16-30-A2 Sat Nov 17 09:42:17.743 [initandlisten] Sat Nov 17 09:42:17.743 [initandlisten] ** NOTE: This is a development version (2.3.1-pre-) of MongoDB. Sat Nov 17 09:42:17.743 [initandlisten] ** Not recommended for production. Sat Nov 17 09:42:17.744 [initandlisten] Sat Nov 17 09:42:17.744 [initandlisten] db version v2.3.1-pre-, pdfile version 4.5 Sat Nov 17 09:42:17.744 [initandlisten] git version: 38d64e3ffb528a597fb13d8aa459cd21f0967490 Sat Nov 17 09:42:17.744 [initandlisten] build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 BOOST_LIB_VERSION=1_49 Sat Nov 17 09:42:17.744 [initandlisten] options: { dbpath: "/data/db/sconsTests/", port: 27999 } Sat Nov 17 09:42:17.749 [initandlisten] journal dir=/data/db/sconsTests/journal Sat Nov 17 09:42:17.749 [initandlisten] recover : no journal files present, no recovery needed Sat Nov 17 09:42:18.037 [websvr] admin web console waiting for connections on port 28999 Sat Nov 17 09:42:18.038 [initandlisten] waiting for connections on port 27999 Sat Nov 17 09:42:18.661 [initandlisten] connection accepted from 127.0.0.1:52480 #1 (1 connection now open) Sat Nov 17 09:42:18.662 [conn1] end connection 127.0.0.1:52480 (0 connections now open) MongoDB shell version: 2.3.1-pre- null Replica set test! ReplSetTest Starting Set ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 31100, 31101, 31102 ] 31100 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31100, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "clusteredstale-rs0", "dbpath" : "$set-$node", "useHostname" : true, "noJournalPrealloc" : undefined, "pathOpts" : { "testName" : "clusteredstale", "shard" : 0, "node" : 0, "set" : "clusteredstale-rs0" }, "restart" : undefined } ReplSetTest Starting.... Resetting db path '/data/db/clusteredstale-rs0-0' Sat Nov 17 09:42:19.189 shell: started program /mnt/data/slaves/Linux_64bit_Legacy/mongo/mongod --oplogSize 40 --port 31100 --noprealloc --smallfiles --rest --replSet clusteredstale-rs0 --dbpath /data/db/clusteredstale-rs0-0 m31100| note: noprealloc may hurt performance in many applications m31100| Sat Nov 17 09:42:19.260 [initandlisten] MongoDB starting : pid=30603 port=31100 dbpath=/data/db/clusteredstale-rs0-0 64-bit host=domU-12-31-39-16-30-A2 m31100| Sat Nov 17 09:42:19.260 [initandlisten] m31100| Sat Nov 17 09:42:19.260 [initandlisten] ** NOTE: This is a development version (2.3.1-pre-) of MongoDB. m31100| Sat Nov 17 09:42:19.260 [initandlisten] ** Not recommended for production. m31100| Sat Nov 17 09:42:19.260 [initandlisten] m31100| Sat Nov 17 09:42:19.260 [initandlisten] db version v2.3.1-pre-, pdfile version 4.5 m31100| Sat Nov 17 09:42:19.260 [initandlisten] git version: 38d64e3ffb528a597fb13d8aa459cd21f0967490 m31100| Sat Nov 17 09:42:19.260 [initandlisten] build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 BOOST_LIB_VERSION=1_49 m31100| Sat Nov 17 09:42:19.260 [initandlisten] options: { dbpath: "/data/db/clusteredstale-rs0-0", noprealloc: true, oplogSize: 40, port: 31100, replSet: "clusteredstale-rs0", rest: true, smallfiles: true } m31100| Sat Nov 17 09:42:19.265 [initandlisten] journal dir=/data/db/clusteredstale-rs0-0/journal m31100| Sat Nov 17 09:42:19.265 [initandlisten] recover : no journal files present, no recovery needed m31100| Sat Nov 17 09:42:19.330 [initandlisten] waiting for connections on port 31100 m31100| Sat Nov 17 09:42:19.330 [websvr] admin web console waiting for connections on port 32100 m31100| Sat Nov 17 09:42:19.333 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31100| Sat Nov 17 09:42:19.333 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done [ connection to domU-12-31-39-16-30-A2:31100 ] ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 31100, 31101, 31102 ] 31101 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31101, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "clusteredstale-rs0", "dbpath" : "$set-$node", "useHostname" : true, "noJournalPrealloc" : undefined, "pathOpts" : { "testName" : "clusteredstale", "shard" : 0, "node" : 1, "set" : "clusteredstale-rs0" }, "restart" : undefined } ReplSetTest Starting.... Resetting db path '/data/db/clusteredstale-rs0-1' Sat Nov 17 09:42:19.398 shell: started program /mnt/data/slaves/Linux_64bit_Legacy/mongo/mongod --oplogSize 40 --port 31101 --noprealloc --smallfiles --rest --replSet clusteredstale-rs0 --dbpath /data/db/clusteredstale-rs0-1 m31100| Sat Nov 17 09:42:19.399 [initandlisten] connection accepted from 127.0.0.1:49214 #1 (1 connection now open) m31101| note: noprealloc may hurt performance in many applications m31101| Sat Nov 17 09:42:19.476 [initandlisten] MongoDB starting : pid=30652 port=31101 dbpath=/data/db/clusteredstale-rs0-1 64-bit host=domU-12-31-39-16-30-A2 m31101| Sat Nov 17 09:42:19.476 [initandlisten] m31101| Sat Nov 17 09:42:19.476 [initandlisten] ** NOTE: This is a development version (2.3.1-pre-) of MongoDB. m31101| Sat Nov 17 09:42:19.476 [initandlisten] ** Not recommended for production. m31101| Sat Nov 17 09:42:19.476 [initandlisten] m31101| Sat Nov 17 09:42:19.476 [initandlisten] db version v2.3.1-pre-, pdfile version 4.5 m31101| Sat Nov 17 09:42:19.476 [initandlisten] git version: 38d64e3ffb528a597fb13d8aa459cd21f0967490 m31101| Sat Nov 17 09:42:19.476 [initandlisten] build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 BOOST_LIB_VERSION=1_49 m31101| Sat Nov 17 09:42:19.476 [initandlisten] options: { dbpath: "/data/db/clusteredstale-rs0-1", noprealloc: true, oplogSize: 40, port: 31101, replSet: "clusteredstale-rs0", rest: true, smallfiles: true } m31101| Sat Nov 17 09:42:19.479 [initandlisten] journal dir=/data/db/clusteredstale-rs0-1/journal m31101| Sat Nov 17 09:42:19.480 [initandlisten] recover : no journal files present, no recovery needed m31101| Sat Nov 17 09:42:19.543 [initandlisten] waiting for connections on port 31101 m31101| Sat Nov 17 09:42:19.543 [websvr] admin web console waiting for connections on port 32101 m31101| Sat Nov 17 09:42:19.546 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31101| Sat Nov 17 09:42:19.546 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done [ connection to domU-12-31-39-16-30-A2:31100, connection to domU-12-31-39-16-30-A2:31101 ] ReplSetTest n is : 2 ReplSetTest n: 2 ports: [ 31100, 31101, 31102 ] 31102 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31102, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "clusteredstale-rs0", "dbpath" : "$set-$node", "useHostname" : true, "noJournalPrealloc" : undefined, "pathOpts" : { "testName" : "clusteredstale", "shard" : 0, "node" : 2, "set" : "clusteredstale-rs0" }, "restart" : undefined } ReplSetTest Starting.... Resetting db path '/data/db/clusteredstale-rs0-2' Sat Nov 17 09:42:19.608 shell: started program /mnt/data/slaves/Linux_64bit_Legacy/mongo/mongod --oplogSize 40 --port 31102 --noprealloc --smallfiles --rest --replSet clusteredstale-rs0 --dbpath /data/db/clusteredstale-rs0-2 m31101| Sat Nov 17 09:42:19.609 [initandlisten] connection accepted from 127.0.0.1:37028 #1 (1 connection now open) m31102| note: noprealloc may hurt performance in many applications m31102| Sat Nov 17 09:42:19.679 [initandlisten] MongoDB starting : pid=30702 port=31102 dbpath=/data/db/clusteredstale-rs0-2 64-bit host=domU-12-31-39-16-30-A2 m31102| Sat Nov 17 09:42:19.679 [initandlisten] m31102| Sat Nov 17 09:42:19.679 [initandlisten] ** NOTE: This is a development version (2.3.1-pre-) of MongoDB. m31102| Sat Nov 17 09:42:19.679 [initandlisten] ** Not recommended for production. m31102| Sat Nov 17 09:42:19.679 [initandlisten] m31102| Sat Nov 17 09:42:19.679 [initandlisten] db version v2.3.1-pre-, pdfile version 4.5 m31102| Sat Nov 17 09:42:19.679 [initandlisten] git version: 38d64e3ffb528a597fb13d8aa459cd21f0967490 m31102| Sat Nov 17 09:42:19.679 [initandlisten] build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 BOOST_LIB_VERSION=1_49 m31102| Sat Nov 17 09:42:19.679 [initandlisten] options: { dbpath: "/data/db/clusteredstale-rs0-2", noprealloc: true, oplogSize: 40, port: 31102, replSet: "clusteredstale-rs0", rest: true, smallfiles: true } m31102| Sat Nov 17 09:42:19.695 [initandlisten] journal dir=/data/db/clusteredstale-rs0-2/journal m31102| Sat Nov 17 09:42:19.695 [initandlisten] recover : no journal files present, no recovery needed m31102| Sat Nov 17 09:42:19.766 [initandlisten] waiting for connections on port 31102 m31102| Sat Nov 17 09:42:19.766 [websvr] admin web console waiting for connections on port 32102 m31102| Sat Nov 17 09:42:19.769 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31102| Sat Nov 17 09:42:19.769 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31102| Sat Nov 17 09:42:19.810 [initandlisten] connection accepted from 127.0.0.1:55069 #1 (1 connection now open) [ connection to domU-12-31-39-16-30-A2:31100, connection to domU-12-31-39-16-30-A2:31101, connection to domU-12-31-39-16-30-A2:31102 ] { "replSetInitiate" : { "_id" : "clusteredstale-rs0", "members" : [ { "_id" : 0, "host" : "domU-12-31-39-16-30-A2:31100" }, { "_id" : 1, "host" : "domU-12-31-39-16-30-A2:31101" }, { "_id" : 2, "host" : "domU-12-31-39-16-30-A2:31102" } ] } } m31100| Sat Nov 17 09:42:19.815 [conn1] replSet replSetInitiate admin command received from client m31100| Sat Nov 17 09:42:19.815 [conn1] replSet replSetInitiate config object parses ok, 3 members specified m31101| Sat Nov 17 09:42:19.816 [initandlisten] connection accepted from 127.0.0.1:37033 #2 (2 connections now open) m31102| Sat Nov 17 09:42:19.826 [initandlisten] connection accepted from 127.0.0.1:55071 #2 (2 connections now open) m31100| Sat Nov 17 09:42:19.827 [conn1] replSet replSetInitiate all members seem up m31100| Sat Nov 17 09:42:19.827 [conn1] ****** m31100| Sat Nov 17 09:42:19.827 [conn1] creating replication oplog of size: 40MB... m31100| Sat Nov 17 09:42:19.829 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-0/local.ns, filling with zeroes... m31100| Sat Nov 17 09:42:19.829 [FileAllocator] creating directory /data/db/clusteredstale-rs0-0/_tmp m31100| Sat Nov 17 09:42:20.040 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-0/local.ns, size: 16MB, took 0.209 secs m31100| Sat Nov 17 09:42:20.041 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-0/local.0, filling with zeroes... m31100| Sat Nov 17 09:42:20.955 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-0/local.0, size: 64MB, took 0.914 secs m31100| Sat Nov 17 09:42:21.011 [conn1] ****** m31100| Sat Nov 17 09:42:21.011 [conn1] replSet info saving a newer config version to local.system.replset m31100| Sat Nov 17 09:42:21.012 [conn1] replSet saveConfigLocally done m31100| Sat Nov 17 09:42:21.012 [conn1] replSet replSetInitiate config now saved locally. Should come online in about a minute. m31100| Sat Nov 17 09:42:21.012 [conn1] command admin.$cmd command: { replSetInitiate: { _id: "clusteredstale-rs0", members: [ { _id: 0.0, host: "domU-12-31-39-16-30-A2:31100" }, { _id: 1.0, host: "domU-12-31-39-16-30-A2:31101" }, { _id: 2.0, host: "domU-12-31-39-16-30-A2:31102" } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:1185611 reslen:112 1197ms { "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1 } Replica set test! ReplSetTest Starting Set ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 31200, 31201, 31202 ] 31200 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31200, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "clusteredstale-rs1", "dbpath" : "$set-$node", "useHostname" : true, "noJournalPrealloc" : undefined, "pathOpts" : { "testName" : "clusteredstale", "shard" : 1, "node" : 0, "set" : "clusteredstale-rs1" }, "restart" : undefined } ReplSetTest Starting.... Resetting db path '/data/db/clusteredstale-rs1-0' Sat Nov 17 09:42:21.098 shell: started program /mnt/data/slaves/Linux_64bit_Legacy/mongo/mongod --oplogSize 40 --port 31200 --noprealloc --smallfiles --rest --replSet clusteredstale-rs1 --dbpath /data/db/clusteredstale-rs1-0 m31200| note: noprealloc may hurt performance in many applications m31200| Sat Nov 17 09:42:21.170 [initandlisten] MongoDB starting : pid=30758 port=31200 dbpath=/data/db/clusteredstale-rs1-0 64-bit host=domU-12-31-39-16-30-A2 m31200| Sat Nov 17 09:42:21.170 [initandlisten] m31200| Sat Nov 17 09:42:21.170 [initandlisten] ** NOTE: This is a development version (2.3.1-pre-) of MongoDB. m31200| Sat Nov 17 09:42:21.170 [initandlisten] ** Not recommended for production. m31200| Sat Nov 17 09:42:21.170 [initandlisten] m31200| Sat Nov 17 09:42:21.170 [initandlisten] db version v2.3.1-pre-, pdfile version 4.5 m31200| Sat Nov 17 09:42:21.170 [initandlisten] git version: 38d64e3ffb528a597fb13d8aa459cd21f0967490 m31200| Sat Nov 17 09:42:21.170 [initandlisten] build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 BOOST_LIB_VERSION=1_49 m31200| Sat Nov 17 09:42:21.170 [initandlisten] options: { dbpath: "/data/db/clusteredstale-rs1-0", noprealloc: true, oplogSize: 40, port: 31200, replSet: "clusteredstale-rs1", rest: true, smallfiles: true } m31200| Sat Nov 17 09:42:21.173 [initandlisten] journal dir=/data/db/clusteredstale-rs1-0/journal m31200| Sat Nov 17 09:42:21.173 [initandlisten] recover : no journal files present, no recovery needed m31200| Sat Nov 17 09:42:21.239 [websvr] admin web console waiting for connections on port 32200 m31200| Sat Nov 17 09:42:21.242 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31200| Sat Nov 17 09:42:21.242 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31200| Sat Nov 17 09:42:21.252 [initandlisten] waiting for connections on port 31200 [ connection to domU-12-31-39-16-30-A2:31200 ] ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 31200, 31201, 31202 ] 31201 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31201, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "clusteredstale-rs1", "dbpath" : "$set-$node", "useHostname" : true, "noJournalPrealloc" : undefined, "pathOpts" : { "testName" : "clusteredstale", "shard" : 1, "node" : 1, "set" : "clusteredstale-rs1" }, "restart" : undefined } ReplSetTest Starting.... Resetting db path '/data/db/clusteredstale-rs1-1' Sat Nov 17 09:42:21.306 shell: started program /mnt/data/slaves/Linux_64bit_Legacy/mongo/mongod --oplogSize 40 --port 31201 --noprealloc --smallfiles --rest --replSet clusteredstale-rs1 --dbpath /data/db/clusteredstale-rs1-1 m31200| Sat Nov 17 09:42:21.307 [initandlisten] connection accepted from 127.0.0.1:34073 #1 (1 connection now open) m31201| note: noprealloc may hurt performance in many applications m31201| Sat Nov 17 09:42:21.380 [initandlisten] MongoDB starting : pid=30806 port=31201 dbpath=/data/db/clusteredstale-rs1-1 64-bit host=domU-12-31-39-16-30-A2 m31201| Sat Nov 17 09:42:21.381 [initandlisten] m31201| Sat Nov 17 09:42:21.381 [initandlisten] ** NOTE: This is a development version (2.3.1-pre-) of MongoDB. m31201| Sat Nov 17 09:42:21.381 [initandlisten] ** Not recommended for production. m31201| Sat Nov 17 09:42:21.381 [initandlisten] m31201| Sat Nov 17 09:42:21.381 [initandlisten] db version v2.3.1-pre-, pdfile version 4.5 m31201| Sat Nov 17 09:42:21.381 [initandlisten] git version: 38d64e3ffb528a597fb13d8aa459cd21f0967490 m31201| Sat Nov 17 09:42:21.381 [initandlisten] build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 BOOST_LIB_VERSION=1_49 m31201| Sat Nov 17 09:42:21.381 [initandlisten] options: { dbpath: "/data/db/clusteredstale-rs1-1", noprealloc: true, oplogSize: 40, port: 31201, replSet: "clusteredstale-rs1", rest: true, smallfiles: true } m31201| Sat Nov 17 09:42:21.382 [initandlisten] journal dir=/data/db/clusteredstale-rs1-1/journal m31201| Sat Nov 17 09:42:21.382 [initandlisten] recover : no journal files present, no recovery needed m31201| Sat Nov 17 09:42:21.447 [initandlisten] waiting for connections on port 31201 m31201| Sat Nov 17 09:42:21.448 [websvr] admin web console waiting for connections on port 32201 m31201| Sat Nov 17 09:42:21.450 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31201| Sat Nov 17 09:42:21.450 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done [ connection to domU-12-31-39-16-30-A2:31200, connection to domU-12-31-39-16-30-A2:31201 ] ReplSetTest n is : 2 ReplSetTest n: 2 ports: [ 31200, 31201, 31202 ] 31202 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31202, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "clusteredstale-rs1", "dbpath" : "$set-$node", "useHostname" : true, "noJournalPrealloc" : undefined, "pathOpts" : { "testName" : "clusteredstale", "shard" : 1, "node" : 2, "set" : "clusteredstale-rs1" }, "restart" : undefined } ReplSetTest Starting.... Resetting db path '/data/db/clusteredstale-rs1-2' Sat Nov 17 09:42:21.514 shell: started program /mnt/data/slaves/Linux_64bit_Legacy/mongo/mongod --oplogSize 40 --port 31202 --noprealloc --smallfiles --rest --replSet clusteredstale-rs1 --dbpath /data/db/clusteredstale-rs1-2 m31201| Sat Nov 17 09:42:21.515 [initandlisten] connection accepted from 127.0.0.1:36280 #1 (1 connection now open) m31202| note: noprealloc may hurt performance in many applications m31202| Sat Nov 17 09:42:21.576 [initandlisten] MongoDB starting : pid=30855 port=31202 dbpath=/data/db/clusteredstale-rs1-2 64-bit host=domU-12-31-39-16-30-A2 m31202| Sat Nov 17 09:42:21.576 [initandlisten] m31202| Sat Nov 17 09:42:21.576 [initandlisten] ** NOTE: This is a development version (2.3.1-pre-) of MongoDB. m31202| Sat Nov 17 09:42:21.576 [initandlisten] ** Not recommended for production. m31202| Sat Nov 17 09:42:21.576 [initandlisten] m31202| Sat Nov 17 09:42:21.576 [initandlisten] db version v2.3.1-pre-, pdfile version 4.5 m31202| Sat Nov 17 09:42:21.576 [initandlisten] git version: 38d64e3ffb528a597fb13d8aa459cd21f0967490 m31202| Sat Nov 17 09:42:21.576 [initandlisten] build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 BOOST_LIB_VERSION=1_49 m31202| Sat Nov 17 09:42:21.576 [initandlisten] options: { dbpath: "/data/db/clusteredstale-rs1-2", noprealloc: true, oplogSize: 40, port: 31202, replSet: "clusteredstale-rs1", rest: true, smallfiles: true } m31202| Sat Nov 17 09:42:21.580 [initandlisten] journal dir=/data/db/clusteredstale-rs1-2/journal m31202| Sat Nov 17 09:42:21.580 [initandlisten] recover : no journal files present, no recovery needed m31202| Sat Nov 17 09:42:21.631 [initandlisten] waiting for connections on port 31202 m31202| Sat Nov 17 09:42:21.631 [websvr] admin web console waiting for connections on port 32202 m31202| Sat Nov 17 09:42:21.633 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31202| Sat Nov 17 09:42:21.633 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done [ connection to domU-12-31-39-16-30-A2:31200, connection to domU-12-31-39-16-30-A2:31201, connection to domU-12-31-39-16-30-A2:31202 ] { "replSetInitiate" : { "_id" : "clusteredstale-rs1", "members" : [ { "_id" : 0, "host" : "domU-12-31-39-16-30-A2:31200" }, { "_id" : 1, "host" : "domU-12-31-39-16-30-A2:31201" }, { "_id" : 2, "host" : "domU-12-31-39-16-30-A2:31202" } ] } } m31200| Sat Nov 17 09:42:21.737 [conn1] replSet replSetInitiate admin command received from client m31200| Sat Nov 17 09:42:21.738 [conn1] replSet replSetInitiate config object parses ok, 3 members specified m31201| Sat Nov 17 09:42:21.739 [initandlisten] connection accepted from 127.0.0.1:36283 #2 (2 connections now open) m31202| Sat Nov 17 09:42:21.739 [initandlisten] connection accepted from 127.0.0.1:60805 #1 (1 connection now open) m31202| Sat Nov 17 09:42:21.739 [initandlisten] connection accepted from 127.0.0.1:60807 #2 (2 connections now open) m31200| Sat Nov 17 09:42:21.740 [conn1] replSet replSetInitiate all members seem up m31200| Sat Nov 17 09:42:21.740 [conn1] ****** m31200| Sat Nov 17 09:42:21.740 [conn1] creating replication oplog of size: 40MB... m31200| Sat Nov 17 09:42:21.740 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-0/local.ns, filling with zeroes... m31200| Sat Nov 17 09:42:21.740 [FileAllocator] creating directory /data/db/clusteredstale-rs1-0/_tmp m31200| Sat Nov 17 09:42:21.965 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-0/local.ns, size: 16MB, took 0.223 secs m31200| Sat Nov 17 09:42:21.965 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-0/local.0, filling with zeroes... m31200| Sat Nov 17 09:42:22.850 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-0/local.0, size: 64MB, took 0.884 secs m31200| Sat Nov 17 09:42:22.914 [conn1] ****** m31200| Sat Nov 17 09:42:22.915 [conn1] replSet info saving a newer config version to local.system.replset m31200| Sat Nov 17 09:42:22.917 [conn1] replSet saveConfigLocally done m31200| Sat Nov 17 09:42:22.917 [conn1] replSet replSetInitiate config now saved locally. Should come online in about a minute. { "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1 } m31200| Sat Nov 17 09:42:22.917 [conn1] command admin.$cmd command: { replSetInitiate: { _id: "clusteredstale-rs1", members: [ { _id: 0.0, host: "domU-12-31-39-16-30-A2:31200" }, { _id: 1.0, host: "domU-12-31-39-16-30-A2:31201" }, { _id: 2.0, host: "domU-12-31-39-16-30-A2:31202" } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:1177118 reslen:112 1179ms m31100| Sat Nov 17 09:42:29.334 [rsStart] replSet I am domU-12-31-39-16-30-A2:31100 m31100| Sat Nov 17 09:42:29.334 [rsStart] replSet STARTUP2 m31100| Sat Nov 17 09:42:29.334 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is up m31100| Sat Nov 17 09:42:29.335 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is up m31100| Sat Nov 17 09:42:29.547 [initandlisten] connection accepted from 127.0.0.1:49234 #2 (2 connections now open) m31101| Sat Nov 17 09:42:29.547 [rsStart] trying to contact domU-12-31-39-16-30-A2:31100 m31101| Sat Nov 17 09:42:29.548 [rsStart] replSet I am domU-12-31-39-16-30-A2:31101 m31101| Sat Nov 17 09:42:29.548 [rsStart] replSet got config version 1 from a remote, saving locally m31101| Sat Nov 17 09:42:29.548 [rsStart] replSet info saving a newer config version to local.system.replset m31101| Sat Nov 17 09:42:29.548 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/local.ns, filling with zeroes... m31101| Sat Nov 17 09:42:29.548 [FileAllocator] creating directory /data/db/clusteredstale-rs0-1/_tmp m31102| Sat Nov 17 09:42:29.770 [rsStart] trying to contact domU-12-31-39-16-30-A2:31100 m31100| Sat Nov 17 09:42:29.770 [initandlisten] connection accepted from 127.0.0.1:49237 #3 (3 connections now open) m31102| Sat Nov 17 09:42:29.771 [rsStart] replSet I am domU-12-31-39-16-30-A2:31102 m31102| Sat Nov 17 09:42:29.771 [rsStart] replSet got config version 1 from a remote, saving locally m31102| Sat Nov 17 09:42:29.771 [rsStart] replSet info saving a newer config version to local.system.replset m31102| Sat Nov 17 09:42:29.773 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/local.ns, filling with zeroes... m31102| Sat Nov 17 09:42:29.773 [FileAllocator] creating directory /data/db/clusteredstale-rs0-2/_tmp m31101| Sat Nov 17 09:42:29.799 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/local.ns, size: 16MB, took 0.244 secs m31101| Sat Nov 17 09:42:29.799 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/local.0, filling with zeroes... m31101| Sat Nov 17 09:42:30.089 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/local.0, size: 16MB, took 0.289 secs m31102| Sat Nov 17 09:42:30.109 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/local.ns, size: 16MB, took 0.309 secs m31101| Sat Nov 17 09:42:30.110 [rsStart] replSet saveConfigLocally done m31101| Sat Nov 17 09:42:30.110 [rsStart] replSet STARTUP2 m31101| Sat Nov 17 09:42:30.111 [rsSync] ****** m31101| Sat Nov 17 09:42:30.111 [rsSync] creating replication oplog of size: 40MB... m31102| Sat Nov 17 09:42:30.110 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/local.0, filling with zeroes... m31102| Sat Nov 17 09:42:30.319 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/local.0, size: 16MB, took 0.208 secs m31101| Sat Nov 17 09:42:30.325 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/local.1, filling with zeroes... m31100| Sat Nov 17 09:42:30.336 [rsSync] replSet SECONDARY m31102| Sat Nov 17 09:42:30.388 [rsStart] replSet saveConfigLocally done m31102| Sat Nov 17 09:42:30.389 [rsStart] replSet STARTUP2 m31102| Sat Nov 17 09:42:30.389 [rsSync] ****** m31102| Sat Nov 17 09:42:30.389 [rsSync] creating replication oplog of size: 40MB... m31102| Sat Nov 17 09:42:30.687 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/local.1, filling with zeroes... m31200| Sat Nov 17 09:42:31.242 [rsStart] replSet I am domU-12-31-39-16-30-A2:31200 m31200| Sat Nov 17 09:42:31.243 [rsStart] replSet STARTUP2 m31200| Sat Nov 17 09:42:31.245 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31202 is up m31200| Sat Nov 17 09:42:31.245 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31201 is up m31101| Sat Nov 17 09:42:31.267 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/local.1, size: 64MB, took 0.941 secs m31100| Sat Nov 17 09:42:31.336 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 thinks that we are down m31100| Sat Nov 17 09:42:31.336 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state STARTUP2 m31100| Sat Nov 17 09:42:31.336 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31101 thinks that we are down m31100| Sat Nov 17 09:42:31.336 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is now in state STARTUP2 m31100| Sat Nov 17 09:42:31.336 [rsMgr] not electing self, domU-12-31-39-16-30-A2:31102 would veto m31100| Sat Nov 17 09:42:31.336 [rsMgr] not electing self, domU-12-31-39-16-30-A2:31102 would veto m31201| Sat Nov 17 09:42:31.451 [rsStart] trying to contact domU-12-31-39-16-30-A2:31200 m31200| Sat Nov 17 09:42:31.452 [initandlisten] connection accepted from 127.0.0.1:34085 #2 (2 connections now open) m31201| Sat Nov 17 09:42:31.453 [rsStart] replSet I am domU-12-31-39-16-30-A2:31201 m31201| Sat Nov 17 09:42:31.453 [rsStart] replSet got config version 1 from a remote, saving locally m31201| Sat Nov 17 09:42:31.453 [rsStart] replSet info saving a newer config version to local.system.replset m31201| Sat Nov 17 09:42:31.453 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-1/local.ns, filling with zeroes... m31201| Sat Nov 17 09:42:31.453 [FileAllocator] creating directory /data/db/clusteredstale-rs1-1/_tmp m31101| Sat Nov 17 09:42:31.550 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31100 is up m31101| Sat Nov 17 09:42:31.550 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31100 is now in state SECONDARY m31102| Sat Nov 17 09:42:31.550 [initandlisten] connection accepted from 127.0.0.1:55088 #3 (3 connections now open) m31101| Sat Nov 17 09:42:31.552 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 thinks that we are down m31101| Sat Nov 17 09:42:31.552 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is up m31101| Sat Nov 17 09:42:31.552 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state STARTUP2 m31202| Sat Nov 17 09:42:31.634 [rsStart] trying to contact domU-12-31-39-16-30-A2:31200 m31200| Sat Nov 17 09:42:31.635 [initandlisten] connection accepted from 127.0.0.1:34089 #3 (3 connections now open) m31202| Sat Nov 17 09:42:31.636 [rsStart] replSet I am domU-12-31-39-16-30-A2:31202 m31202| Sat Nov 17 09:42:31.636 [rsStart] replSet got config version 1 from a remote, saving locally m31202| Sat Nov 17 09:42:31.636 [rsStart] replSet info saving a newer config version to local.system.replset m31202| Sat Nov 17 09:42:31.636 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-2/local.ns, filling with zeroes... m31202| Sat Nov 17 09:42:31.636 [FileAllocator] creating directory /data/db/clusteredstale-rs1-2/_tmp m31102| Sat Nov 17 09:42:31.787 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31100 is up m31102| Sat Nov 17 09:42:31.787 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31100 is now in state SECONDARY m31101| Sat Nov 17 09:42:31.787 [initandlisten] connection accepted from 127.0.0.1:37055 #3 (3 connections now open) m31102| Sat Nov 17 09:42:31.788 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is up m31102| Sat Nov 17 09:42:31.788 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is now in state STARTUP2 m31102| Sat Nov 17 09:42:32.056 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/local.1, size: 64MB, took 1.368 secs m31200| Sat Nov 17 09:42:32.245 [rsSync] replSet SECONDARY m31202| Sat Nov 17 09:42:32.397 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-2/local.ns, size: 16MB, took 0.335 secs m31101| Sat Nov 17 09:42:32.438 [rsSync] ****** m31101| Sat Nov 17 09:42:32.438 [rsSync] replSet initial sync pending m31101| Sat Nov 17 09:42:32.438 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31201| Sat Nov 17 09:42:32.439 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-1/local.ns, size: 16MB, took 0.347 secs m31202| Sat Nov 17 09:42:32.398 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-2/local.0, filling with zeroes... m31201| Sat Nov 17 09:42:32.439 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-1/local.0, filling with zeroes... m31202| Sat Nov 17 09:42:32.912 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-2/local.0, size: 16MB, took 0.513 secs m31201| Sat Nov 17 09:42:32.938 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-1/local.0, size: 16MB, took 0.498 secs m31102| Sat Nov 17 09:42:32.938 [rsSync] ****** m31102| Sat Nov 17 09:42:32.938 [rsSync] replSet initial sync pending m31102| Sat Nov 17 09:42:32.938 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31202| Sat Nov 17 09:42:32.938 [rsStart] replSet saveConfigLocally done m31202| Sat Nov 17 09:42:32.940 [rsStart] replSet STARTUP2 m31202| Sat Nov 17 09:42:32.941 [rsSync] ****** m31202| Sat Nov 17 09:42:32.941 [rsSync] creating replication oplog of size: 40MB... m31202| Sat Nov 17 09:42:32.942 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-2/local.1, filling with zeroes... m31201| Sat Nov 17 09:42:33.193 [rsStart] replSet saveConfigLocally done m31201| Sat Nov 17 09:42:33.194 [rsStart] replSet STARTUP2 m31201| Sat Nov 17 09:42:33.194 [rsSync] ****** m31201| Sat Nov 17 09:42:33.194 [rsSync] creating replication oplog of size: 40MB... m31200| Sat Nov 17 09:42:33.246 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31201 thinks that we are down m31200| Sat Nov 17 09:42:33.246 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31202 thinks that we are down m31200| Sat Nov 17 09:42:33.246 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31202 is now in state STARTUP2 m31200| Sat Nov 17 09:42:33.247 [rsMgr] not electing self, domU-12-31-39-16-30-A2:31202 would veto m31200| Sat Nov 17 09:42:33.247 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31201 is now in state STARTUP2 m31200| Sat Nov 17 09:42:33.248 [rsMgr] not electing self, domU-12-31-39-16-30-A2:31202 would veto m31201| Sat Nov 17 09:42:33.454 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31200 is up m31201| Sat Nov 17 09:42:33.454 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31200 is now in state SECONDARY m31202| Sat Nov 17 09:42:33.455 [initandlisten] connection accepted from 127.0.0.1:60820 #3 (3 connections now open) m31201| Sat Nov 17 09:42:33.455 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31202 thinks that we are down m31201| Sat Nov 17 09:42:33.455 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31202 is up m31201| Sat Nov 17 09:42:33.457 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31202 is now in state STARTUP2 m31202| Sat Nov 17 09:42:33.637 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31200 is up m31202| Sat Nov 17 09:42:33.637 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31200 is now in state SECONDARY m31201| Sat Nov 17 09:42:33.637 [initandlisten] connection accepted from 127.0.0.1:36298 #3 (3 connections now open) m31202| Sat Nov 17 09:42:33.638 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31201 is up m31202| Sat Nov 17 09:42:33.638 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31201 is now in state STARTUP2 m31202| Sat Nov 17 09:42:33.871 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-2/local.1, size: 64MB, took 0.929 secs m31201| Sat Nov 17 09:42:33.913 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-1/local.1, filling with zeroes... m31202| Sat Nov 17 09:42:33.931 [rsSync] ****** m31202| Sat Nov 17 09:42:33.931 [rsSync] replSet initial sync pending m31202| Sat Nov 17 09:42:33.931 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31201| Sat Nov 17 09:42:34.807 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-1/local.1, size: 64MB, took 0.877 secs m31201| Sat Nov 17 09:42:34.877 [rsSync] ****** m31201| Sat Nov 17 09:42:34.877 [rsSync] replSet initial sync pending m31201| Sat Nov 17 09:42:34.877 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31100| Sat Nov 17 09:42:37.338 [rsMgr] replSet info electSelf 0 m31102| Sat Nov 17 09:42:37.338 [conn2] replSet RECOVERING m31102| Sat Nov 17 09:42:37.338 [conn2] replSet info voting yea for domU-12-31-39-16-30-A2:31100 (0) m31101| Sat Nov 17 09:42:37.338 [conn2] replSet RECOVERING m31101| Sat Nov 17 09:42:37.338 [conn2] replSet info voting yea for domU-12-31-39-16-30-A2:31100 (0) m31100| Sat Nov 17 09:42:37.338 [rsMgr] replSet PRIMARY m31101| Sat Nov 17 09:42:37.552 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31100 is now in state PRIMARY m31101| Sat Nov 17 09:42:37.554 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state RECOVERING m31102| Sat Nov 17 09:42:37.810 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is now in state RECOVERING m31102| Sat Nov 17 09:42:37.811 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31100 is now in state PRIMARY m31100| Sat Nov 17 09:42:38.948 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-0/admin.ns, filling with zeroes... m31100| Sat Nov 17 09:42:39.153 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-0/admin.ns, size: 16MB, took 0.204 secs m31100| Sat Nov 17 09:42:39.154 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-0/admin.0, filling with zeroes... m31200| Sat Nov 17 09:42:39.252 [rsMgr] replSet info electSelf 0 m31200| Sat Nov 17 09:42:39.252 [rsMgr] replSet PRIMARY m31201| Sat Nov 17 09:42:39.252 [conn2] replSet RECOVERING m31201| Sat Nov 17 09:42:39.252 [conn2] replSet info voting yea for domU-12-31-39-16-30-A2:31200 (0) m31202| Sat Nov 17 09:42:39.252 [conn2] replSet RECOVERING m31202| Sat Nov 17 09:42:39.252 [conn2] replSet info voting yea for domU-12-31-39-16-30-A2:31200 (0) m31100| Sat Nov 17 09:42:39.338 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state RECOVERING m31100| Sat Nov 17 09:42:39.338 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is now in state RECOVERING m31100| Sat Nov 17 09:42:39.415 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-0/admin.0, size: 16MB, took 0.261 secs m31100| Sat Nov 17 09:42:39.417 [conn1] build index admin.foo { _id: 1 } m31100| Sat Nov 17 09:42:39.418 [conn1] build index done. scanned 0 total records. 0 secs m31100| Sat Nov 17 09:42:39.418 [conn1] insert admin.foo keyUpdates:0 485ms ReplSetTest [object Object] ReplSetTest waiting for connection to domU-12-31-39-16-30-A2:31101 to have an oplog built. ReplSetTest waiting for connection to domU-12-31-39-16-30-A2:31102 to have an oplog built. m31201| Sat Nov 17 09:42:39.456 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31200 is now in state PRIMARY m31201| Sat Nov 17 09:42:39.459 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31202 is now in state RECOVERING m31202| Sat Nov 17 09:42:39.639 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31200 is now in state PRIMARY m31202| Sat Nov 17 09:42:39.640 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31201 is now in state RECOVERING m31200| Sat Nov 17 09:42:41.252 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31202 is now in state RECOVERING m31200| Sat Nov 17 09:42:41.252 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31201 is now in state RECOVERING ReplSetTest waiting for connection to domU-12-31-39-16-30-A2:31101 to have an oplog built. ReplSetTest waiting for connection to domU-12-31-39-16-30-A2:31102 to have an oplog built. m31101| Sat Nov 17 09:42:43.340 [conn2] end connection 127.0.0.1:37033 (2 connections now open) m31101| Sat Nov 17 09:42:43.340 [initandlisten] connection accepted from 127.0.0.1:37070 #4 (3 connections now open) ReplSetTest waiting for connection to domU-12-31-39-16-30-A2:31101 to have an oplog built. ReplSetTest waiting for connection to domU-12-31-39-16-30-A2:31102 to have an oplog built. m31201| Sat Nov 17 09:42:45.253 [conn2] end connection 127.0.0.1:36283 (2 connections now open) m31201| Sat Nov 17 09:42:45.254 [initandlisten] connection accepted from 127.0.0.1:36315 #4 (3 connections now open) ReplSetTest waiting for connection to domU-12-31-39-16-30-A2:31101 to have an oplog built. ReplSetTest waiting for connection to domU-12-31-39-16-30-A2:31102 to have an oplog built. m31100| Sat Nov 17 09:42:45.562 [conn2] end connection 127.0.0.1:49234 (2 connections now open) m31100| Sat Nov 17 09:42:45.563 [initandlisten] connection accepted from 127.0.0.1:49265 #4 (3 connections now open) m31100| Sat Nov 17 09:42:45.813 [conn3] end connection 127.0.0.1:49237 (2 connections now open) m31100| Sat Nov 17 09:42:45.813 [initandlisten] connection accepted from 127.0.0.1:49266 #5 (3 connections now open) ReplSetTest waiting for connection to domU-12-31-39-16-30-A2:31101 to have an oplog built. ReplSetTest waiting for connection to domU-12-31-39-16-30-A2:31102 to have an oplog built. m31200| Sat Nov 17 09:42:47.463 [conn2] end connection 127.0.0.1:34085 (2 connections now open) m31200| Sat Nov 17 09:42:47.463 [initandlisten] connection accepted from 127.0.0.1:34123 #4 (3 connections now open) m31200| Sat Nov 17 09:42:47.642 [conn3] end connection 127.0.0.1:34089 (2 connections now open) m31200| Sat Nov 17 09:42:47.642 [initandlisten] connection accepted from 127.0.0.1:34124 #5 (3 connections now open) m31101| Sat Nov 17 09:42:48.440 [rsSync] replSet initial sync pending m31101| Sat Nov 17 09:42:48.440 [rsSync] replSet syncing to: domU-12-31-39-16-30-A2:31100 m31101| Sat Nov 17 09:42:48.442 [rsSync] build index local.me { _id: 1 } m31101| Sat Nov 17 09:42:48.443 [rsSync] build index done. scanned 0 total records. 0 secs m31100| Sat Nov 17 09:42:48.440 [initandlisten] connection accepted from 127.0.0.1:49278 #6 (4 connections now open) m31101| Sat Nov 17 09:42:48.444 [rsSync] replSet initial sync drop all databases m31101| Sat Nov 17 09:42:48.444 [rsSync] dropAllDatabasesExceptLocal 1 m31101| Sat Nov 17 09:42:48.444 [rsSync] replSet initial sync clone all databases m31101| Sat Nov 17 09:42:48.445 [rsSync] replSet initial sync cloning db: admin m31100| Sat Nov 17 09:42:48.445 [initandlisten] connection accepted from 127.0.0.1:49280 #7 (5 connections now open) m31101| Sat Nov 17 09:42:48.445 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/admin.ns, filling with zeroes... m31101| Sat Nov 17 09:42:48.673 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/admin.ns, size: 16MB, took 0.227 secs m31101| Sat Nov 17 09:42:48.673 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/admin.0, filling with zeroes... m31101| Sat Nov 17 09:42:48.915 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/admin.0, size: 16MB, took 0.241 secs m31101| Sat Nov 17 09:42:48.919 [rsSync] build index admin.foo { _id: 1 } m31101| Sat Nov 17 09:42:48.919 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Sat Nov 17 09:42:48.920 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Sat Nov 17 09:42:48.920 [rsSync] replSet initial sync data copy, starting syncup m31101| Sat Nov 17 09:42:48.920 [rsSync] oplog sync 1 of 3 m31100| Sat Nov 17 09:42:48.920 [conn7] end connection 127.0.0.1:49280 (4 connections now open) m31101| Sat Nov 17 09:42:48.920 [rsSync] oplog sync 2 of 3 m31101| Sat Nov 17 09:42:48.920 [rsSync] replSet initial sync building indexes m31101| Sat Nov 17 09:42:48.920 [rsSync] replSet initial sync cloning indexes for : admin m31100| Sat Nov 17 09:42:48.920 [initandlisten] connection accepted from 127.0.0.1:49282 #8 (5 connections now open) m31101| Sat Nov 17 09:42:48.921 [rsSync] oplog sync 3 of 3 m31101| Sat Nov 17 09:42:48.921 [rsSync] replSet initial sync finishing up m31100| Sat Nov 17 09:42:48.921 [conn8] end connection 127.0.0.1:49282 (4 connections now open) m31101| Sat Nov 17 09:42:48.922 [rsSync] replSet set minValid=50a7a25f:1 m31101| Sat Nov 17 09:42:48.923 [rsSync] build index local.replset.minvalid { _id: 1 } m31101| Sat Nov 17 09:42:48.923 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Sat Nov 17 09:42:48.923 [rsSync] replSet initial sync done m31100| Sat Nov 17 09:42:48.923 [conn6] end connection 127.0.0.1:49278 (3 connections now open) m31102| Sat Nov 17 09:42:48.940 [rsSync] replSet initial sync pending m31102| Sat Nov 17 09:42:48.940 [rsSync] replSet syncing to: domU-12-31-39-16-30-A2:31100 m31102| Sat Nov 17 09:42:48.941 [rsSync] build index local.me { _id: 1 } m31102| Sat Nov 17 09:42:48.942 [rsSync] build index done. scanned 0 total records. 0 secs m31100| Sat Nov 17 09:42:48.945 [initandlisten] connection accepted from 127.0.0.1:49283 #9 (4 connections now open) m31102| Sat Nov 17 09:42:48.947 [rsSync] replSet initial sync drop all databases m31102| Sat Nov 17 09:42:48.947 [rsSync] dropAllDatabasesExceptLocal 1 m31102| Sat Nov 17 09:42:48.947 [rsSync] replSet initial sync clone all databases m31102| Sat Nov 17 09:42:48.947 [rsSync] replSet initial sync cloning db: admin m31100| Sat Nov 17 09:42:48.951 [initandlisten] connection accepted from 127.0.0.1:49284 #10 (5 connections now open) m31102| Sat Nov 17 09:42:48.951 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/admin.ns, filling with zeroes... m31101| Sat Nov 17 09:42:49.135 [rsBackgroundSync] replSet syncing to: domU-12-31-39-16-30-A2:31100 m31100| Sat Nov 17 09:42:49.136 [initandlisten] connection accepted from 127.0.0.1:49285 #11 (6 connections now open) m31102| Sat Nov 17 09:42:49.152 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/admin.ns, size: 16MB, took 0.2 secs m31102| Sat Nov 17 09:42:49.152 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/admin.0, filling with zeroes... m31102| Sat Nov 17 09:42:49.364 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/admin.0, size: 16MB, took 0.211 secs m31102| Sat Nov 17 09:42:49.378 [rsSync] build index admin.foo { _id: 1 } m31102| Sat Nov 17 09:42:49.379 [rsSync] fastBuildIndex dupsToDrop:0 m31102| Sat Nov 17 09:42:49.379 [rsSync] build index done. scanned 1 total records. 0 secs m31102| Sat Nov 17 09:42:49.379 [rsSync] replSet initial sync data copy, starting syncup m31102| Sat Nov 17 09:42:49.379 [rsSync] oplog sync 1 of 3 m31100| Sat Nov 17 09:42:49.379 [conn10] end connection 127.0.0.1:49284 (5 connections now open) m31102| Sat Nov 17 09:42:49.380 [rsSync] oplog sync 2 of 3 m31102| Sat Nov 17 09:42:49.380 [rsSync] replSet initial sync building indexes m31102| Sat Nov 17 09:42:49.380 [rsSync] replSet initial sync cloning indexes for : admin m31100| Sat Nov 17 09:42:49.380 [initandlisten] connection accepted from 127.0.0.1:49286 #12 (6 connections now open) m31100| Sat Nov 17 09:42:49.381 [conn12] end connection 127.0.0.1:49286 (5 connections now open) m31102| Sat Nov 17 09:42:49.381 [rsSync] oplog sync 3 of 3 m31102| Sat Nov 17 09:42:49.381 [rsSync] replSet initial sync finishing up m31102| Sat Nov 17 09:42:49.382 [rsSync] replSet set minValid=50a7a25f:1 m31102| Sat Nov 17 09:42:49.383 [rsSync] build index local.replset.minvalid { _id: 1 } m31102| Sat Nov 17 09:42:49.383 [rsSync] build index done. scanned 0 total records. 0 secs m31102| Sat Nov 17 09:42:49.383 [rsSync] replSet initial sync done m31100| Sat Nov 17 09:42:49.383 [conn9] end connection 127.0.0.1:49283 (4 connections now open) m31100| Sat Nov 17 09:42:49.417 [initandlisten] connection accepted from 127.0.0.1:49287 #13 (5 connections now open) m31102| Sat Nov 17 09:42:49.416 [rsBackgroundSync] replSet syncing to: domU-12-31-39-16-30-A2:31100 { "ts" : { "t" : 1353163359000, "i" : 1 }, "h" : NumberLong("6566749973310865425"), "v" : 2, "op" : "i", "ns" : "admin.foo", "o" : { "_id" : ObjectId("50a7a25e9e9ba0197a982fcd"), "x" : 1 } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163359000:1 and latest is 1353163359000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 1 { "ts" : { "t" : 1353163359000, "i" : 1 }, "h" : NumberLong("6566749973310865425"), "v" : 2, "op" : "i", "ns" : "admin.foo", "o" : { "_id" : ObjectId("50a7a25e9e9ba0197a982fcd"), "x" : 1 } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31102 is 1353163359000:1 and latest is 1353163359000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31102 is 1 ReplSetTest await synced=true m31101| Sat Nov 17 09:42:49.920 [rsSyncNotifier] replset setting oplog notifier to domU-12-31-39-16-30-A2:31100 m31100| Sat Nov 17 09:42:49.921 [initandlisten] connection accepted from 127.0.0.1:49288 #14 (6 connections now open) m31202| Sat Nov 17 09:42:49.946 [rsSync] replSet initial sync pending m31202| Sat Nov 17 09:42:49.946 [rsSync] replSet syncing to: domU-12-31-39-16-30-A2:31200 m31202| Sat Nov 17 09:42:49.948 [rsSync] build index local.me { _id: 1 } m31200| Sat Nov 17 09:42:49.947 [initandlisten] connection accepted from 127.0.0.1:34136 #6 (4 connections now open) m31202| Sat Nov 17 09:42:49.949 [rsSync] build index done. scanned 0 total records. 0 secs m31202| Sat Nov 17 09:42:49.950 [rsSync] replSet initial sync drop all databases m31202| Sat Nov 17 09:42:49.950 [rsSync] dropAllDatabasesExceptLocal 1 m31202| Sat Nov 17 09:42:49.950 [rsSync] replSet initial sync clone all databases m31202| Sat Nov 17 09:42:49.950 [rsSync] replSet initial sync data copy, starting syncup m31202| Sat Nov 17 09:42:49.950 [rsSync] oplog sync 1 of 3 m31202| Sat Nov 17 09:42:49.950 [rsSync] oplog sync 2 of 3 m31202| Sat Nov 17 09:42:49.950 [rsSync] replSet initial sync building indexes m31202| Sat Nov 17 09:42:49.950 [rsSync] oplog sync 3 of 3 m31202| Sat Nov 17 09:42:49.951 [rsSync] replSet initial sync finishing up m31202| Sat Nov 17 09:42:49.951 [rsSync] replSet set minValid=50a7a24e:1 m31202| Sat Nov 17 09:42:49.952 [rsSync] build index local.replset.minvalid { _id: 1 } m31202| Sat Nov 17 09:42:49.952 [rsSync] build index done. scanned 0 total records. 0 secs m31202| Sat Nov 17 09:42:49.952 [rsSync] replSet initial sync done m31200| Sat Nov 17 09:42:49.952 [conn6] end connection 127.0.0.1:34136 (3 connections now open) m31202| Sat Nov 17 09:42:49.966 [rsBackgroundSync] replSet syncing to: domU-12-31-39-16-30-A2:31200 m31200| Sat Nov 17 09:42:49.967 [initandlisten] connection accepted from 127.0.0.1:34137 #7 (4 connections now open) m31102| Sat Nov 17 09:42:50.380 [rsSyncNotifier] replset setting oplog notifier to domU-12-31-39-16-30-A2:31100 m31100| Sat Nov 17 09:42:50.381 [initandlisten] connection accepted from 127.0.0.1:49291 #15 (7 connections now open) m31200| Sat Nov 17 09:42:50.879 [initandlisten] connection accepted from 127.0.0.1:34139 #8 (5 connections now open) m31201| Sat Nov 17 09:42:50.878 [rsSync] replSet initial sync pending m31201| Sat Nov 17 09:42:50.878 [rsSync] replSet syncing to: domU-12-31-39-16-30-A2:31200 m31201| Sat Nov 17 09:42:50.880 [rsSync] build index local.me { _id: 1 } m31201| Sat Nov 17 09:42:50.881 [rsSync] build index done. scanned 0 total records. 0 secs m31201| Sat Nov 17 09:42:50.886 [rsSync] replSet initial sync drop all databases m31201| Sat Nov 17 09:42:50.886 [rsSync] dropAllDatabasesExceptLocal 1 m31201| Sat Nov 17 09:42:50.886 [rsSync] replSet initial sync clone all databases m31201| Sat Nov 17 09:42:50.886 [rsSync] replSet initial sync data copy, starting syncup m31201| Sat Nov 17 09:42:50.886 [rsSync] oplog sync 1 of 3 m31201| Sat Nov 17 09:42:50.886 [rsSync] oplog sync 2 of 3 m31201| Sat Nov 17 09:42:50.886 [rsSync] replSet initial sync building indexes m31201| Sat Nov 17 09:42:50.886 [rsSync] oplog sync 3 of 3 m31201| Sat Nov 17 09:42:50.886 [rsSync] replSet initial sync finishing up m31201| Sat Nov 17 09:42:50.905 [rsSync] replSet set minValid=50a7a24e:1 m31201| Sat Nov 17 09:42:50.906 [rsSync] build index local.replset.minvalid { _id: 1 } m31201| Sat Nov 17 09:42:50.906 [rsSync] build index done. scanned 0 total records. 0 secs m31201| Sat Nov 17 09:42:50.906 [rsSync] replSet initial sync done m31200| Sat Nov 17 09:42:50.906 [conn8] end connection 127.0.0.1:34139 (4 connections now open) m31101| Sat Nov 17 09:42:50.926 [rsSync] replSet SECONDARY m31100| Sat Nov 17 09:42:50.923 [slaveTracking] build index local.slaves { _id: 1 } m31100| Sat Nov 17 09:42:50.924 [slaveTracking] build index done. scanned 0 total records. 0 secs m31202| Sat Nov 17 09:42:50.951 [rsSyncNotifier] replset setting oplog notifier to domU-12-31-39-16-30-A2:31200 m31200| Sat Nov 17 09:42:50.952 [initandlisten] connection accepted from 127.0.0.1:34142 #9 (5 connections now open) m31200| Sat Nov 17 09:42:51.212 [initandlisten] connection accepted from 127.0.0.1:34143 #10 (6 connections now open) m31201| Sat Nov 17 09:42:51.212 [rsBackgroundSync] replSet syncing to: domU-12-31-39-16-30-A2:31200 m31100| Sat Nov 17 09:42:51.344 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is now in state SECONDARY m31102| Sat Nov 17 09:42:51.385 [rsSync] replSet SECONDARY Sat Nov 17 09:42:51.447 starting new replica set monitor for replica set clusteredstale-rs0 with seed of domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:42:51.447 successfully connected to seed domU-12-31-39-16-30-A2:31100 for replica set clusteredstale-rs0 Sat Nov 17 09:42:51.448 changing hosts to { 0: "domU-12-31-39-16-30-A2:31100", 1: "domU-12-31-39-16-30-A2:31102", 2: "domU-12-31-39-16-30-A2:31101" } from clusteredstale-rs0/ Sat Nov 17 09:42:51.448 trying to add new host domU-12-31-39-16-30-A2:31100 to replica set clusteredstale-rs0 Sat Nov 17 09:42:51.448 successfully connected to new host domU-12-31-39-16-30-A2:31100 in replica set clusteredstale-rs0 Sat Nov 17 09:42:51.448 trying to add new host domU-12-31-39-16-30-A2:31101 to replica set clusteredstale-rs0 Sat Nov 17 09:42:51.448 successfully connected to new host domU-12-31-39-16-30-A2:31101 in replica set clusteredstale-rs0 Sat Nov 17 09:42:51.448 trying to add new host domU-12-31-39-16-30-A2:31102 to replica set clusteredstale-rs0 Sat Nov 17 09:42:51.449 successfully connected to new host domU-12-31-39-16-30-A2:31102 in replica set clusteredstale-rs0 Sat Nov 17 09:42:51.449 Primary for replica set clusteredstale-rs0 changed to domU-12-31-39-16-30-A2:31100 m31100| Sat Nov 17 09:42:51.447 [initandlisten] connection accepted from 127.0.0.1:49297 #16 (8 connections now open) m31100| Sat Nov 17 09:42:51.448 [initandlisten] connection accepted from 127.0.0.1:49298 #17 (9 connections now open) m31100| Sat Nov 17 09:42:51.449 [initandlisten] connection accepted from 127.0.0.1:49301 #18 (10 connections now open) m31100| Sat Nov 17 09:42:51.449 [conn16] end connection 127.0.0.1:49297 (9 connections now open) m31102| Sat Nov 17 09:42:51.448 [initandlisten] connection accepted from 127.0.0.1:55149 #4 (4 connections now open) m31102| Sat Nov 17 09:42:51.450 [initandlisten] connection accepted from 127.0.0.1:55152 #5 (5 connections now open) Sat Nov 17 09:42:51.450 replica set monitor for replica set clusteredstale-rs0 started, address is clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:42:51.451 [ReplicaSetMonitorWatcher] starting m31101| Sat Nov 17 09:42:51.448 [initandlisten] connection accepted from 127.0.0.1:37111 #5 (4 connections now open) m31101| Sat Nov 17 09:42:51.450 [initandlisten] connection accepted from 127.0.0.1:37114 #6 (5 connections now open) m31200| Sat Nov 17 09:42:51.454 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-0/admin.ns, filling with zeroes... m31101| Sat Nov 17 09:42:51.564 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state SECONDARY m31200| Sat Nov 17 09:42:51.682 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-0/admin.ns, size: 16MB, took 0.228 secs m31200| Sat Nov 17 09:42:51.682 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-0/admin.0, filling with zeroes... m31102| Sat Nov 17 09:42:51.815 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is now in state SECONDARY m31200| Sat Nov 17 09:42:51.882 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-0/admin.0, size: 16MB, took 0.2 secs m31200| Sat Nov 17 09:42:51.884 [conn1] build index admin.foo { _id: 1 } m31200| Sat Nov 17 09:42:51.885 [conn1] build index done. scanned 0 total records. 0 secs m31200| Sat Nov 17 09:42:51.885 [conn1] insert admin.foo keyUpdates:0 431ms m31202| Sat Nov 17 09:42:51.886 [rsSync] replSet SECONDARY ReplSetTest [object Object] { "ts" : { "t" : 1353163342000, "i" : 1 }, "h" : NumberLong(0), "v" : 2, "op" : "n", "ns" : "", "o" : { "msg" : "initiating set" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31201 is 1353163342000:1 and latest is 1353163371000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31201 is 1 m31201| Sat Nov 17 09:42:51.905 [rsSyncNotifier] replset setting oplog notifier to domU-12-31-39-16-30-A2:31200 m31200| Sat Nov 17 09:42:51.905 [initandlisten] connection accepted from 127.0.0.1:34151 #11 (7 connections now open) m31202| Sat Nov 17 09:42:51.887 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-2/admin.ns, filling with zeroes... m31201| Sat Nov 17 09:42:51.926 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-1/admin.ns, filling with zeroes... m31200| Sat Nov 17 09:42:51.994 [slaveTracking] build index local.slaves { _id: 1 } m31200| Sat Nov 17 09:42:51.995 [slaveTracking] build index done. scanned 0 total records. 0.001 secs m31201| Sat Nov 17 09:42:52.231 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-1/admin.ns, size: 16MB, took 0.304 secs m31202| Sat Nov 17 09:42:52.231 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-2/admin.ns, size: 16MB, took 0.343 secs m31202| Sat Nov 17 09:42:52.231 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-2/admin.0, filling with zeroes... m31201| Sat Nov 17 09:42:52.231 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-1/admin.0, filling with zeroes... m31201| Sat Nov 17 09:42:52.971 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-1/admin.0, size: 16MB, took 0.74 secs m31202| Sat Nov 17 09:42:52.971 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-2/admin.0, size: 16MB, took 0.74 secs m31202| Sat Nov 17 09:42:52.973 [repl writer worker 1] build index admin.foo { _id: 1 } m31202| Sat Nov 17 09:42:52.974 [repl writer worker 1] build index done. scanned 0 total records. 0 secs { "ts" : { "t" : 1353163371000, "i" : 1 }, "h" : NumberLong("6566750849484193809"), "v" : 2, "op" : "i", "ns" : "admin.foo", "o" : { "_id" : ObjectId("50a7a26b9e9ba0197a982fce"), "x" : 1 } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31202 is 1353163371000:1 and latest is 1353163371000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31202 is 2 m31201| Sat Nov 17 09:42:52.976 [repl writer worker 1] build index admin.foo { _id: 1 } m31201| Sat Nov 17 09:42:52.977 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31200| Sat Nov 17 09:42:53.256 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31202 is now in state SECONDARY m31100| Sat Nov 17 09:42:53.343 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state SECONDARY m31201| Sat Nov 17 09:42:53.474 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31202 is now in state SECONDARY m31201| Sat Nov 17 09:42:53.991 [rsSync] replSet SECONDARY { "ts" : { "t" : 1353163371000, "i" : 1 }, "h" : NumberLong("6566750849484193809"), "v" : 2, "op" : "i", "ns" : "admin.foo", "o" : { "_id" : ObjectId("50a7a26b9e9ba0197a982fce"), "x" : 1 } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31201 is 1353163371000:1 and latest is 1353163371000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31201 is 2 { "ts" : { "t" : 1353163371000, "i" : 1 }, "h" : NumberLong("6566750849484193809"), "v" : 2, "op" : "i", "ns" : "admin.foo", "o" : { "_id" : ObjectId("50a7a26b9e9ba0197a982fce"), "x" : 1 } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31202 is 1353163371000:1 and latest is 1353163371000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31202 is 2 ReplSetTest await synced=true Sat Nov 17 09:42:54.980 starting new replica set monitor for replica set clusteredstale-rs1 with seed of domU-12-31-39-16-30-A2:31200,domU-12-31-39-16-30-A2:31201,domU-12-31-39-16-30-A2:31202 Sat Nov 17 09:42:54.980 successfully connected to seed domU-12-31-39-16-30-A2:31200 for replica set clusteredstale-rs1 Sat Nov 17 09:42:54.981 changing hosts to { 0: "domU-12-31-39-16-30-A2:31200", 1: "domU-12-31-39-16-30-A2:31202", 2: "domU-12-31-39-16-30-A2:31201" } from clusteredstale-rs1/ Sat Nov 17 09:42:54.981 trying to add new host domU-12-31-39-16-30-A2:31200 to replica set clusteredstale-rs1 Sat Nov 17 09:42:54.981 successfully connected to new host domU-12-31-39-16-30-A2:31200 in replica set clusteredstale-rs1 Sat Nov 17 09:42:54.981 trying to add new host domU-12-31-39-16-30-A2:31201 to replica set clusteredstale-rs1 m31201| Sat Nov 17 09:42:54.981 [initandlisten] connection accepted from 127.0.0.1:36361 #5 (4 connections now open) Sat Nov 17 09:42:54.981 successfully connected to new host domU-12-31-39-16-30-A2:31201 in replica set clusteredstale-rs1 Sat Nov 17 09:42:54.981 trying to add new host domU-12-31-39-16-30-A2:31202 to replica set clusteredstale-rs1 Sat Nov 17 09:42:54.981 successfully connected to new host domU-12-31-39-16-30-A2:31202 in replica set clusteredstale-rs1 Sat Nov 17 09:42:54.982 Primary for replica set clusteredstale-rs1 changed to domU-12-31-39-16-30-A2:31200 m31201| Sat Nov 17 09:42:54.984 [initandlisten] connection accepted from 127.0.0.1:36364 #6 (5 connections now open) Sat Nov 17 09:42:54.985 replica set monitor for replica set clusteredstale-rs1 started, address is clusteredstale-rs1/domU-12-31-39-16-30-A2:31200,domU-12-31-39-16-30-A2:31201,domU-12-31-39-16-30-A2:31202 Resetting db path '/data/db/clusteredstale-config0' Sat Nov 17 09:42:54.988 shell: started program /mnt/data/slaves/Linux_64bit_Legacy/mongo/mongod --port 29000 --dbpath /data/db/clusteredstale-config0 --configsvr m31200| Sat Nov 17 09:42:54.980 [initandlisten] connection accepted from 127.0.0.1:34154 #12 (8 connections now open) m31200| Sat Nov 17 09:42:54.981 [initandlisten] connection accepted from 127.0.0.1:34155 #13 (9 connections now open) m31200| Sat Nov 17 09:42:54.982 [initandlisten] connection accepted from 127.0.0.1:34158 #14 (10 connections now open) m31200| Sat Nov 17 09:42:54.982 [conn12] end connection 127.0.0.1:34154 (9 connections now open) m31202| Sat Nov 17 09:42:54.981 [initandlisten] connection accepted from 127.0.0.1:60885 #4 (4 connections now open) m31202| Sat Nov 17 09:42:54.985 [initandlisten] connection accepted from 127.0.0.1:60888 #5 (5 connections now open) m29000| Sat Nov 17 09:42:55.038 [initandlisten] MongoDB starting : pid=31110 port=29000 dbpath=/data/db/clusteredstale-config0 64-bit host=domU-12-31-39-16-30-A2 m29000| Sat Nov 17 09:42:55.038 [initandlisten] m29000| Sat Nov 17 09:42:55.038 [initandlisten] ** NOTE: This is a development version (2.3.1-pre-) of MongoDB. m29000| Sat Nov 17 09:42:55.038 [initandlisten] ** Not recommended for production. m29000| Sat Nov 17 09:42:55.038 [initandlisten] m29000| Sat Nov 17 09:42:55.038 [initandlisten] db version v2.3.1-pre-, pdfile version 4.5 m29000| Sat Nov 17 09:42:55.038 [initandlisten] git version: 38d64e3ffb528a597fb13d8aa459cd21f0967490 m29000| Sat Nov 17 09:42:55.038 [initandlisten] build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 BOOST_LIB_VERSION=1_49 m29000| Sat Nov 17 09:42:55.038 [initandlisten] options: { configsvr: true, dbpath: "/data/db/clusteredstale-config0", port: 29000 } m29000| Sat Nov 17 09:42:55.043 [initandlisten] journal dir=/data/db/clusteredstale-config0/journal m29000| Sat Nov 17 09:42:55.043 [initandlisten] recover : no journal files present, no recovery needed m29000| Sat Nov 17 09:42:55.127 [initandlisten] waiting for connections on port 29000 m29000| Sat Nov 17 09:42:55.127 [websvr] admin web console waiting for connections on port 30000 "domU-12-31-39-16-30-A2:29000" m29000| Sat Nov 17 09:42:55.190 [initandlisten] connection accepted from 127.0.0.1:56611 #1 (1 connection now open) ShardingTest clusteredstale : { "config" : "domU-12-31-39-16-30-A2:29000", "shards" : [ connection to clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102, connection to clusteredstale-rs1/domU-12-31-39-16-30-A2:31200,domU-12-31-39-16-30-A2:31201,domU-12-31-39-16-30-A2:31202 ] } Sat Nov 17 09:42:55.193 shell: started program /mnt/data/slaves/Linux_64bit_Legacy/mongo/mongos --port 30999 --configdb domU-12-31-39-16-30-A2:29000 --chunkSize 50 m29000| Sat Nov 17 09:42:55.190 [initandlisten] connection accepted from 127.0.0.1:56612 #2 (2 connections now open) m31200| Sat Nov 17 09:42:55.259 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31201 is now in state SECONDARY m30999| Sat Nov 17 09:42:55.561 running with 1 config server should be done only for testing purposes and is not recommended for production m30999| Sat Nov 17 09:42:55.561 [mongosMain] MongoS version 2.3.1-pre- starting: pid=31129 port=30999 64-bit host=domU-12-31-39-16-30-A2 (--help for usage) m30999| Sat Nov 17 09:42:55.562 [mongosMain] git version: 38d64e3ffb528a597fb13d8aa459cd21f0967490 m30999| Sat Nov 17 09:42:55.562 [mongosMain] build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 BOOST_LIB_VERSION=1_49 m30999| Sat Nov 17 09:42:55.562 [mongosMain] options: { chunkSize: 50, configdb: "domU-12-31-39-16-30-A2:29000", port: 30999 } m29000| Sat Nov 17 09:42:55.562 [initandlisten] connection accepted from 127.0.0.1:56616 #3 (3 connections now open) m29000| Sat Nov 17 09:42:55.567 [initandlisten] connection accepted from 127.0.0.1:56617 #4 (4 connections now open) m29000| Sat Nov 17 09:42:55.567 [FileAllocator] allocating new datafile /data/db/clusteredstale-config0/config.ns, filling with zeroes... m29000| Sat Nov 17 09:42:55.567 [FileAllocator] creating directory /data/db/clusteredstale-config0/_tmp m31202| Sat Nov 17 09:42:55.647 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31201 is now in state SECONDARY m29000| Sat Nov 17 09:42:55.823 [FileAllocator] done allocating datafile /data/db/clusteredstale-config0/config.ns, size: 16MB, took 0.24 secs m29000| Sat Nov 17 09:42:55.823 [FileAllocator] allocating new datafile /data/db/clusteredstale-config0/config.0, filling with zeroes... m29000| Sat Nov 17 09:42:56.064 [FileAllocator] done allocating datafile /data/db/clusteredstale-config0/config.0, size: 16MB, took 0.241 secs m29000| Sat Nov 17 09:42:56.065 [FileAllocator] allocating new datafile /data/db/clusteredstale-config0/config.1, filling with zeroes... m29000| Sat Nov 17 09:42:56.067 [conn4] build index config.version { _id: 1 } m29000| Sat Nov 17 09:42:56.067 [conn4] build index done. scanned 0 total records. 0 secs m29000| Sat Nov 17 09:42:56.067 [conn4] insert config.version keyUpdates:0 locks(micros) w:500083 500ms m29000| Sat Nov 17 09:42:56.085 [initandlisten] connection accepted from 127.0.0.1:56621 #5 (5 connections now open) m29000| Sat Nov 17 09:42:56.089 [conn5] build index config.settings { _id: 1 } m30999| Sat Nov 17 09:42:56.088 [mongosMain] waiting for connections on port 30999 m30999| Sat Nov 17 09:42:56.089 [websvr] admin web console waiting for connections on port 31999 m30999| Sat Nov 17 09:42:56.089 [Balancer] about to contact config servers and shards Sat Nov 17 09:42:56.211 shell: started program /mnt/data/slaves/Linux_64bit_Legacy/mongo/mongos --port 30998 --configdb domU-12-31-39-16-30-A2:29000 --chunkSize 50 m30999| Sat Nov 17 09:42:56.208 [mongosMain] connection accepted from 127.0.0.1:38783 #1 (1 connection now open) m30998| Sat Nov 17 09:42:56.224 running with 1 config server should be done only for testing purposes and is not recommended for production m30998| Sat Nov 17 09:42:56.225 [mongosMain] MongoS version 2.3.1-pre- starting: pid=31150 port=30998 64-bit host=domU-12-31-39-16-30-A2 (--help for usage) m30998| Sat Nov 17 09:42:56.225 [mongosMain] git version: 38d64e3ffb528a597fb13d8aa459cd21f0967490 m30998| Sat Nov 17 09:42:56.225 [mongosMain] build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 BOOST_LIB_VERSION=1_49 m30998| Sat Nov 17 09:42:56.225 [mongosMain] options: { chunkSize: 50, configdb: "domU-12-31-39-16-30-A2:29000", port: 30998 } m29000| Sat Nov 17 09:42:56.226 [initandlisten] connection accepted from 127.0.0.1:56624 #6 (6 connections now open) m30998| Sat Nov 17 09:42:56.256 [mongosMain] waiting for connections on port 30998 m30998| Sat Nov 17 09:42:56.256 [websvr] admin web console waiting for connections on port 31998 m30998| Sat Nov 17 09:42:56.256 [Balancer] about to contact config servers and shards m29000| Sat Nov 17 09:42:56.249 [conn5] build index done. scanned 0 total records. 0.16 secs m29000| Sat Nov 17 09:42:56.249 [conn5] insert config.settings keyUpdates:0 locks(micros) w:160398 160ms m29000| Sat Nov 17 09:42:56.249 [conn5] build index config.chunks { _id: 1 } m29000| Sat Nov 17 09:42:56.249 [conn5] build index done. scanned 0 total records. 0 secs m29000| Sat Nov 17 09:42:56.249 [conn5] info: creating collection config.chunks on add index m29000| Sat Nov 17 09:42:56.249 [conn5] build index config.chunks { ns: 1, min: 1 } m29000| Sat Nov 17 09:42:56.250 [conn5] build index done. scanned 0 total records. 0 secs m29000| Sat Nov 17 09:42:56.250 [conn5] build index config.chunks { ns: 1, shard: 1, min: 1 } m29000| Sat Nov 17 09:42:56.250 [conn5] build index done. scanned 0 total records. 0 secs m29000| Sat Nov 17 09:42:56.250 [conn5] build index config.chunks { ns: 1, lastmod: 1 } m29000| Sat Nov 17 09:42:56.250 [conn5] build index done. scanned 0 total records. 0 secs m29000| Sat Nov 17 09:42:56.250 [conn5] build index config.shards { _id: 1 } m29000| Sat Nov 17 09:42:56.251 [conn5] build index done. scanned 0 total records. 0 secs m29000| Sat Nov 17 09:42:56.251 [conn5] info: creating collection config.shards on add index m29000| Sat Nov 17 09:42:56.251 [conn5] build index config.shards { host: 1 } m29000| Sat Nov 17 09:42:56.251 [conn5] build index done. scanned 0 total records. 0 secs m29000| Sat Nov 17 09:42:56.252 [conn4] build index config.mongos { _id: 1 } m29000| Sat Nov 17 09:42:56.252 [conn4] build index done. scanned 0 total records. 0 secs m29000| Sat Nov 17 09:42:56.253 [initandlisten] connection accepted from 127.0.0.1:56625 #7 (7 connections now open) m29000| Sat Nov 17 09:42:56.255 [conn7] build index config.locks { _id: 1 } m29000| Sat Nov 17 09:42:56.256 [conn7] build index done. scanned 0 total records. 0 secs m29000| Sat Nov 17 09:42:56.256 [conn5] build index config.lockpings { _id: 1 } m29000| Sat Nov 17 09:42:56.256 [conn5] build index done. scanned 0 total records. 0 secs m29000| Sat Nov 17 09:42:56.257 [initandlisten] connection accepted from 127.0.0.1:56626 #8 (8 connections now open) m29000| Sat Nov 17 09:42:56.257 [conn5] build index config.lockpings { ping: 1 } m29000| Sat Nov 17 09:42:56.257 [initandlisten] connection accepted from 127.0.0.1:56627 #9 (9 connections now open) m29000| Sat Nov 17 09:42:56.257 [conn5] build index done. scanned 1 total records. 0 secs m30999| Sat Nov 17 09:42:56.251 [Balancer] config servers and shards contacted successfully m30999| Sat Nov 17 09:42:56.251 [Balancer] balancer id: domU-12-31-39-16-30-A2:30999 started at Nov 17 09:42:56 m30999| Sat Nov 17 09:42:56.251 [Balancer] created new distributed lock for balancer on domU-12-31-39-16-30-A2:29000 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m30999| Sat Nov 17 09:42:56.253 [conn1] couldn't find database [admin] in config db m30999| Sat Nov 17 09:42:56.256 [LockPinger] creating distributed lock ping thread for domU-12-31-39-16-30-A2:29000 and process domU-12-31-39-16-30-A2:30999:1353163376:1804289383 (sleeping for 30000ms) m30998| Sat Nov 17 09:42:56.258 [Balancer] config servers and shards contacted successfully m30998| Sat Nov 17 09:42:56.258 [Balancer] balancer id: domU-12-31-39-16-30-A2:30998 started at Nov 17 09:42:56 m30998| Sat Nov 17 09:42:56.258 [Balancer] created new distributed lock for balancer on domU-12-31-39-16-30-A2:29000 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m30998| Sat Nov 17 09:42:56.260 [LockPinger] creating distributed lock ping thread for domU-12-31-39-16-30-A2:29000 and process domU-12-31-39-16-30-A2:30998:1353163376:1804289383 (sleeping for 30000ms) m29000| Sat Nov 17 09:42:56.258 [conn5] build index config.databases { _id: 1 } m29000| Sat Nov 17 09:42:56.258 [initandlisten] connection accepted from 127.0.0.1:56628 #10 (10 connections now open) m29000| Sat Nov 17 09:42:56.258 [conn5] build index done. scanned 0 total records. 0 secs m29000| Sat Nov 17 09:42:56.258 [conn5] end connection 127.0.0.1:56621 (9 connections now open) m30999| Sat Nov 17 09:42:56.258 [conn1] put [admin] on: config:domU-12-31-39-16-30-A2:29000 m30999| Sat Nov 17 09:42:56.258 [conn1] scoped connection to domU-12-31-39-16-30-A2:29000 not being returned to the pool m30999| Sat Nov 17 09:42:56.313 [Balancer] could not acquire lock 'balancer/domU-12-31-39-16-30-A2:30999:1353163376:1804289383' (another update won) m30999| Sat Nov 17 09:42:56.313 [Balancer] distributed lock 'balancer/domU-12-31-39-16-30-A2:30999:1353163376:1804289383' was not acquired. m30998| Sat Nov 17 09:42:56.314 [Balancer] distributed lock 'balancer/domU-12-31-39-16-30-A2:30998:1353163376:1804289383' acquired, ts : 50a7a270e54d65b18b787922 m30998| Sat Nov 17 09:42:56.314 [Balancer] distributed lock 'balancer/domU-12-31-39-16-30-A2:30998:1353163376:1804289383' unlocked. m30998| Sat Nov 17 09:42:56.414 [mongosMain] connection accepted from 127.0.0.1:51673 #1 (1 connection now open) ShardingTest undefined going to add shard : clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:42:56.414 [conn1] scoped connection to domU-12-31-39-16-30-A2:29000 not being returned to the pool m31101| Sat Nov 17 09:42:56.416 [initandlisten] connection accepted from 127.0.0.1:37150 #7 (6 connections now open) m31101| Sat Nov 17 09:42:56.417 [initandlisten] connection accepted from 127.0.0.1:37153 #8 (7 connections now open) m29000| Sat Nov 17 09:42:56.414 [conn6] end connection 127.0.0.1:56624 (8 connections now open) m30999| Sat Nov 17 09:42:56.415 [conn1] starting new replica set monitor for replica set clusteredstale-rs0 with seed of domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:42:56.415 [conn1] successfully connected to seed domU-12-31-39-16-30-A2:31100 for replica set clusteredstale-rs0 m30999| Sat Nov 17 09:42:56.415 [conn1] changing hosts to { 0: "domU-12-31-39-16-30-A2:31100", 1: "domU-12-31-39-16-30-A2:31102", 2: "domU-12-31-39-16-30-A2:31101" } from clusteredstale-rs0/ m30999| Sat Nov 17 09:42:56.415 [conn1] trying to add new host domU-12-31-39-16-30-A2:31100 to replica set clusteredstale-rs0 m30999| Sat Nov 17 09:42:56.415 [conn1] successfully connected to new host domU-12-31-39-16-30-A2:31100 in replica set clusteredstale-rs0 m30999| Sat Nov 17 09:42:56.415 [conn1] trying to add new host domU-12-31-39-16-30-A2:31101 to replica set clusteredstale-rs0 m30999| Sat Nov 17 09:42:56.416 [conn1] successfully connected to new host domU-12-31-39-16-30-A2:31101 in replica set clusteredstale-rs0 m30999| Sat Nov 17 09:42:56.416 [conn1] trying to add new host domU-12-31-39-16-30-A2:31102 to replica set clusteredstale-rs0 m30999| Sat Nov 17 09:42:56.416 [conn1] successfully connected to new host domU-12-31-39-16-30-A2:31102 in replica set clusteredstale-rs0 m30999| Sat Nov 17 09:42:56.416 [conn1] Primary for replica set clusteredstale-rs0 changed to domU-12-31-39-16-30-A2:31100 m30999| Sat Nov 17 09:42:56.417 [conn1] replica set monitor for replica set clusteredstale-rs0 started, address is clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:42:56.418 [ReplicaSetMonitorWatcher] starting m30999| Sat Nov 17 09:42:56.418 [conn1] going to add shard: { _id: "clusteredstale-rs0", host: "clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102" } m31100| Sat Nov 17 09:42:56.415 [initandlisten] connection accepted from 127.0.0.1:49336 #19 (10 connections now open) m31100| Sat Nov 17 09:42:56.416 [initandlisten] connection accepted from 127.0.0.1:49337 #20 (11 connections now open) m31100| Sat Nov 17 09:42:56.416 [initandlisten] connection accepted from 127.0.0.1:49340 #21 (12 connections now open) m31100| Sat Nov 17 09:42:56.416 [conn19] end connection 127.0.0.1:49336 (11 connections now open) m31100| Sat Nov 17 09:42:56.418 [initandlisten] connection accepted from 127.0.0.1:49343 #22 (12 connections now open) m31102| Sat Nov 17 09:42:56.416 [initandlisten] connection accepted from 127.0.0.1:55188 #6 (6 connections now open) m31102| Sat Nov 17 09:42:56.417 [initandlisten] connection accepted from 127.0.0.1:55191 #7 (7 connections now open) m29000| Sat Nov 17 09:42:56.425 [FileAllocator] done allocating datafile /data/db/clusteredstale-config0/config.1, size: 32MB, took 0.36 secs { "shardAdded" : "clusteredstale-rs0", "ok" : 1 } ShardingTest undefined going to add shard : clusteredstale-rs1/domU-12-31-39-16-30-A2:31200,domU-12-31-39-16-30-A2:31201,domU-12-31-39-16-30-A2:31202 m30999| Sat Nov 17 09:42:56.440 [conn1] starting new replica set monitor for replica set clusteredstale-rs1 with seed of domU-12-31-39-16-30-A2:31200,domU-12-31-39-16-30-A2:31201,domU-12-31-39-16-30-A2:31202 m31200| Sat Nov 17 09:42:56.441 [initandlisten] connection accepted from 127.0.0.1:34191 #15 (10 connections now open) m30999| Sat Nov 17 09:42:56.441 [conn1] successfully connected to seed domU-12-31-39-16-30-A2:31200 for replica set clusteredstale-rs1 m31200| Sat Nov 17 09:42:56.441 [initandlisten] connection accepted from 127.0.0.1:34192 #16 (11 connections now open) m31200| Sat Nov 17 09:42:56.441 [initandlisten] connection accepted from 127.0.0.1:34195 #17 (12 connections now open) m31202| Sat Nov 17 09:42:56.441 [initandlisten] connection accepted from 127.0.0.1:60922 #6 (6 connections now open) m31200| Sat Nov 17 09:42:56.442 [conn15] end connection 127.0.0.1:34191 (11 connections now open) m31202| Sat Nov 17 09:42:56.443 [initandlisten] connection accepted from 127.0.0.1:60925 #7 (7 connections now open) m31200| Sat Nov 17 09:42:56.443 [initandlisten] connection accepted from 127.0.0.1:34198 #18 (12 connections now open) m31201| Sat Nov 17 09:42:56.441 [initandlisten] connection accepted from 127.0.0.1:36398 #7 (6 connections now open) m31201| Sat Nov 17 09:42:56.442 [initandlisten] connection accepted from 127.0.0.1:36401 #8 (7 connections now open) m30999| Sat Nov 17 09:42:56.441 [conn1] changing hosts to { 0: "domU-12-31-39-16-30-A2:31200", 1: "domU-12-31-39-16-30-A2:31202", 2: "domU-12-31-39-16-30-A2:31201" } from clusteredstale-rs1/ m30999| Sat Nov 17 09:42:56.441 [conn1] trying to add new host domU-12-31-39-16-30-A2:31200 to replica set clusteredstale-rs1 m30999| Sat Nov 17 09:42:56.441 [conn1] successfully connected to new host domU-12-31-39-16-30-A2:31200 in replica set clusteredstale-rs1 m30999| Sat Nov 17 09:42:56.441 [conn1] trying to add new host domU-12-31-39-16-30-A2:31201 to replica set clusteredstale-rs1 m30999| Sat Nov 17 09:42:56.441 [conn1] successfully connected to new host domU-12-31-39-16-30-A2:31201 in replica set clusteredstale-rs1 m30999| Sat Nov 17 09:42:56.441 [conn1] trying to add new host domU-12-31-39-16-30-A2:31202 to replica set clusteredstale-rs1 m30999| Sat Nov 17 09:42:56.441 [conn1] successfully connected to new host domU-12-31-39-16-30-A2:31202 in replica set clusteredstale-rs1 m30999| Sat Nov 17 09:42:56.442 [conn1] Primary for replica set clusteredstale-rs1 changed to domU-12-31-39-16-30-A2:31200 m30999| Sat Nov 17 09:42:56.443 [conn1] replica set monitor for replica set clusteredstale-rs1 started, address is clusteredstale-rs1/domU-12-31-39-16-30-A2:31200,domU-12-31-39-16-30-A2:31201,domU-12-31-39-16-30-A2:31202 m30999| Sat Nov 17 09:42:56.444 [conn1] going to add shard: { _id: "clusteredstale-rs1", host: "clusteredstale-rs1/domU-12-31-39-16-30-A2:31200,domU-12-31-39-16-30-A2:31201,domU-12-31-39-16-30-A2:31202" } { "shardAdded" : "clusteredstale-rs1", "ok" : 1 } m31200| Sat Nov 17 09:42:56.447 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-0/test_b.ns, filling with zeroes... m31100| Sat Nov 17 09:42:56.446 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-0/test_a.ns, filling with zeroes... m30999| Sat Nov 17 09:42:56.466 [conn1] creating WriteBackListener for: domU-12-31-39-16-30-A2:31100 serverID: 50a7a270e10b3cbd7ae1b530 m30999| Sat Nov 17 09:42:56.466 [conn1] creating WriteBackListener for: domU-12-31-39-16-30-A2:31101 serverID: 50a7a270e10b3cbd7ae1b530 m30999| Sat Nov 17 09:42:56.466 [conn1] creating WriteBackListener for: domU-12-31-39-16-30-A2:31102 serverID: 50a7a270e10b3cbd7ae1b530 m31100| Sat Nov 17 09:42:56.527 [initandlisten] connection accepted from 127.0.0.1:49352 #23 (13 connections now open) m31100| Sat Nov 17 09:42:57.014 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-0/test_a.ns, size: 16MB, took 0.567 secs m31100| Sat Nov 17 09:42:57.014 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-0/test_a.0, filling with zeroes... m31200| Sat Nov 17 09:42:57.014 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-0/test_b.ns, size: 16MB, took 0.566 secs m31200| Sat Nov 17 09:42:57.014 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-0/test_b.0, filling with zeroes... m31102| Sat Nov 17 09:42:57.344 [conn2] end connection 127.0.0.1:55071 (6 connections now open) m31102| Sat Nov 17 09:42:57.345 [initandlisten] connection accepted from 127.0.0.1:55205 #8 (7 connections now open) m31201| Sat Nov 17 09:42:57.649 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-1/test_b.ns, filling with zeroes... m31200| Sat Nov 17 09:42:57.645 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-0/test_b.0, size: 16MB, took 0.63 secs m31200| Sat Nov 17 09:42:57.647 [conn1] build index test_b.dummy { _id: 1 } m31200| Sat Nov 17 09:42:57.648 [conn1] build index done. scanned 0 total records. 0 secs m31200| Sat Nov 17 09:42:57.648 [conn1] insert test_b.dummy keyUpdates:0 locks(micros) w:1201518 1201ms ReplSetTest [object Object] { "ts" : { "t" : 1353163359000, "i" : 1 }, "h" : NumberLong("6566749973310865425"), "v" : 2, "op" : "i", "ns" : "admin.foo", "o" : { "_id" : ObjectId("50a7a25e9e9ba0197a982fcd"), "x" : 1 } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163359000:1 and latest is 1353163377000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 1 { "ts" : { "t" : 1353163359000, "i" : 1 }, "h" : NumberLong("6566749973310865425"), "v" : 2, "op" : "i", "ns" : "admin.foo", "o" : { "_id" : ObjectId("50a7a25e9e9ba0197a982fcd"), "x" : 1 } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31102 is 1353163359000:1 and latest is 1353163377000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31102 is 1 m31200| Sat Nov 17 09:42:57.719 [initandlisten] connection accepted from 127.0.0.1:34204 #19 (13 connections now open) m29000| Sat Nov 17 09:42:57.720 [initandlisten] connection accepted from 127.0.0.1:56652 #11 (9 connections now open) m31102| Sat Nov 17 09:42:57.724 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/test_a.ns, filling with zeroes... m30999| Sat Nov 17 09:42:57.718 [conn1] creating WriteBackListener for: domU-12-31-39-16-30-A2:31200 serverID: 50a7a270e10b3cbd7ae1b530 m30999| Sat Nov 17 09:42:57.718 [conn1] creating WriteBackListener for: domU-12-31-39-16-30-A2:31201 serverID: 50a7a270e10b3cbd7ae1b530 m30999| Sat Nov 17 09:42:57.718 [conn1] creating WriteBackListener for: domU-12-31-39-16-30-A2:31202 serverID: 50a7a270e10b3cbd7ae1b530 m30999| Sat Nov 17 09:42:57.720 [conn1] creating WriteBackListener for: domU-12-31-39-16-30-A2:29000 serverID: 50a7a270e10b3cbd7ae1b530 m31101| Sat Nov 17 09:42:57.789 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/test_a.ns, filling with zeroes... m31100| Sat Nov 17 09:42:57.710 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-0/test_a.0, size: 16MB, took 0.696 secs m31100| Sat Nov 17 09:42:57.712 [conn1] build index test_a.dummy { _id: 1 } m31100| Sat Nov 17 09:42:57.713 [conn1] build index done. scanned 0 total records. 0 secs m31100| Sat Nov 17 09:42:57.713 [conn1] insert test_a.dummy keyUpdates:0 locks(micros) w:1267638 1267ms m31100| Sat Nov 17 09:42:57.715 [conn23] command admin.$cmd command: { setShardVersion: "", init: true, configdb: "domU-12-31-39-16-30-A2:29000", serverID: ObjectId('50a7a270e10b3cbd7ae1b530'), authoritative: true, $auth: {} } ntoreturn:1 keyUpdates:0 locks(micros) W:181 reslen:51 1187ms m31100| Sat Nov 17 09:42:57.715 [conn11] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1353163359000|1 } } cursorid:122887652802144 ntoreturn:0 keyUpdates:0 numYields: 1 locks(micros) r:157 nreturned:1 reslen:122 3554ms m31100| Sat Nov 17 09:42:57.715 [conn13] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1353163359000|1 } } cursorid:124096317311705 ntoreturn:0 keyUpdates:0 numYields: 1 locks(micros) r:127 nreturned:1 reslen:122 3260ms m31201| Sat Nov 17 09:42:57.999 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-1/test_b.ns, size: 16MB, took 0.35 secs m31201| Sat Nov 17 09:42:58.000 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-1/test_b.0, filling with zeroes... m31202| Sat Nov 17 09:42:58.002 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-2/test_b.ns, filling with zeroes... m31102| Sat Nov 17 09:42:58.503 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/test_a.ns, size: 16MB, took 0.778 secs m31102| Sat Nov 17 09:42:58.503 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/test_a.0, filling with zeroes... m31101| Sat Nov 17 09:42:58.570 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/test_a.ns, size: 16MB, took 0.78 secs m31101| Sat Nov 17 09:42:58.570 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/test_a.0, filling with zeroes... m31202| Sat Nov 17 09:42:59.020 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-2/test_b.ns, size: 16MB, took 1.017 secs m31202| Sat Nov 17 09:42:59.021 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-2/test_b.0, filling with zeroes... m31201| Sat Nov 17 09:42:59.021 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-1/test_b.0, size: 16MB, took 1.02 secs m31201| Sat Nov 17 09:42:59.023 [repl writer worker 1] build index test_b.dummy { _id: 1 } m31201| Sat Nov 17 09:42:59.024 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31202| Sat Nov 17 09:42:59.258 [conn2] end connection 127.0.0.1:60807 (6 connections now open) m31202| Sat Nov 17 09:42:59.259 [initandlisten] connection accepted from 127.0.0.1:38275 #8 (7 connections now open) m31102| Sat Nov 17 09:42:59.567 [conn3] end connection 127.0.0.1:55088 (6 connections now open) m31102| Sat Nov 17 09:42:59.567 [initandlisten] connection accepted from 127.0.0.1:44488 #9 (7 connections now open) m31101| Sat Nov 17 09:42:59.842 [conn3] end connection 127.0.0.1:37055 (6 connections now open) m31101| Sat Nov 17 09:42:59.843 [initandlisten] connection accepted from 127.0.0.1:57064 #9 (7 connections now open) { "ts" : { "t" : 1353163377000, "i" : 1 }, "h" : NumberLong("2450891399193727988"), "v" : 2, "op" : "i", "ns" : "test_a.dummy", "o" : { "_id" : ObjectId("50a7a2709e9ba0197a982fd2"), "x" : 1 } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163377000:1 and latest is 1353163377000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 2 m31101| Sat Nov 17 09:42:59.961 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/test_a.0, size: 16MB, took 1.391 secs m31101| Sat Nov 17 09:42:59.964 [repl writer worker 1] build index test_a.dummy { _id: 1 } m31101| Sat Nov 17 09:42:59.965 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31102| Sat Nov 17 09:42:59.964 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/test_a.0, size: 16MB, took 1.46 secs m31102| Sat Nov 17 09:42:59.966 [repl writer worker 1] build index test_a.dummy { _id: 1 } m31102| Sat Nov 17 09:42:59.967 [repl writer worker 1] build index done. scanned 0 total records. 0 secs { "ts" : { "t" : 1353163377000, "i" : 1 }, "h" : NumberLong("2450891399193727988"), "v" : 2, "op" : "i", "ns" : "test_a.dummy", "o" : { "_id" : ObjectId("50a7a2709e9ba0197a982fd2"), "x" : 1 } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31102 is 1353163377000:1 and latest is 1353163377000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31102 is 2 ReplSetTest await synced=true ReplSetTest [object Object] { "ts" : { "t" : 1353163377000, "i" : 1 }, "h" : NumberLong("2452842637196039156"), "v" : 2, "op" : "i", "ns" : "test_b.dummy", "o" : { "_id" : ObjectId("50a7a2709e9ba0197a982fd3"), "x" : 1 } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31201 is 1353163377000:1 and latest is 1353163377000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31201 is 3 m31202| Sat Nov 17 09:43:00.311 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-2/test_b.0, size: 16MB, took 1.072 secs m31202| Sat Nov 17 09:43:00.314 [repl writer worker 1] build index test_b.dummy { _id: 1 } m31202| Sat Nov 17 09:43:00.315 [repl writer worker 1] build index done. scanned 0 total records. 0 secs { "ts" : { "t" : 1353163377000, "i" : 1 }, "h" : NumberLong("2452842637196039156"), "v" : 2, "op" : "i", "ns" : "test_b.dummy", "o" : { "_id" : ObjectId("50a7a2709e9ba0197a982fd3"), "x" : 1 } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31202 is 1353163377000:1 and latest is 1353163377000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31202 is 3 ReplSetTest await synced=true stale_clustered.js 1: initial insert stale_clustered.js 2: shard collection m31100| Sat Nov 17 09:43:00.337 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-0/test.ns, filling with zeroes... m30999| Sat Nov 17 09:43:00.334 [conn1] couldn't find database [test] in config db m30999| Sat Nov 17 09:43:00.336 [conn1] put [test] on: clusteredstale-rs0:clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:00.809 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-0/test.ns, size: 16MB, took 0.472 secs m31100| Sat Nov 17 09:43:00.809 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-0/test.0, filling with zeroes... m31100| Sat Nov 17 09:43:01.065 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-0/test.0, size: 16MB, took 0.255 secs m31100| Sat Nov 17 09:43:01.068 [conn23] build index test.foo { _id: 1 } m31100| Sat Nov 17 09:43:01.069 [conn23] build index done. scanned 0 total records. 0 secs m31100| Sat Nov 17 09:43:01.069 [conn23] update test.foo query: { _id: -1.0 } update: { _id: -1.0, a: "a", date: new Date(1353163380333) } nscanned:0 nupdated:1 upsert:1 keyUpdates:0 locks(micros) w:732479 732ms m31102| Sat Nov 17 09:43:01.071 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/test.ns, filling with zeroes... m30999| Sat Nov 17 09:43:01.071 [conn1] enabling sharding on: test m30999| Sat Nov 17 09:43:01.077 [conn1] CMD: shardcollection: { shardcollection: "test.foo", key: { _id: 1.0 } } m30999| Sat Nov 17 09:43:01.077 [conn1] enable sharding on: test.foo with shard key: { _id: 1.0 } m30999| Sat Nov 17 09:43:01.143 [conn1] going to create 1 chunk(s) for: test.foo using new epoch 50a7a275e10b3cbd7ae1b532 m30999| Sat Nov 17 09:43:01.144 [conn1] ChunkManager: time to load chunks for test.foo: 0ms sequenceNumber: 2 version: 1|0||50a7a275e10b3cbd7ae1b532 based on: (empty) m29000| Sat Nov 17 09:43:01.145 [conn3] build index config.collections { _id: 1 } m31101| Sat Nov 17 09:43:01.079 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/test.ns, filling with zeroes... m29000| Sat Nov 17 09:43:01.155 [conn3] build index done. scanned 0 total records. 0.009 secs m31100| Sat Nov 17 09:43:01.166 [conn23] no current chunk manager found for this shard, will initialize m29000| Sat Nov 17 09:43:01.168 [initandlisten] connection accepted from 127.0.0.1:46337 #12 (10 connections now open) m30999| Sat Nov 17 09:43:01.170 [conn1] splitting: test.foo shard: ns:test.fooshard: clusteredstale-rs0:clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102lastmod: 1|0||000000000000000000000000min: { _id: MinKey }max: { _id: MaxKey } m29000| Sat Nov 17 09:43:01.171 [initandlisten] connection accepted from 127.0.0.1:46338 #13 (11 connections now open) m31100| Sat Nov 17 09:43:01.171 [conn22] received splitChunk request: { splitChunk: "test.foo", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: "clusteredstale-rs0", splitKeys: [ { _id: 0.0 } ], shardId: "test.foo-_id_MinKey", configdb: "domU-12-31-39-16-30-A2:29000" } m31100| Sat Nov 17 09:43:01.171 [conn22] created new distributed lock for test.foo on domU-12-31-39-16-30-A2:29000 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m29000| Sat Nov 17 09:43:01.175 [initandlisten] connection accepted from 127.0.0.1:46339 #14 (12 connections now open) m31100| Sat Nov 17 09:43:01.172 [LockPinger] creating distributed lock ping thread for domU-12-31-39-16-30-A2:29000 and process domU-12-31-39-16-30-A2:31100:1353163381:745512520 (sleeping for 30000ms) m31100| Sat Nov 17 09:43:01.174 [conn22] distributed lock 'test.foo/domU-12-31-39-16-30-A2:31100:1353163381:745512520' acquired, ts : 50a7a2750b7b1461e028b270 m31100| Sat Nov 17 09:43:01.192 [conn22] splitChunk accepted at version 1|0||50a7a275e10b3cbd7ae1b532 m31202| Sat Nov 17 09:43:01.489 [conn3] end connection 127.0.0.1:60820 (6 connections now open) m31202| Sat Nov 17 09:43:01.489 [initandlisten] connection accepted from 127.0.0.1:38284 #9 (7 connections now open) m31102| Sat Nov 17 09:43:01.614 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/test.ns, size: 16MB, took 0.543 secs m31102| Sat Nov 17 09:43:01.614 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/test.0, filling with zeroes... m29000| Sat Nov 17 09:43:01.633 [conn13] command config.$cmd command: { applyOps: [ { op: "u", b: true, ns: "config.chunks", o: { _id: "test.foo-_id_MinKey", lastmod: Timestamp 1000|1, lastmodEpoch: ObjectId('50a7a275e10b3cbd7ae1b532'), ns: "test.foo", min: { _id: MinKey }, max: { _id: 0.0 }, shard: "clusteredstale-rs0" }, o2: { _id: "test.foo-_id_MinKey" } }, { op: "u", b: true, ns: "config.chunks", o: { _id: "test.foo-_id_0.0", lastmod: Timestamp 1000|2, lastmodEpoch: ObjectId('50a7a275e10b3cbd7ae1b532'), ns: "test.foo", min: { _id: 0.0 }, max: { _id: MaxKey }, shard: "clusteredstale-rs0" }, o2: { _id: "test.foo-_id_0.0" } } ], preCondition: [ { ns: "config.chunks", q: { query: { ns: "test.foo" }, orderby: { lastmod: -1 } }, res: { lastmod: Timestamp 1000|0 } } ] } ntoreturn:1 keyUpdates:0 locks(micros) W:441300 reslen:72 441ms m29000| Sat Nov 17 09:43:01.635 [conn14] build index config.changelog { _id: 1 } m29000| Sat Nov 17 09:43:01.635 [conn14] build index done. scanned 0 total records. 0 secs m31202| Sat Nov 17 09:43:01.695 [initandlisten] connection accepted from 127.0.0.1:38288 #10 (8 connections now open) m31202| Sat Nov 17 09:43:01.697 [initandlisten] connection accepted from 127.0.0.1:38291 #11 (9 connections now open) m31200| Sat Nov 17 09:43:01.694 [initandlisten] connection accepted from 127.0.0.1:46161 #20 (14 connections now open) m31200| Sat Nov 17 09:43:01.695 [initandlisten] connection accepted from 127.0.0.1:46162 #21 (15 connections now open) m31200| Sat Nov 17 09:43:01.696 [initandlisten] connection accepted from 127.0.0.1:46165 #22 (16 connections now open) m31200| Sat Nov 17 09:43:01.696 [conn20] end connection 127.0.0.1:46161 (15 connections now open) m31200| Sat Nov 17 09:43:01.697 [initandlisten] connection accepted from 127.0.0.1:46168 #23 (16 connections now open) m31200| Sat Nov 17 09:43:01.700 [migrateThread] starting new replica set monitor for replica set clusteredstale-rs0 with seed of domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102 m31201| Sat Nov 17 09:43:01.695 [initandlisten] connection accepted from 127.0.0.1:53827 #9 (8 connections now open) m31201| Sat Nov 17 09:43:01.697 [initandlisten] connection accepted from 127.0.0.1:53830 #10 (9 connections now open) m31200| Sat Nov 17 09:43:01.702 [migrateThread] successfully connected to seed domU-12-31-39-16-30-A2:31100 for replica set clusteredstale-rs0 m31200| Sat Nov 17 09:43:01.702 [migrateThread] changing hosts to { 0: "domU-12-31-39-16-30-A2:31100", 1: "domU-12-31-39-16-30-A2:31102", 2: "domU-12-31-39-16-30-A2:31101" } from clusteredstale-rs0/ m31200| Sat Nov 17 09:43:01.702 [migrateThread] trying to add new host domU-12-31-39-16-30-A2:31100 to replica set clusteredstale-rs0 m31200| Sat Nov 17 09:43:01.703 [migrateThread] successfully connected to new host domU-12-31-39-16-30-A2:31100 in replica set clusteredstale-rs0 m31200| Sat Nov 17 09:43:01.703 [migrateThread] trying to add new host domU-12-31-39-16-30-A2:31101 to replica set clusteredstale-rs0 m31201| Sat Nov 17 09:43:01.703 [conn3] end connection 127.0.0.1:36298 (8 connections now open) m31200| Sat Nov 17 09:43:01.704 [migrateThread] successfully connected to new host domU-12-31-39-16-30-A2:31101 in replica set clusteredstale-rs0 m31200| Sat Nov 17 09:43:01.704 [migrateThread] trying to add new host domU-12-31-39-16-30-A2:31102 to replica set clusteredstale-rs0 m31102| Sat Nov 17 09:43:01.706 [initandlisten] connection accepted from 127.0.0.1:44508 #10 (8 connections now open) m31200| Sat Nov 17 09:43:01.706 [migrateThread] successfully connected to new host domU-12-31-39-16-30-A2:31102 in replica set clusteredstale-rs0 m31200| Sat Nov 17 09:43:01.707 [migrateThread] Primary for replica set clusteredstale-rs0 changed to domU-12-31-39-16-30-A2:31100 m31102| Sat Nov 17 09:43:01.707 [initandlisten] connection accepted from 127.0.0.1:44511 #11 (9 connections now open) m31200| Sat Nov 17 09:43:01.708 [migrateThread] replica set monitor for replica set clusteredstale-rs0 started, address is clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:01.544 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/test.ns, size: 16MB, took 0.464 secs m31101| Sat Nov 17 09:43:01.544 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/test.0, filling with zeroes... m31101| Sat Nov 17 09:43:01.704 [initandlisten] connection accepted from 127.0.0.1:57081 #10 (8 connections now open) m31101| Sat Nov 17 09:43:01.707 [initandlisten] connection accepted from 127.0.0.1:57084 #11 (9 connections now open) m31100| Sat Nov 17 09:43:01.634 [conn22] about to log metadata event: { _id: "domU-12-31-39-16-30-A2-2012-11-17T14:43:01-0", server: "domU-12-31-39-16-30-A2", clientAddr: "127.0.0.1:49343", time: new Date(1353163381634), what: "split", ns: "test.foo", details: { before: { min: { _id: MinKey }, max: { _id: MaxKey }, lastmod: Timestamp 1000|0, lastmodEpoch: ObjectId('000000000000000000000000') }, left: { min: { _id: MinKey }, max: { _id: 0.0 }, lastmod: Timestamp 1000|1, lastmodEpoch: ObjectId('50a7a275e10b3cbd7ae1b532') }, right: { min: { _id: 0.0 }, max: { _id: MaxKey }, lastmod: Timestamp 1000|2, lastmodEpoch: ObjectId('50a7a275e10b3cbd7ae1b532') } } } m31100| Sat Nov 17 09:43:01.674 [conn22] distributed lock 'test.foo/domU-12-31-39-16-30-A2:31100:1353163381:745512520' unlocked. m31100| Sat Nov 17 09:43:01.674 [conn22] command admin.$cmd command: { splitChunk: "test.foo", keyPattern: { _id: 1.0 }, min: { _id: MinKey }, max: { _id: MaxKey }, from: "clusteredstale-rs0", splitKeys: [ { _id: 0.0 } ], shardId: "test.foo-_id_MinKey", configdb: "domU-12-31-39-16-30-A2:29000" } ntoreturn:1 keyUpdates:0 locks(micros) r:38 reslen:95 503ms m31100| Sat Nov 17 09:43:01.693 [conn22] received moveChunk request: { moveChunk: "test.foo", from: "clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102", to: "clusteredstale-rs1/domU-12-31-39-16-30-A2:31200,domU-12-31-39-16-30-A2:31201,domU-12-31-39-16-30-A2:31202", fromShard: "clusteredstale-rs0", toShard: "clusteredstale-rs1", min: { _id: 0.0 }, max: { _id: MaxKey }, maxChunkSizeBytes: 52428800, shardId: "test.foo-_id_0.0", configdb: "domU-12-31-39-16-30-A2:29000", secondaryThrottle: false } m31100| Sat Nov 17 09:43:01.693 [conn22] created new distributed lock for test.foo on domU-12-31-39-16-30-A2:29000 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m31100| Sat Nov 17 09:43:01.693 [conn22] distributed lock 'test.foo/domU-12-31-39-16-30-A2:31100:1353163381:745512520' acquired, ts : 50a7a2750b7b1461e028b271 m31100| Sat Nov 17 09:43:01.694 [conn22] about to log metadata event: { _id: "domU-12-31-39-16-30-A2-2012-11-17T14:43:01-1", server: "domU-12-31-39-16-30-A2", clientAddr: "127.0.0.1:49343", time: new Date(1353163381694), what: "moveChunk.start", ns: "test.foo", details: { min: { _id: 0.0 }, max: { _id: MaxKey }, from: "clusteredstale-rs0", to: "clusteredstale-rs1" } } m31100| Sat Nov 17 09:43:01.694 [conn22] moveChunk request accepted at version 1|2||50a7a275e10b3cbd7ae1b532 m31100| Sat Nov 17 09:43:01.694 [conn22] moveChunk number of documents: 1 m31100| Sat Nov 17 09:43:01.694 [conn22] starting new replica set monitor for replica set clusteredstale-rs1 with seed of domU-12-31-39-16-30-A2:31200,domU-12-31-39-16-30-A2:31201,domU-12-31-39-16-30-A2:31202 m31100| Sat Nov 17 09:43:01.694 [conn22] successfully connected to seed domU-12-31-39-16-30-A2:31200 for replica set clusteredstale-rs1 m31100| Sat Nov 17 09:43:01.695 [conn22] changing hosts to { 0: "domU-12-31-39-16-30-A2:31200", 1: "domU-12-31-39-16-30-A2:31202", 2: "domU-12-31-39-16-30-A2:31201" } from clusteredstale-rs1/ m31100| Sat Nov 17 09:43:01.695 [conn22] trying to add new host domU-12-31-39-16-30-A2:31200 to replica set clusteredstale-rs1 m31100| Sat Nov 17 09:43:01.695 [conn22] successfully connected to new host domU-12-31-39-16-30-A2:31200 in replica set clusteredstale-rs1 m31100| Sat Nov 17 09:43:01.695 [conn22] trying to add new host domU-12-31-39-16-30-A2:31201 to replica set clusteredstale-rs1 m31100| Sat Nov 17 09:43:01.695 [conn22] successfully connected to new host domU-12-31-39-16-30-A2:31201 in replica set clusteredstale-rs1 m31100| Sat Nov 17 09:43:01.695 [conn22] trying to add new host domU-12-31-39-16-30-A2:31202 to replica set clusteredstale-rs1 m31100| Sat Nov 17 09:43:01.695 [conn22] successfully connected to new host domU-12-31-39-16-30-A2:31202 in replica set clusteredstale-rs1 m31100| Sat Nov 17 09:43:01.696 [conn22] Primary for replica set clusteredstale-rs1 changed to domU-12-31-39-16-30-A2:31200 m31100| Sat Nov 17 09:43:01.697 [conn22] replica set monitor for replica set clusteredstale-rs1 started, address is clusteredstale-rs1/domU-12-31-39-16-30-A2:31200,domU-12-31-39-16-30-A2:31201,domU-12-31-39-16-30-A2:31202 m31100| Sat Nov 17 09:43:01.697 [ReplicaSetMonitorWatcher] starting m31100| Sat Nov 17 09:43:01.702 [initandlisten] connection accepted from 127.0.0.1:59307 #24 (14 connections now open) m31100| Sat Nov 17 09:43:01.702 [initandlisten] connection accepted from 127.0.0.1:59308 #25 (15 connections now open) m31100| Sat Nov 17 09:43:01.703 [conn22] moveChunk data transfer progress: { active: true, ns: "test.foo", from: "clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, shardKeyPattern: { _id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 m31100| Sat Nov 17 09:43:01.706 [initandlisten] connection accepted from 127.0.0.1:59311 #26 (16 connections now open) m31100| Sat Nov 17 09:43:01.706 [conn24] end connection 127.0.0.1:59307 (15 connections now open) m31100| Sat Nov 17 09:43:01.710 [initandlisten] connection accepted from 127.0.0.1:59314 #27 (16 connections now open) m30999| Sat Nov 17 09:43:01.678 [conn1] ChunkManager: time to load chunks for test.foo: 0ms sequenceNumber: 3 version: 1|2||50a7a275e10b3cbd7ae1b532 based on: 1|0||50a7a275e10b3cbd7ae1b532 m30999| Sat Nov 17 09:43:01.692 [conn1] CMD: movechunk: { movechunk: "test.foo", find: { _id: 0.0 }, to: "clusteredstale-rs1/domU-12-31-39-16-30-A2:31200,domU-12-31-39-16-30-A2:31201,domU-12-31-39-16-30-A2:31202" } m30999| Sat Nov 17 09:43:01.693 [conn1] moving chunk ns: test.foo moving ( ns:test.fooshard: clusteredstale-rs0:clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102lastmod: 1|2||000000000000000000000000min: { _id: 0.0 }max: { _id: MaxKey }) clusteredstale-rs0:clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102 -> clusteredstale-rs1:clusteredstale-rs1/domU-12-31-39-16-30-A2:31200,domU-12-31-39-16-30-A2:31201,domU-12-31-39-16-30-A2:31202 m31100| Sat Nov 17 09:43:01.771 [conn22] moveChunk data transfer progress: { active: true, ns: "test.foo", from: "clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, shardKeyPattern: { _id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 m31200| Sat Nov 17 09:43:01.710 [ReplicaSetMonitorWatcher] starting m31200| Sat Nov 17 09:43:01.710 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-0/test.ns, filling with zeroes... m31201| Sat Nov 17 09:43:01.773 [initandlisten] connection accepted from 127.0.0.1:53841 #11 (9 connections now open) m31100| Sat Nov 17 09:43:01.787 [conn22] moveChunk data transfer progress: { active: true, ns: "test.foo", from: "clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, shardKeyPattern: { _id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 m31100| Sat Nov 17 09:43:01.796 [conn22] moveChunk data transfer progress: { active: true, ns: "test.foo", from: "clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, shardKeyPattern: { _id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 m31100| Sat Nov 17 09:43:01.813 [conn22] moveChunk data transfer progress: { active: true, ns: "test.foo", from: "clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, shardKeyPattern: { _id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 m31100| Sat Nov 17 09:43:01.846 [conn22] moveChunk data transfer progress: { active: true, ns: "test.foo", from: "clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, shardKeyPattern: { _id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 m31100| Sat Nov 17 09:43:01.911 [conn22] moveChunk data transfer progress: { active: true, ns: "test.foo", from: "clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, shardKeyPattern: { _id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 m31100| Sat Nov 17 09:43:02.040 [conn22] moveChunk data transfer progress: { active: true, ns: "test.foo", from: "clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, shardKeyPattern: { _id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 m31101| Sat Nov 17 09:43:02.048 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/test.0, size: 16MB, took 0.503 secs m31101| Sat Nov 17 09:43:02.050 [repl writer worker 1] build index test.foo { _id: 1 } m31101| Sat Nov 17 09:43:02.051 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31102| Sat Nov 17 09:43:02.052 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/test.0, size: 16MB, took 0.437 secs m31102| Sat Nov 17 09:43:02.054 [repl writer worker 1] build index test.foo { _id: 1 } m31102| Sat Nov 17 09:43:02.055 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31200| Sat Nov 17 09:43:02.185 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-0/test.ns, size: 16MB, took 0.475 secs m31200| Sat Nov 17 09:43:02.186 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-0/test.0, filling with zeroes... m31100| Sat Nov 17 09:43:02.302 [conn22] moveChunk data transfer progress: { active: true, ns: "test.foo", from: "clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, shardKeyPattern: { _id: 1.0 }, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 m31202| Sat Nov 17 09:43:02.383 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-2/test.ns, filling with zeroes... m31200| Sat Nov 17 09:43:02.379 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-0/test.0, size: 16MB, took 0.193 secs m31200| Sat Nov 17 09:43:02.382 [migrateThread] build index test.foo { _id: 1 } m31200| Sat Nov 17 09:43:02.382 [migrateThread] build index done. scanned 0 total records. 0 secs m31200| Sat Nov 17 09:43:02.382 [migrateThread] info: creating collection test.foo on add index m31200| Sat Nov 17 09:43:02.522 [migrateThread] Waiting for replication to catch up before entering critical section m31201| Sat Nov 17 09:43:02.524 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-1/test.ns, filling with zeroes... m31100| Sat Nov 17 09:43:02.819 [conn22] moveChunk data transfer progress: { active: true, ns: "test.foo", from: "clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, shardKeyPattern: { _id: 1.0 }, state: "catchup", counts: { cloned: 1, clonedBytes: 41, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 m31202| Sat Nov 17 09:43:02.920 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-2/test.ns, size: 16MB, took 0.535 secs m31202| Sat Nov 17 09:43:02.920 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-2/test.0, filling with zeroes... m31201| Sat Nov 17 09:43:03.341 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-1/test.ns, size: 16MB, took 0.621 secs m31201| Sat Nov 17 09:43:03.342 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs1-1/test.0, filling with zeroes... m31202| Sat Nov 17 09:43:03.381 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-2/test.0, size: 16MB, took 0.46 secs m31202| Sat Nov 17 09:43:03.384 [repl writer worker 1] build index test.foo { _id: 1 } m31202| Sat Nov 17 09:43:03.384 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31202| Sat Nov 17 09:43:03.384 [repl writer worker 1] info: creating collection test.foo on add index m31200| Sat Nov 17 09:43:03.523 [migrateThread] Waiting for replication to catch up before entering critical section m31200| Sat Nov 17 09:43:03.523 [migrateThread] migrate commit succeeded flushing to secondaries for 'test.foo' { _id: 0.0 } -> { _id: MaxKey } m31200| Sat Nov 17 09:43:03.523 [migrateThread] migrate commit flushed to journal for 'test.foo' { _id: 0.0 } -> { _id: MaxKey } m31201| Sat Nov 17 09:43:03.546 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs1-1/test.0, size: 16MB, took 0.204 secs m31201| Sat Nov 17 09:43:03.548 [repl writer worker 1] build index test.foo { _id: 1 } m31201| Sat Nov 17 09:43:03.549 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31201| Sat Nov 17 09:43:03.549 [repl writer worker 1] info: creating collection test.foo on add index m31100| Sat Nov 17 09:43:03.820 [conn22] moveChunk data transfer progress: { active: true, ns: "test.foo", from: "clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, shardKeyPattern: { _id: 1.0 }, state: "steady", counts: { cloned: 1, clonedBytes: 41, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0 m31100| Sat Nov 17 09:43:03.820 [conn22] moveChunk setting version to: 2|0||50a7a275e10b3cbd7ae1b532 m31200| Sat Nov 17 09:43:03.821 [initandlisten] connection accepted from 127.0.0.1:46182 #24 (17 connections now open) m31200| Sat Nov 17 09:43:03.821 [conn24] Waiting for commit to finish m31200| Sat Nov 17 09:43:03.822 [conn24] Waiting for commit to finish m31200| Sat Nov 17 09:43:03.824 [conn24] Waiting for commit to finish m31200| Sat Nov 17 09:43:03.826 [conn24] Waiting for commit to finish m31200| Sat Nov 17 09:43:03.828 [conn24] Waiting for commit to finish m31200| Sat Nov 17 09:43:03.830 [conn24] Waiting for commit to finish m31200| Sat Nov 17 09:43:03.831 [migrateThread] migrate commit succeeded flushing to secondaries for 'test.foo' { _id: 0.0 } -> { _id: MaxKey } m31200| Sat Nov 17 09:43:03.831 [migrateThread] migrate commit flushed to journal for 'test.foo' { _id: 0.0 } -> { _id: MaxKey } m31200| Sat Nov 17 09:43:03.831 [migrateThread] about to log metadata event: { _id: "domU-12-31-39-16-30-A2-2012-11-17T14:43:03-0", server: "domU-12-31-39-16-30-A2", clientAddr: ":27017", time: new Date(1353163383831), what: "moveChunk.to", ns: "test.foo", details: { min: { _id: 0.0 }, max: { _id: MaxKey }, step1 of 5: 683, step2 of 5: 44, step3 of 5: 95, step4 of 5: 0, step5 of 5: 1308 } } m29000| Sat Nov 17 09:43:03.832 [initandlisten] connection accepted from 127.0.0.1:46364 #15 (13 connections now open) m31100| Sat Nov 17 09:43:03.832 [conn22] moveChunk migrate commit accepted by TO-shard: { active: false, ns: "test.foo", from: "clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102", min: { _id: 0.0 }, max: { _id: MaxKey }, shardKeyPattern: { _id: 1.0 }, state: "done", counts: { cloned: 1, clonedBytes: 41, catchup: 0, steady: 0 }, ok: 1.0 } m31100| Sat Nov 17 09:43:03.832 [conn22] moveChunk updating self version to: 2|1||50a7a275e10b3cbd7ae1b532 through { _id: MinKey } -> { _id: 0.0 } for collection 'test.foo' m29000| Sat Nov 17 09:43:03.833 [initandlisten] connection accepted from 127.0.0.1:46365 #16 (14 connections now open) m31100| Sat Nov 17 09:43:03.834 [conn22] about to log metadata event: { _id: "domU-12-31-39-16-30-A2-2012-11-17T14:43:03-2", server: "domU-12-31-39-16-30-A2", clientAddr: "127.0.0.1:49343", time: new Date(1353163383834), what: "moveChunk.commit", ns: "test.foo", details: { min: { _id: 0.0 }, max: { _id: MaxKey }, from: "clusteredstale-rs0", to: "clusteredstale-rs1" } } m31100| Sat Nov 17 09:43:03.834 [conn22] MigrateFromStatus::done About to acquire global write lock to exit critical section m31100| Sat Nov 17 09:43:03.834 [conn22] MigrateFromStatus::done Global lock acquired m31100| Sat Nov 17 09:43:03.834 [conn22] doing delete inline m31100| Sat Nov 17 09:43:03.834 [conn22] moveChunk starting delete for: test.foo from { _id: 0.0 } -> { _id: MaxKey } m31100| Sat Nov 17 09:43:03.834 [conn22] moveChunk deleted 1 documents for test.foo from { _id: 0.0 } -> { _id: MaxKey } m31100| Sat Nov 17 09:43:03.849 [conn22] MigrateFromStatus::done About to acquire global write lock to exit critical section m31100| Sat Nov 17 09:43:03.849 [conn22] MigrateFromStatus::done Global lock acquired m31100| Sat Nov 17 09:43:03.849 [conn22] distributed lock 'test.foo/domU-12-31-39-16-30-A2:31100:1353163381:745512520' unlocked. m31100| Sat Nov 17 09:43:03.849 [conn22] about to log metadata event: { _id: "domU-12-31-39-16-30-A2-2012-11-17T14:43:03-3", server: "domU-12-31-39-16-30-A2", clientAddr: "127.0.0.1:49343", time: new Date(1353163383849), what: "moveChunk.from", ns: "test.foo", details: { min: { _id: 0.0 }, max: { _id: MaxKey }, step1 of 6: 0, step2 of 6: 1, step3 of 6: 3, step4 of 6: 2122, step5 of 6: 13, step6 of 6: 14 } } m31100| Sat Nov 17 09:43:03.849 [conn22] command admin.$cmd command: { moveChunk: "test.foo", from: "clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102", to: "clusteredstale-rs1/domU-12-31-39-16-30-A2:31200,domU-12-31-39-16-30-A2:31201,domU-12-31-39-16-30-A2:31202", fromShard: "clusteredstale-rs0", toShard: "clusteredstale-rs1", min: { _id: 0.0 }, max: { _id: MaxKey }, maxChunkSizeBytes: 52428800, shardId: "test.foo-_id_0.0", configdb: "domU-12-31-39-16-30-A2:29000", secondaryThrottle: false } ntoreturn:1 keyUpdates:0 locks(micros) W:17 r:67 w:322 reslen:37 2156ms { "millis" : 2157, "ok" : 1 } stale_clustered.js 3: test normal and slaveOk queries m30999| Sat Nov 17 09:43:03.850 [conn1] ChunkManager: time to load chunks for test.foo: 0ms sequenceNumber: 4 version: 2|1||50a7a275e10b3cbd7ae1b532 based on: 1|2||50a7a275e10b3cbd7ae1b532 m31200| Sat Nov 17 09:43:03.870 [conn19] no current chunk manager found for this shard, will initialize ReplSetTest [object Object] { "ts" : { "t" : 1353163383000, "i" : 1 }, "h" : NumberLong("2022236725213804428"), "v" : 2, "op" : "d", "ns" : "test.foo", "fromMigrate" : true, "o" : { "_id" : 1 } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163383000:1 and latest is 1353163383000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 5 { "ts" : { "t" : 1353163383000, "i" : 1 }, "h" : NumberLong("2022236725213804428"), "v" : 2, "op" : "d", "ns" : "test.foo", "fromMigrate" : true, "o" : { "_id" : 1 } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31102 is 1353163383000:1 and latest is 1353163383000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31102 is 5 ReplSetTest await synced=true ReplSetTest [object Object] { "ts" : { "t" : 1353163382000, "i" : 2 }, "h" : NumberLong("1459161149945686553"), "v" : 2, "op" : "i", "ns" : "test.foo", "fromMigrate" : true, "o" : { "_id" : 1, "b" : "b", "date" : ISODate("2012-11-17T14:43:00.333Z") } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31201 is 1353163382000:2 and latest is 1353163382000:2 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31201 is 5 { "ts" : { "t" : 1353163382000, "i" : 2 }, "h" : NumberLong("1459161149945686553"), "v" : 2, "op" : "i", "ns" : "test.foo", "fromMigrate" : true, "o" : { "_id" : 1, "b" : "b", "date" : ISODate("2012-11-17T14:43:00.333Z") } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31202 is 1353163382000:2 and latest is 1353163382000:2 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31202 is 5 ReplSetTest await synced=true m29000| Sat Nov 17 09:43:03.884 [initandlisten] connection accepted from 127.0.0.1:46366 #17 (15 connections now open) m30998| Sat Nov 17 09:43:03.885 [conn1] ChunkManager: time to load chunks for test.foo: 0ms sequenceNumber: 2 version: 2|1||50a7a275e10b3cbd7ae1b532 based on: (empty) m30998| Sat Nov 17 09:43:03.885 [conn1] starting new replica set monitor for replica set clusteredstale-rs0 with seed of domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:03.885 [initandlisten] connection accepted from 127.0.0.1:59324 #28 (17 connections now open) m30998| Sat Nov 17 09:43:03.886 [conn1] successfully connected to seed domU-12-31-39-16-30-A2:31100 for replica set clusteredstale-rs0 m30998| Sat Nov 17 09:43:03.886 [conn1] changing hosts to { 0: "domU-12-31-39-16-30-A2:31100", 1: "domU-12-31-39-16-30-A2:31102", 2: "domU-12-31-39-16-30-A2:31101" } from clusteredstale-rs0/ m30998| Sat Nov 17 09:43:03.886 [conn1] trying to add new host domU-12-31-39-16-30-A2:31100 to replica set clusteredstale-rs0 m31100| Sat Nov 17 09:43:03.886 [initandlisten] connection accepted from 127.0.0.1:59325 #29 (18 connections now open) m30998| Sat Nov 17 09:43:03.886 [conn1] successfully connected to new host domU-12-31-39-16-30-A2:31100 in replica set clusteredstale-rs0 m30998| Sat Nov 17 09:43:03.886 [conn1] trying to add new host domU-12-31-39-16-30-A2:31101 to replica set clusteredstale-rs0 m30998| Sat Nov 17 09:43:03.886 [conn1] successfully connected to new host domU-12-31-39-16-30-A2:31101 in replica set clusteredstale-rs0 m30998| Sat Nov 17 09:43:03.886 [conn1] trying to add new host domU-12-31-39-16-30-A2:31102 to replica set clusteredstale-rs0 m30998| Sat Nov 17 09:43:03.887 [conn1] successfully connected to new host domU-12-31-39-16-30-A2:31102 in replica set clusteredstale-rs0 m31100| Sat Nov 17 09:43:03.887 [initandlisten] connection accepted from 127.0.0.1:59328 #30 (19 connections now open) m31100| Sat Nov 17 09:43:03.887 [conn28] end connection 127.0.0.1:59324 (18 connections now open) m30998| Sat Nov 17 09:43:03.888 [conn1] Primary for replica set clusteredstale-rs0 changed to domU-12-31-39-16-30-A2:31100 m30998| Sat Nov 17 09:43:03.889 [conn1] replica set monitor for replica set clusteredstale-rs0 started, address is clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:43:03.889 [conn1] creating WriteBackListener for: domU-12-31-39-16-30-A2:31100 serverID: 50a7a270e54d65b18b787921 m30998| Sat Nov 17 09:43:03.889 [conn1] creating WriteBackListener for: domU-12-31-39-16-30-A2:31101 serverID: 50a7a270e54d65b18b787921 m30998| Sat Nov 17 09:43:03.889 [conn1] creating WriteBackListener for: domU-12-31-39-16-30-A2:31102 serverID: 50a7a270e54d65b18b787921 m30998| Sat Nov 17 09:43:03.889 [ReplicaSetMonitorWatcher] starting m31100| Sat Nov 17 09:43:03.889 [initandlisten] connection accepted from 127.0.0.1:59331 #31 (19 connections now open) m30998| Sat Nov 17 09:43:03.890 [conn1] starting new replica set monitor for replica set clusteredstale-rs1 with seed of domU-12-31-39-16-30-A2:31200,domU-12-31-39-16-30-A2:31201,domU-12-31-39-16-30-A2:31202 m30998| Sat Nov 17 09:43:03.890 [conn1] successfully connected to seed domU-12-31-39-16-30-A2:31200 for replica set clusteredstale-rs1 m30998| Sat Nov 17 09:43:03.890 [conn1] changing hosts to { 0: "domU-12-31-39-16-30-A2:31200", 1: "domU-12-31-39-16-30-A2:31202", 2: "domU-12-31-39-16-30-A2:31201" } from clusteredstale-rs1/ m30998| Sat Nov 17 09:43:03.890 [conn1] trying to add new host domU-12-31-39-16-30-A2:31200 to replica set clusteredstale-rs1 m30998| Sat Nov 17 09:43:03.890 [conn1] successfully connected to new host domU-12-31-39-16-30-A2:31200 in replica set clusteredstale-rs1 m30998| Sat Nov 17 09:43:03.890 [conn1] trying to add new host domU-12-31-39-16-30-A2:31201 to replica set clusteredstale-rs1 m31201| Sat Nov 17 09:43:03.890 [initandlisten] connection accepted from 127.0.0.1:53860 #12 (10 connections now open) m30998| Sat Nov 17 09:43:03.891 [conn1] successfully connected to new host domU-12-31-39-16-30-A2:31201 in replica set clusteredstale-rs1 m30998| Sat Nov 17 09:43:03.891 [conn1] trying to add new host domU-12-31-39-16-30-A2:31202 to replica set clusteredstale-rs1 m31202| Sat Nov 17 09:43:03.891 [initandlisten] connection accepted from 127.0.0.1:38321 #12 (10 connections now open) m30998| Sat Nov 17 09:43:03.891 [conn1] successfully connected to new host domU-12-31-39-16-30-A2:31202 in replica set clusteredstale-rs1 m30998| Sat Nov 17 09:43:03.893 [conn1] Primary for replica set clusteredstale-rs1 changed to domU-12-31-39-16-30-A2:31200 m31201| Sat Nov 17 09:43:03.893 [initandlisten] connection accepted from 127.0.0.1:53863 #13 (11 connections now open) m31202| Sat Nov 17 09:43:03.893 [initandlisten] connection accepted from 127.0.0.1:38324 #13 (11 connections now open) m30998| Sat Nov 17 09:43:03.894 [conn1] replica set monitor for replica set clusteredstale-rs1 started, address is clusteredstale-rs1/domU-12-31-39-16-30-A2:31200,domU-12-31-39-16-30-A2:31201,domU-12-31-39-16-30-A2:31202 m30998| Sat Nov 17 09:43:03.894 [conn1] creating WriteBackListener for: domU-12-31-39-16-30-A2:31200 serverID: 50a7a270e54d65b18b787921 m30998| Sat Nov 17 09:43:03.894 [conn1] creating WriteBackListener for: domU-12-31-39-16-30-A2:31201 serverID: 50a7a270e54d65b18b787921 m30998| Sat Nov 17 09:43:03.894 [conn1] creating WriteBackListener for: domU-12-31-39-16-30-A2:31202 serverID: 50a7a270e54d65b18b787921 m31201| Sat Nov 17 09:43:03.895 [initandlisten] connection accepted from 127.0.0.1:53867 #14 (12 connections now open) m31100| Sat Nov 17 09:43:03.896 [initandlisten] connection accepted from 127.0.0.1:59342 #32 (20 connections now open) m31101| Sat Nov 17 09:43:03.886 [initandlisten] connection accepted from 127.0.0.1:57098 #12 (10 connections now open) m31101| Sat Nov 17 09:43:03.888 [initandlisten] connection accepted from 127.0.0.1:57101 #13 (11 connections now open) m31101| Sat Nov 17 09:43:03.895 [initandlisten] connection accepted from 127.0.0.1:57112 #14 (12 connections now open) m31102| Sat Nov 17 09:43:03.887 [initandlisten] connection accepted from 127.0.0.1:44525 #12 (10 connections now open) m31102| Sat Nov 17 09:43:03.888 [initandlisten] connection accepted from 127.0.0.1:44528 #13 (11 connections now open) stale_clustered.js 5: overflow oplog m31200| Sat Nov 17 09:43:03.890 [initandlisten] connection accepted from 127.0.0.1:46194 #25 (18 connections now open) m31200| Sat Nov 17 09:43:03.890 [initandlisten] connection accepted from 127.0.0.1:46195 #26 (19 connections now open) m31200| Sat Nov 17 09:43:03.891 [initandlisten] connection accepted from 127.0.0.1:46198 #27 (20 connections now open) m31200| Sat Nov 17 09:43:03.892 [conn25] end connection 127.0.0.1:46194 (19 connections now open) m31200| Sat Nov 17 09:43:03.894 [initandlisten] connection accepted from 127.0.0.1:46201 #28 (20 connections now open) m31100| Sat Nov 17 09:43:03.900 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-0/_overflow.ns, filling with zeroes... m31100| Sat Nov 17 09:43:04.165 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-0/_overflow.ns, size: 16MB, took 0.264 secs m31100| Sat Nov 17 09:43:04.165 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-0/_overflow.0, filling with zeroes... ReplSetTest [object Object] m31100| Sat Nov 17 09:43:04.398 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-0/_overflow.0, size: 16MB, took 0.23 secs m31100| Sat Nov 17 09:43:04.400 [conn1] build index _overflow.coll { _id: 1 } m31100| Sat Nov 17 09:43:04.400 [conn1] build index done. scanned 0 total records. 0 secs m31100| Sat Nov 17 09:43:04.400 [conn1] insert _overflow.coll keyUpdates:0 locks(micros) w:500764 500ms m31101| Sat Nov 17 09:43:04.402 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/_overflow.ns, filling with zeroes... m31102| Sat Nov 17 09:43:04.403 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/_overflow.ns, filling with zeroes... m31102| Sat Nov 17 09:43:04.962 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/_overflow.ns, size: 16MB, took 0.558 secs m31101| Sat Nov 17 09:43:04.953 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/_overflow.ns, size: 16MB, took 0.55 secs m31101| Sat Nov 17 09:43:04.953 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/_overflow.0, filling with zeroes... m31102| Sat Nov 17 09:43:04.962 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/_overflow.0, filling with zeroes... m31101| Sat Nov 17 09:43:05.453 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/_overflow.0, size: 16MB, took 0.499 secs m31102| Sat Nov 17 09:43:05.453 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/_overflow.0, size: 16MB, took 0.49 secs m31102| Sat Nov 17 09:43:05.455 [repl writer worker 1] build index _overflow.coll { _id: 1 } m31102| Sat Nov 17 09:43:05.456 [repl writer worker 1] build index done. scanned 0 total records. 0 secs { "ts" : { "t" : 1353163384000, "i" : 1 }, "h" : NumberLong("9082384864683833077"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2779e9ba0197a982fd4"), "replicated" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163384000:1 and latest is 1353163384000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 6 { "ts" : { "t" : 1353163384000, "i" : 1 }, "h" : NumberLong("9082384864683833077"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2779e9ba0197a982fd4"), "replicated" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31102 is 1353163384000:1 and latest is 1353163384000:1 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31102 is 6 ReplSetTest await synced=true ReplSetTest n: 2 ports: [ 31100, 31101, 31102 ] 31102 number ReplSetTest stop *** Shutting down mongod in port 31102 *** m31101| Sat Nov 17 09:43:05.455 [repl writer worker 1] build index _overflow.coll { _id: 1 } m31101| Sat Nov 17 09:43:05.456 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m30999| Sat Nov 17 09:43:05.462 [WriteBackListener-domU-12-31-39-16-30-A2:31102] DBClientCursor::init call() failed m31101| Sat Nov 17 09:43:05.462 [conn9] end connection 127.0.0.1:57064 (11 connections now open) m31102| Sat Nov 17 09:43:05.461 got signal 15 (Terminated), will terminate after current cmd ends m31102| Sat Nov 17 09:43:05.461 [interruptThread] now exiting m31102| Sat Nov 17 09:43:05.461 dbexit: m31102| Sat Nov 17 09:43:05.461 [interruptThread] shutdown: going to close listening sockets... m31102| Sat Nov 17 09:43:05.461 [interruptThread] closing listening socket: 17 m31102| Sat Nov 17 09:43:05.461 [interruptThread] closing listening socket: 18 m31102| Sat Nov 17 09:43:05.461 [interruptThread] closing listening socket: 19 m31102| Sat Nov 17 09:43:05.461 [interruptThread] removing socket file: /tmp/mongodb-31102.sock m31102| Sat Nov 17 09:43:05.461 [interruptThread] shutdown: going to flush diaglog... m31102| Sat Nov 17 09:43:05.461 [interruptThread] shutdown: going to close sockets... m31102| Sat Nov 17 09:43:05.461 [interruptThread] shutdown: waiting for fs preallocator... m31102| Sat Nov 17 09:43:05.461 [interruptThread] shutdown: lock for final commit... m31102| Sat Nov 17 09:43:05.461 [interruptThread] shutdown: final commit... m31100| Sat Nov 17 09:43:05.462 [conn5] end connection 127.0.0.1:49266 (19 connections now open) m30998| Sat Nov 17 09:43:05.463 [WriteBackListener-domU-12-31-39-16-30-A2:31102] DBClientCursor::init call() failed m30999| Sat Nov 17 09:43:05.483 [WriteBackListener-domU-12-31-39-16-30-A2:31102] Detecting bad connection created at 0 microSec, clearing pool for domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:43:05.483 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : DBClientBase::findN: transport error: domU-12-31-39-16-30-A2:31102 ns: admin.$cmd query: { writebacklisten: ObjectId('50a7a270e10b3cbd7ae1b530') } m30998| Sat Nov 17 09:43:05.483 [WriteBackListener-domU-12-31-39-16-30-A2:31102] Detecting bad connection created at 0 microSec, clearing pool for domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:43:05.485 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : DBClientBase::findN: transport error: domU-12-31-39-16-30-A2:31102 ns: admin.$cmd query: { writebacklisten: ObjectId('50a7a270e54d65b18b787921') } m31102| Sat Nov 17 09:43:05.524 [interruptThread] shutdown: closing all files... m31102| Sat Nov 17 09:43:05.525 [interruptThread] closeAllFiles() finished m31102| Sat Nov 17 09:43:05.525 [interruptThread] journalCleanup... m31102| Sat Nov 17 09:43:05.525 [interruptThread] removeJournalFiles m31102| Sat Nov 17 09:43:05.528 [interruptThread] shutdown: removing fs lock... m31102| Sat Nov 17 09:43:05.528 dbexit: really exiting now m31101| Sat Nov 17 09:43:05.583 [rsHealthPoll] DBClientCursor::init call() failed m31101| Sat Nov 17 09:43:05.583 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:05.584 [rsHealthPoll] replSet info domU-12-31-39-16-30-A2:31102 is down (or slow to respond): m31101| Sat Nov 17 09:43:05.584 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state DOWN m31100| Sat Nov 17 09:43:06.418 [initandlisten] connection accepted from 127.0.0.1:59349 #33 (20 connections now open) m31101| Sat Nov 17 09:43:06.419 [initandlisten] connection accepted from 127.0.0.1:57122 #15 (12 connections now open) m30999| Sat Nov 17 09:43:06.419 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m31200| Sat Nov 17 09:43:06.420 [initandlisten] connection accepted from 127.0.0.1:46213 #29 (21 connections now open) m31202| Sat Nov 17 09:43:06.421 [initandlisten] connection accepted from 127.0.0.1:38339 #14 (12 connections now open) m31201| Sat Nov 17 09:43:06.421 [initandlisten] connection accepted from 127.0.0.1:53878 #15 (13 connections now open) Sat Nov 17 09:43:06.462 shell: stopped mongo program on port 31102 ReplSetTest overflow inserting 10000 m31100| Sat Nov 17 09:43:06.523 [conn15] end connection 127.0.0.1:49291 (19 connections now open) m31100| Sat Nov 17 09:43:06.523 [conn13] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:49287] m30999| Sat Nov 17 09:43:06.637 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:43:06.638 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:43:07.359 DBClientCursor::init call() failed ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] m31100| Sat Nov 17 09:43:07.368 [rsHealthPoll] DBClientCursor::init call() failed m31100| Sat Nov 17 09:43:07.368 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:07.368 [rsHealthPoll] replSet info domU-12-31-39-16-30-A2:31102 is down (or slow to respond): m31100| Sat Nov 17 09:43:07.368 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state DOWN { "ts" : { "t" : 1353163387000, "i" : 3404 }, "h" : NumberLong("-2885278966615254399"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a27a9e9ba0197a985477"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163387000:3404 and latest is 1353163387000:4025 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 9385 m31101| Sat Nov 17 09:43:07.585 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m30999| Sat Nov 17 09:43:08.639 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:43:08.639 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:09.369 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying { "ts" : { "t" : 1353163387000, "i" : 4025 }, "h" : NumberLong("-4182107704755367770"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a27b9e9ba0197a9856e4"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163387000:4025 and latest is 1353163387000:4025 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 10006 ReplSetTest await synced=true ReplSetTest overflow count : 10007 prev : 7 ReplSetTest overflow inserting 10000 m31101| Sat Nov 17 09:43:09.619 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying Sat Nov 17 09:43:10.247 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:10.249 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163389000, "i" : 4468 }, "h" : NumberLong("7586116269779208600"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a27d9e9ba0197a986858"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163389000:4468 and latest is 1353163390000:3941 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 14474 m31100| Sat Nov 17 09:43:11.369 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying Sat Nov 17 09:43:11.455 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m31101| Sat Nov 17 09:43:11.620 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m30999| Sat Nov 17 09:43:11.641 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:43:11.641 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:43:11.711 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed { "ts" : { "t" : 1353163390000, "i" : 3941 }, "h" : NumberLong("-5244169851279185557"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a27d9e9ba0197a987df4"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163390000:3941 and latest is 1353163390000:3941 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 20006 ReplSetTest await synced=true ReplSetTest overflow count : 20007 prev : 10007 ReplSetTest overflow inserting 10000 Sat Nov 17 09:43:13.018 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:13.018 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163392000, "i" : 2191 }, "h" : NumberLong("6136608892088748849"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2809e9ba0197a988683"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163392000:2191 and latest is 1353163393000:334 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 27198 m31101| Sat Nov 17 09:43:13.368 [conn4] end connection 127.0.0.1:37070 (11 connections now open) m31101| Sat Nov 17 09:43:13.368 [initandlisten] connection accepted from 127.0.0.1:57155 #16 (12 connections now open) m31100| Sat Nov 17 09:43:13.370 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:13.620 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:13.890 [initandlisten] connection accepted from 127.0.0.1:59387 #34 (19 connections now open) m31101| Sat Nov 17 09:43:13.893 [initandlisten] connection accepted from 127.0.0.1:57160 #17 (13 connections now open) m30998| Sat Nov 17 09:43:13.893 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m31200| Sat Nov 17 09:43:13.894 [initandlisten] connection accepted from 127.0.0.1:46251 #30 (22 connections now open) m31201| Sat Nov 17 09:43:13.895 [initandlisten] connection accepted from 127.0.0.1:53916 #16 (14 connections now open) m31202| Sat Nov 17 09:43:13.895 [initandlisten] connection accepted from 127.0.0.1:38377 #15 (13 connections now open) { "ts" : { "t" : 1353163393000, "i" : 334 }, "h" : NumberLong("-2120517803263340421"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2809e9ba0197a98a504"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163393000:334 and latest is 1353163393000:334 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 30006 ReplSetTest await synced=true ReplSetTest overflow count : 30007 prev : 20007 ReplSetTest overflow inserting 10000 m31201| Sat Nov 17 09:43:15.323 [conn4] end connection 127.0.0.1:36315 (13 connections now open) m31201| Sat Nov 17 09:43:15.326 [initandlisten] connection accepted from 127.0.0.1:53923 #17 (14 connections now open) m31100| Sat Nov 17 09:43:15.379 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m30998| Sat Nov 17 09:43:15.748 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:43:15.749 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:15.779 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:15.749 [conn4] end connection 127.0.0.1:49265 (18 connections now open) Sat Nov 17 09:43:15.952 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:15.952 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] m31100| Sat Nov 17 09:43:15.788 [initandlisten] connection accepted from 127.0.0.1:59402 #35 (19 connections now open) { "ts" : { "t" : 1353163395000, "i" : 7062 }, "h" : NumberLong("4576998053072282474"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2839e9ba0197a98c09a"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163395000:7062 and latest is 1353163395000:10000 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 37068 m30999| Sat Nov 17 09:43:16.424 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:43:16.424 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:17.381 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31200| Sat Nov 17 09:43:17.522 [conn4] end connection 127.0.0.1:34123 (21 connections now open) m31200| Sat Nov 17 09:43:17.523 [initandlisten] connection accepted from 127.0.0.1:46268 #31 (22 connections now open) m31101| Sat Nov 17 09:43:17.780 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying { "ts" : { "t" : 1353163395000, "i" : 10000 }, "h" : NumberLong("-4858164165572459485"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2839e9ba0197a98cc14"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163395000:10000 and latest is 1353163395000:10000 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 40006 ReplSetTest await synced=true ReplSetTest overflow count : 40007 prev : 30007 ReplSetTest overflow inserting 10000 m31200| Sat Nov 17 09:43:18.513 [conn5] end connection 127.0.0.1:34124 (21 connections now open) m31200| Sat Nov 17 09:43:18.514 [initandlisten] connection accepted from 127.0.0.1:46273 #32 (22 connections now open) m31100| Sat Nov 17 09:43:18.559 [conn1] insert _overflow.coll keyUpdates:0 locks(micros) w:370165 370ms Sat Nov 17 09:43:19.153 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:19.154 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163399000, "i" : 102 }, "h" : NumberLong("-3437985844075535216"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2869e9ba0197a98eb16"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163399000:102 and latest is 1353163399000:2164 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 47944 m31100| Sat Nov 17 09:43:19.385 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:19.802 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m30998| Sat Nov 17 09:43:20.750 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:43:20.751 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163399000, "i" : 2164 }, "h" : NumberLong("5355091066700200043"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2869e9ba0197a98f324"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163399000:2164 and latest is 1353163399000:2164 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 50006 ReplSetTest await synced=true ReplSetTest overflow count : 50007 prev : 40007 ReplSetTest overflow inserting 10000 Sat Nov 17 09:43:21.670 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:43:21.673 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:43:21.723 [TTLMonitor] query admin.system.indexes query: { expireAfterSeconds: { $exists: true } } ntoreturn:0 ntoskip:0 nscanned:1 keyUpdates:0 locks(micros) r:476611 nreturned:0 reslen:20 476ms m31200| Sat Nov 17 09:43:21.766 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:43:21.767 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:21.817 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:21.671 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying Sat Nov 17 09:43:22.192 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:22.192 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163401000, "i" : 6043 }, "h" : NumberLong("5018749292593789623"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2899e9ba0197a990abf"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163401000:6043 and latest is 1353163402000:1584 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 56049 m31100| Sat Nov 17 09:43:23.672 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:23.817 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m30998| Sat Nov 17 09:43:23.897 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:43:23.897 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163402000, "i" : 1584 }, "h" : NumberLong("-8205997906010098669"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2899e9ba0197a991a34"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163402000:1584 and latest is 1353163402000:1584 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 60006 ReplSetTest await synced=true ReplSetTest overflow count : 60007 prev : 50007 ReplSetTest overflow inserting 10000 Sat Nov 17 09:43:25.066 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:25.069 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163404000, "i" : 5002 }, "h" : NumberLong("-7909352486825157084"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a28c9e9ba0197a992dbe"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163404000:5002 and latest is 1353163405000:365 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 65008 m31100| Sat Nov 17 09:43:25.672 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:25.818 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m30999| Sat Nov 17 09:43:26.427 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:43:26.427 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:43:26.752 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:43:26.753 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163405000, "i" : 365 }, "h" : NumberLong("-6012824362054548400"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a28c9e9ba0197a994144"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163405000:365 and latest is 1353163405000:365 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 70006 ReplSetTest await synced=true ReplSetTest overflow count : 70007 prev : 60007 ReplSetTest overflow inserting 10000 m31100| Sat Nov 17 09:43:27.679 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:27.680 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:27.680 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:27.680 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:27.680 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:27.680 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:43:27.945 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:27.948 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163407000, "i" : 2923 }, "h" : NumberLong("568355323012376694"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a28f9e9ba0197a994caf"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163407000:2923 and latest is 1353163407000:10000 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 72929 m31101| Sat Nov 17 09:43:27.906 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31202| Sat Nov 17 09:43:29.659 [conn8] end connection 127.0.0.1:38275 (12 connections now open) m31202| Sat Nov 17 09:43:29.659 [initandlisten] connection accepted from 127.0.0.1:38432 #16 (13 connections now open) m31100| Sat Nov 17 09:43:29.685 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:29.685 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:29.685 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:29.685 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:29.685 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:29.685 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:29.906 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:29.907 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:29.907 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:29.907 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:29.907 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:29.907 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163407000, "i" : 10000 }, "h" : NumberLong("6125017546618306872"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a28f9e9ba0197a996854"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163407000:10000 and latest is 1353163407000:10000 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 80006 ReplSetTest await synced=true ReplSetTest overflow count : 80007 prev : 70007 ReplSetTest overflow inserting 10000 Sat Nov 17 09:43:30.659 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:30.660 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163410000, "i" : 6054 }, "h" : NumberLong("8487941063673450184"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2929e9ba0197a998364"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163410000:6054 and latest is 1353163410000:9126 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 86934 m31202| Sat Nov 17 09:43:31.675 [conn9] end connection 127.0.0.1:38284 (12 connections now open) m31202| Sat Nov 17 09:43:31.675 [initandlisten] connection accepted from 127.0.0.1:38449 #17 (13 connections now open) Sat Nov 17 09:43:31.677 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:43:31.677 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:31.686 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:31.686 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:31.686 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:31.687 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:31.687 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:31.687 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:43:31.769 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:43:31.769 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:31.908 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:31.908 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:31.908 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:31.909 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:31.909 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:31.909 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31201| Sat Nov 17 09:43:32.050 [conn11] end connection 127.0.0.1:53841 (13 connections now open) m31201| Sat Nov 17 09:43:32.050 [initandlisten] connection accepted from 127.0.0.1:54007 #18 (14 connections now open) { "ts" : { "t" : 1353163410000, "i" : 9126 }, "h" : NumberLong("-2957099028040012600"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2929e9ba0197a998f64"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163410000:9126 and latest is 1353163410000:9126 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 90006 ReplSetTest await synced=true ReplSetTest overflow count : 90007 prev : 80007 ReplSetTest overflow inserting 10000 m31100| Sat Nov 17 09:43:33.254 [conn1] insert _overflow.coll keyUpdates:0 locks(micros) w:378288 189ms m31100| Sat Nov 17 09:43:33.429 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-0/_overflow.1, filling with zeroes... m31100| Sat Nov 17 09:43:33.688 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:33.688 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:33.688 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:33.688 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:33.689 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:33.689 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:43:33.753 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:43:33.754 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:33.826 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-0/_overflow.1, size: 32MB, took 0.396 secs m31100| Sat Nov 17 09:43:33.826 [conn1] insert _overflow.coll keyUpdates:0 locks(micros) w:397326 397ms m30998| Sat Nov 17 09:43:33.910 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:43:33.910 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:33.913 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:33.913 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:33.914 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:33.915 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:33.927 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:33.928 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:33.993 [conn1] insert _overflow.coll keyUpdates:0 locks(micros) w:232055 116ms m31101| Sat Nov 17 09:43:34.004 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/_overflow.1, filling with zeroes... Sat Nov 17 09:43:34.558 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:34.561 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] m31101| Sat Nov 17 09:43:34.526 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/_overflow.1, size: 32MB, took 0.521 secs m31100| Sat Nov 17 09:43:34.529 [conn1] insert _overflow.coll keyUpdates:0 locks(micros) w:731390 365ms { "ts" : { "t" : 1353163413000, "i" : 4359 }, "h" : NumberLong("-6348193969002955871"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2959e9ba0197a99a7c8"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163413000:4359 and latest is 1353163414000:3521 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 96250 m31100| Sat Nov 17 09:43:35.690 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:35.690 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:35.690 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:35.690 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:35.690 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:35.690 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:35.929 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:35.929 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:35.929 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:35.929 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:35.929 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:35.929 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:43:36.429 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:43:36.429 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163414000, "i" : 3521 }, "h" : NumberLong("-1131224681388659419"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2959e9ba0197a99b674"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163414000:3521 and latest is 1353163414000:3521 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 100006 ReplSetTest await synced=true ReplSetTest overflow count : 100007 prev : 90007 ReplSetTest overflow inserting 10000 Sat Nov 17 09:43:37.247 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:37.248 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163416000, "i" : 5002 }, "h" : NumberLong("2729912697108097094"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2989e9ba0197a99c9fe"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163416000:5002 and latest is 1353163417000:2867 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 105008 m31100| Sat Nov 17 09:43:37.691 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:37.692 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:37.692 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:37.692 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:37.692 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:37.692 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:37.930 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:37.930 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:37.930 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:37.931 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:37.931 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:37.931 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163417000, "i" : 2867 }, "h" : NumberLong("-7790348471327726764"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2999e9ba0197a99dd84"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163417000:2867 and latest is 1353163417000:2867 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 110006 ReplSetTest await synced=true ReplSetTest overflow count : 110007 prev : 100007 ReplSetTest overflow inserting 10000 m31100| Sat Nov 17 09:43:39.766 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:39.766 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:39.767 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:39.767 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:39.767 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:39.767 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:39.999 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:39.999 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:40.000 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:40.000 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:40.000 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:40.000 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:43:40.117 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:40.117 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163419000, "i" : 4763 }, "h" : NumberLong("6186418992903003642"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a29b9e9ba0197a99f01f"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163419000:4763 and latest is 1353163420000:1113 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 114769 Sat Nov 17 09:43:41.681 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:43:41.683 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:43:41.759 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:43:41.759 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:41.768 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:41.768 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:41.768 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:41.768 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:41.768 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:41.768 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:43:41.770 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:43:41.770 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:42.001 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:42.001 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:42.001 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:42.001 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:42.001 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:42.001 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163420000, "i" : 1113 }, "h" : NumberLong("6071447187606454565"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a29b9e9ba0197a9a0494"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163420000:1113 and latest is 1353163420000:1113 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 120006 ReplSetTest await synced=true ReplSetTest overflow count : 120007 prev : 110007 ReplSetTest overflow inserting 10000 Sat Nov 17 09:43:42.913 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:42.916 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163422000, "i" : 4832 }, "h" : NumberLong("-2572946345008340139"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a29e9e9ba0197a9a1774"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163422000:4832 and latest is 1353163422000:10000 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 124838 m31101| Sat Nov 17 09:43:43.737 [conn16] end connection 127.0.0.1:57155 (12 connections now open) m31101| Sat Nov 17 09:43:43.738 [initandlisten] connection accepted from 127.0.0.1:57351 #18 (13 connections now open) m31100| Sat Nov 17 09:43:43.769 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:43.769 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:43.770 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:43.770 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:43.770 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:43.770 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:43:43.913 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:43:43.913 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:44.002 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:44.002 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:44.003 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:44.003 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:44.003 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:44.003 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163422000, "i" : 10000 }, "h" : NumberLong("8336038357822097229"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a29e9e9ba0197a9a2ba4"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163422000:10000 and latest is 1353163422000:10000 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 130006 ReplSetTest await synced=true ReplSetTest overflow count : 130007 prev : 120007 ReplSetTest overflow inserting 10000 m31201| Sat Nov 17 09:43:45.803 [conn17] end connection 127.0.0.1:53923 (13 connections now open) m31201| Sat Nov 17 09:43:45.805 [initandlisten] connection accepted from 127.0.0.1:54124 #19 (14 connections now open) m31100| Sat Nov 17 09:43:45.804 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:45.804 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:45.804 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:45.805 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:45.805 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:45.815 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:43:45.857 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:45.858 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163425000, "i" : 5623 }, "h" : NumberLong("4528356762899914363"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2a19e9ba0197a9a419b"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163425000:5623 and latest is 1353163425000:10000 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 135629 m31100| Sat Nov 17 09:43:45.908 [conn35] end connection 127.0.0.1:59402 (18 connections now open) m31100| Sat Nov 17 09:43:45.908 [initandlisten] connection accepted from 127.0.0.1:59603 #36 (19 connections now open) m31101| Sat Nov 17 09:43:46.004 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:46.004 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:46.004 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:46.005 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:46.005 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:46.005 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:43:46.432 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:43:46.432 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:43:47.747 [conn31] end connection 127.0.0.1:46268 (21 connections now open) m31200| Sat Nov 17 09:43:47.747 [initandlisten] connection accepted from 127.0.0.1:46475 #33 (22 connections now open) m31100| Sat Nov 17 09:43:47.816 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:47.816 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:47.816 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:47.816 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:47.816 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:47.816 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163425000, "i" : 10000 }, "h" : NumberLong("-1841580550850939019"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2a19e9ba0197a9a52b4"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163425000:10000 and latest is 1353163425000:10000 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 140006 ReplSetTest await synced=true ReplSetTest overflow count : 140007 prev : 130007 ReplSetTest overflow inserting 10000 m31101| Sat Nov 17 09:43:48.336 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:48.336 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:48.337 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:48.337 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:48.337 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:48.337 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:43:48.725 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:48.726 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163428000, "i" : 6519 }, "h" : NumberLong("899383041906943206"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2a49e9ba0197a9a7568"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163428000:6519 and latest is 1353163428000:7635 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 148890 m31200| Sat Nov 17 09:43:49.558 [conn32] end connection 127.0.0.1:46273 (21 connections now open) m31200| Sat Nov 17 09:43:49.559 [initandlisten] connection accepted from 127.0.0.1:46491 #34 (22 connections now open) m31100| Sat Nov 17 09:43:49.817 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:49.818 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:49.820 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:49.820 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:49.820 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:49.820 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:50.337 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:50.337 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:50.338 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:50.338 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:50.338 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:50.338 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163428000, "i" : 7635 }, "h" : NumberLong("4284449209777977092"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2a49e9ba0197a9a79c4"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163428000:7635 and latest is 1353163428000:7635 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 150006 ReplSetTest await synced=true ReplSetTest overflow count : 150007 prev : 140007 ReplSetTest overflow inserting 10000 m30999| Sat Nov 17 09:43:50.768 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:43:50.768 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:43:51.453 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:51.454 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163431000, "i" : 1597 }, "h" : NumberLong("-8379930703356330098"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2a79e9ba0197a9a9112"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163431000:1597 and latest is 1353163431000:5631 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 155972 Sat Nov 17 09:43:51.707 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:43:51.709 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:43:51.772 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:43:51.772 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:51.821 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:51.821 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:51.821 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:51.822 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:51.822 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:51.822 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:52.339 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:52.339 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:52.339 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:52.340 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:52.340 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:52.340 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163431000, "i" : 5631 }, "h" : NumberLong("4630425656233015495"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2a79e9ba0197a9aa0d4"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163431000:5631 and latest is 1353163431000:5631 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 160006 ReplSetTest await synced=true ReplSetTest overflow count : 160007 prev : 150007 ReplSetTest overflow inserting 10000 m31100| Sat Nov 17 09:43:53.976 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:53.976 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:53.977 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:53.977 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:53.977 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:53.977 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:43:53.991 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:43:53.991 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:54.365 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:54.381 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:54.382 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:54.382 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:54.382 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:54.389 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:43:54.554 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:54.554 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163434000, "i" : 5468 }, "h" : NumberLong("9204159668628618872"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2aa9e9ba0197a9ac3fe"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163434000:5468 and latest is 1353163434000:6466 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 169008 m31100| Sat Nov 17 09:43:55.977 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:55.978 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:55.978 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:55.978 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:55.978 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:55.978 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:56.389 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:56.390 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:56.390 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:56.390 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:56.390 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:56.390 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:43:56.434 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:43:56.434 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163434000, "i" : 6466 }, "h" : NumberLong("-1511960684772132329"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2aa9e9ba0197a9ac7e4"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163434000:6466 and latest is 1353163434000:6466 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 170006 ReplSetTest await synced=true ReplSetTest overflow count : 170007 prev : 160007 ReplSetTest overflow inserting 10000 Sat Nov 17 09:43:57.427 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:43:57.428 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163437000, "i" : 4194 }, "h" : NumberLong("3765018925226300899"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2ad9e9ba0197a9aeb77"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163437000:4194 and latest is 1353163437000:5087 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 179113 m31100| Sat Nov 17 09:43:57.979 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:57.979 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:57.979 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:57.980 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:43:57.980 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:57.980 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:43:57.980 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:58.391 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:58.391 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:58.392 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:43:58.392 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:58.392 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:43:58.392 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163437000, "i" : 5087 }, "h" : NumberLong("7075866539417374906"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2ad9e9ba0197a9aeef4"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163437000:5087 and latest is 1353163437000:5087 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 180006 ReplSetTest await synced=true ReplSetTest overflow count : 180007 prev : 170007 ReplSetTest overflow inserting 10000 m31202| Sat Nov 17 09:43:59.719 [conn16] end connection 127.0.0.1:38432 (12 connections now open) m31202| Sat Nov 17 09:43:59.892 [initandlisten] connection accepted from 127.0.0.1:38699 #18 (13 connections now open) m31100| Sat Nov 17 09:44:00.110 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:00.110 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:00.142 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:00.165 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:00.165 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:00.165 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:00.226 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:44:00.226 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163439000, "i" : 3385 }, "h" : NumberLong("8314648830327519259"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2af9e9ba0197a9afc2d"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163439000:3385 and latest is 1353163440000:2469 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 183391 m31101| Sat Nov 17 09:44:00.399 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:00.399 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:00.399 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:00.399 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:00.399 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:00.400 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:00.400 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:44:00.769 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:00.769 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:01.711 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:01.713 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:44:01.773 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:44:01.773 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31202| Sat Nov 17 09:44:01.894 [conn17] end connection 127.0.0.1:38449 (12 connections now open) m31202| Sat Nov 17 09:44:01.894 [initandlisten] connection accepted from 127.0.0.1:38721 #19 (13 connections now open) m31100| Sat Nov 17 09:44:02.166 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:02.167 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:02.167 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:02.167 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:02.169 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:02.170 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31201| Sat Nov 17 09:44:02.344 [conn18] end connection 127.0.0.1:54007 (13 connections now open) m31201| Sat Nov 17 09:44:02.345 [initandlisten] connection accepted from 127.0.0.1:54270 #20 (14 connections now open) { "ts" : { "t" : 1353163440000, "i" : 2469 }, "h" : NumberLong("467414558790844991"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2af9e9ba0197a9b1604"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163440000:2469 and latest is 1353163440000:2469 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 190006 ReplSetTest await synced=true ReplSetTest overflow count : 190007 prev : 180007 ReplSetTest overflow inserting 10000 m31101| Sat Nov 17 09:44:02.453 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:02.456 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:02.510 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:02.510 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:02.510 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:02.510 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:03.122 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:44:03.126 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163442000, "i" : 5848 }, "h" : NumberLong("-9106354164564686245"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2b29e9ba0197a9b2cdc"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163442000:5848 and latest is 1353163443000:1359 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 195854 m30998| Sat Nov 17 09:44:04.007 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:44:04.008 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:04.170 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:04.170 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:04.172 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:04.172 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:04.173 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:04.173 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:04.511 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:04.511 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:04.512 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:04.512 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:04.512 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:04.512 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163443000, "i" : 1359 }, "h" : NumberLong("-5659950096558262158"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2b29e9ba0197a9b3d14"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163443000:1359 and latest is 1353163443000:1359 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 200006 ReplSetTest await synced=true ReplSetTest overflow count : 200007 prev : 190007 ReplSetTest overflow inserting 10000 Sat Nov 17 09:44:06.019 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:44:06.020 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] m31100| Sat Nov 17 09:44:06.174 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:06.174 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:06.174 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:06.174 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:06.175 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:06.175 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163445000, "i" : 7900 }, "h" : NumberLong("2225405900746303752"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2b59e9ba0197a9b5bf0"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163445000:7900 and latest is 1353163446000:488 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 207906 m30999| Sat Nov 17 09:44:06.436 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:06.437 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:06.513 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:06.513 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:06.513 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:06.513 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:06.513 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:06.514 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:08.176 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:08.176 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:08.176 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:08.176 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:08.176 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:08.176 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163446000, "i" : 488 }, "h" : NumberLong("6017038706113476954"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2b59e9ba0197a9b6424"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163446000:488 and latest is 1353163446000:488 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 210006 ReplSetTest await synced=true ReplSetTest overflow count : 210007 prev : 200007 ReplSetTest overflow inserting 10000 m31101| Sat Nov 17 09:44:08.525 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:08.525 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:08.525 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:08.526 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:08.526 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:08.526 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:09.063 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:44:09.065 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163448000, "i" : 7453 }, "h" : NumberLong("7651790036601380893"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2b89e9ba0197a9b8141"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163448000:7453 and latest is 1353163449000:942 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 217459 m31100| Sat Nov 17 09:44:10.177 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:10.178 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:10.178 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:10.178 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:10.178 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:10.178 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:10.526 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:10.526 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:10.527 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:10.527 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:10.527 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:10.527 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163449000, "i" : 942 }, "h" : NumberLong("4045033987474150186"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2b89e9ba0197a9b8b34"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163449000:942 and latest is 1353163449000:942 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 220006 ReplSetTest await synced=true ReplSetTest overflow count : 220007 prev : 210007 ReplSetTest overflow inserting 10000 Sat Nov 17 09:44:11.724 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:11.730 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:44:11.730 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:11.732 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163451000, "i" : 2185 }, "h" : NumberLong("-8591886639215917741"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2bb9e9ba0197a9b93bd"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163451000:2185 and latest is 1353163451000:10000 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 222191 m31200| Sat Nov 17 09:44:11.775 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:44:11.789 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:11.789 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:44:11.789 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:12.369 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:12.369 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:12.369 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:12.369 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:12.370 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:12.370 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:12.718 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:12.718 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:12.718 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:12.718 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:12.718 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:12.718 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:12.980 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:44:12.980 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163451000, "i" : 10000 }, "h" : NumberLong("4118029781442686610"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2bb9e9ba0197a9bb244"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163451000:10000 and latest is 1353163451000:10000 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 230006 ReplSetTest await synced=true ReplSetTest overflow count : 230007 prev : 220007 ReplSetTest overflow inserting 10000 m31101| Sat Nov 17 09:44:14.419 [conn18] end connection 127.0.0.1:57351 (12 connections now open) m31100| Sat Nov 17 09:44:14.420 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:14.420 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:14.420 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:14.420 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:14.420 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:14.420 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:14.423 [initandlisten] connection accepted from 127.0.0.1:57621 #19 (13 connections now open) m30998| Sat Nov 17 09:44:14.264 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:44:14.425 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:14.592 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:44:14.594 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163453000, "i" : 1079 }, "h" : NumberLong("5816980549640270306"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2bd9e9ba0197a9bb67b"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163453000:1079 and latest is 1353163454000:8921 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 231085 m31101| Sat Nov 17 09:44:14.720 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:14.720 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:14.720 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:14.720 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:14.721 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:14.721 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:14.985 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:44:14.985 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m31201| Sat Nov 17 09:44:16.244 [conn19] end connection 127.0.0.1:54124 (13 connections now open) m31201| Sat Nov 17 09:44:16.244 [initandlisten] connection accepted from 127.0.0.1:54394 #21 (14 connections now open) m31100| Sat Nov 17 09:44:16.421 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:16.421 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:16.421 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:16.422 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:16.422 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:16.422 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:16.424 [conn36] end connection 127.0.0.1:59603 (18 connections now open) m31100| Sat Nov 17 09:44:16.425 [initandlisten] connection accepted from 127.0.0.1:59875 #37 (19 connections now open) { "ts" : { "t" : 1353163454000, "i" : 8921 }, "h" : NumberLong("974430354845600099"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2be9e9ba0197a9bd954"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163454000:8921 and latest is 1353163454000:8921 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 240006 ReplSetTest await synced=true ReplSetTest overflow count : 240007 prev : 230007 ReplSetTest overflow inserting 10000 m31101| Sat Nov 17 09:44:16.825 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:17.100 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:17.183 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:17.184 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:16.922 [conn1] insert _overflow.coll keyUpdates:0 locks(micros) w:227854 113ms m31101| Sat Nov 17 09:44:17.187 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:17.187 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:17.187 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:17.187 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:17.345 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:44:17.347 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163456000, "i" : 3418 }, "h" : NumberLong("-5665178045888158724"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2c09e9ba0197a9be6ae"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163456000:3418 and latest is 1353163457000:5075 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 243424 m30998| Sat Nov 17 09:44:17.986 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:17.987 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:44:18.233 [conn33] end connection 127.0.0.1:46475 (21 connections now open) m31200| Sat Nov 17 09:44:18.233 [initandlisten] connection accepted from 127.0.0.1:46750 #35 (22 connections now open) m31100| Sat Nov 17 09:44:18.423 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:18.423 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:18.423 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:18.423 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:18.423 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:18.424 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:19.188 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:19.190 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:19.190 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:19.191 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:19.191 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:19.191 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163457000, "i" : 5075 }, "h" : NumberLong("6535741922899412594"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2c19e9ba0197a9c0064"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163457000:5075 and latest is 1353163457000:5075 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 250006 ReplSetTest await synced=true ReplSetTest overflow count : 250007 prev : 240007 ReplSetTest overflow inserting 10000 m31200| Sat Nov 17 09:44:20.425 [conn34] end connection 127.0.0.1:46491 (21 connections now open) m31200| Sat Nov 17 09:44:20.426 [initandlisten] connection accepted from 127.0.0.1:46771 #36 (22 connections now open) m31100| Sat Nov 17 09:44:20.425 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:20.425 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:20.425 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:20.425 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:20.425 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:20.425 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:20.777 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:44:20.779 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163459000, "i" : 6739 }, "h" : NumberLong("2927764131753209536"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2c39e9ba0197a9c1ab7"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163459000:6739 and latest is 1353163460000:3261 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 256745 m31101| Sat Nov 17 09:44:21.191 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:21.192 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:21.192 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:21.192 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:21.194 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:21.195 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:21.923 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:21.923 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:44:21.980 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:44:21.980 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:21.988 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:44:22.004 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:22.426 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:22.426 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:22.427 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:22.427 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:22.427 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:22.427 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163460000, "i" : 3261 }, "h" : NumberLong("7810419619680578719"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2c39e9ba0197a9c2774"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163460000:3261 and latest is 1353163460000:3261 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 260006 ReplSetTest await synced=true ReplSetTest overflow count : 260007 prev : 250007 ReplSetTest overflow inserting 10000 m31101| Sat Nov 17 09:44:23.284 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:23.284 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:23.285 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:23.285 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:23.285 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:23.285 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:23.606 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:44:23.608 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163463000, "i" : 6515 }, "h" : NumberLong("1255188998261858579"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2c79e9ba0197a9c473f"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163463000:6515 and latest is 1353163463000:8376 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 268145 m30998| Sat Nov 17 09:44:24.428 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:44:24.428 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:24.428 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:24.429 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:24.430 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:24.430 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:24.430 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:24.430 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:25.286 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:25.286 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:25.286 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:25.286 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:25.286 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:25.287 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163463000, "i" : 8376 }, "h" : NumberLong("-7823233774098024761"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2c79e9ba0197a9c4e84"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163463000:8376 and latest is 1353163463000:8376 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 270006 ReplSetTest await synced=true ReplSetTest overflow count : 270007 prev : 260007 ReplSetTest overflow inserting 10000 m31100| Sat Nov 17 09:44:26.452 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:26.453 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:26.453 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:26.453 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:26.453 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:26.453 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:26.553 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:44:26.554 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163466000, "i" : 1757 }, "h" : NumberLong("-4670104020478655665"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2ca9e9ba0197a9c68f6"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163466000:1757 and latest is 1353163466000:4987 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 276776 m30999| Sat Nov 17 09:44:26.990 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:44:27.004 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:27.186 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:27.186 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:27.287 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:27.288 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:27.288 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:27.288 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:27.288 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:27.288 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:28.454 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:28.454 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:28.455 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:28.455 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:28.455 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:28.455 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:28.455 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163466000, "i" : 4987 }, "h" : NumberLong("-1348769371298996306"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2ca9e9ba0197a9c7594"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163466000:4987 and latest is 1353163466000:4987 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 280006 ReplSetTest await synced=true ReplSetTest overflow count : 280007 prev : 270007 ReplSetTest overflow inserting 10000 m31100| Sat Nov 17 09:44:28.852 [conn11] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1353163359000|1 } } cursorid:122887652802144 ntoreturn:0 keyUpdates:0 locks(micros) r:2680 nreturned:4938 reslen:558014 2298ms m31100| Sat Nov 17 09:44:28.936 [conn14] getmore local.oplog.rs query: { ts: { $gte: Timestamp 1353163359000|1 } } cursorid:126153697063020 ntoreturn:0 keyUpdates:0 numYields: 50 locks(micros) r:13320 nreturned:6291 reslen:106967 2211ms m31100| Sat Nov 17 09:44:29.051 [conn1] insert _overflow.coll keyUpdates:0 locks(micros) w:132147 132ms Sat Nov 17 09:44:29.306 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:44:29.307 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163468000, "i" : 4938 }, "h" : NumberLong("3814468108975999415"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2cc9e9ba0197a9c88de"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163468000:4938 and latest is 1353163469000:3709 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 284944 m31101| Sat Nov 17 09:44:29.309 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:29.309 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:29.310 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:29.310 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:29.310 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:29.310 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31202| Sat Nov 17 09:44:30.303 [conn18] end connection 127.0.0.1:38699 (12 connections now open) m31202| Sat Nov 17 09:44:30.303 [initandlisten] connection accepted from 127.0.0.1:38976 #20 (13 connections now open) m31100| Sat Nov 17 09:44:30.456 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:30.456 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:30.456 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:30.456 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:30.457 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:30.457 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163469000, "i" : 3709 }, "h" : NumberLong("7104871349392870683"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2cd9e9ba0197a9c9ca4"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163469000:3709 and latest is 1353163469000:3709 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 290006 ReplSetTest await synced=true ReplSetTest overflow count : 290007 prev : 280007 ReplSetTest overflow inserting 10000 m31101| Sat Nov 17 09:44:31.617 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:31.617 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:31.617 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:31.617 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:31.618 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:31.618 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:31.618 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:31.976 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:31.977 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:44:31.990 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:44:31.991 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:32.061 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:44:32.062 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163471000, "i" : 6106 }, "h" : NumberLong("8243333488485461940"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2cf9e9ba0197a9cb47e"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163471000:6106 and latest is 1353163472000:738 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 296112 m31202| Sat Nov 17 09:44:32.446 [conn19] end connection 127.0.0.1:38721 (12 connections now open) m31202| Sat Nov 17 09:44:32.446 [initandlisten] connection accepted from 127.0.0.1:38998 #21 (13 connections now open) m31100| Sat Nov 17 09:44:32.458 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:32.458 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:32.458 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:32.458 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:32.458 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:32.459 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:32.991 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:44:33.006 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m31201| Sat Nov 17 09:44:33.053 [conn20] end connection 127.0.0.1:54270 (13 connections now open) m31201| Sat Nov 17 09:44:33.053 [initandlisten] connection accepted from 127.0.0.1:54550 #22 (14 connections now open) m31101| Sat Nov 17 09:44:33.619 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:33.619 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:33.619 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:33.619 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:33.619 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:33.619 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163472000, "i" : 738 }, "h" : NumberLong("-7349945156784870805"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2cf9e9ba0197a9cc3b4"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163472000:738 and latest is 1353163472000:738 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 300006 ReplSetTest await synced=true ReplSetTest overflow count : 300007 prev : 290007 ReplSetTest overflow inserting 10000 m30998| Sat Nov 17 09:44:34.500 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:44:34.500 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:34.502 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:34.503 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:34.503 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:34.503 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:34.503 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:34.503 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:35.063 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:44:35.065 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163474000, "i" : 9377 }, "h" : NumberLong("-1748993930789290702"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2d29e9ba0197a9ce855"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163474000:9377 and latest is 1353163475000:484 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 309383 m31101| Sat Nov 17 09:44:35.620 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:35.620 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:35.621 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:35.621 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:35.621 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:35.621 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:36.504 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:36.504 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:36.504 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:36.505 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:36.505 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:36.505 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:37.189 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:37.189 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163475000, "i" : 484 }, "h" : NumberLong("-6752960465130787917"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2d29e9ba0197a9ceac4"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163475000:484 and latest is 1353163475000:484 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 310006 ReplSetTest await synced=true ReplSetTest overflow count : 310007 prev : 300007 ReplSetTest overflow inserting 10000 m31101| Sat Nov 17 09:44:37.715 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:37.717 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:37.718 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:37.718 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:37.719 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:37.719 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:37.843 [conn1] insert _overflow.coll keyUpdates:0 locks(micros) w:251114 125ms Sat Nov 17 09:44:38.040 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:44:38.040 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163477000, "i" : 6427 }, "h" : NumberLong("-5871383050086194801"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2d59e9ba0197a9d03df"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163477000:6427 and latest is 1353163478000:661 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 316433 m31100| Sat Nov 17 09:44:38.506 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:38.506 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:38.506 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:38.506 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:38.506 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:38.507 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:39.721 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:39.721 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:39.721 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:39.721 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:39.721 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:39.721 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:39.992 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:44:40.007 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163478000, "i" : 661 }, "h" : NumberLong("2120452554835154184"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2d59e9ba0197a9d11d4"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163478000:661 and latest is 1353163478000:661 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 317748 ReplSetTest await synced=true ReplSetTest overflow count : 317748 prev : 310007 ReplSetTest overflow inserting 10000 m31100| Sat Nov 17 09:44:40.517 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:40.517 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:40.518 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:40.518 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:40.518 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:40.518 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:40.686 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:44:40.718 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest [object Object] { "ts" : { "t" : 1353163480000, "i" : 1 }, "h" : NumberLong("6448115556301747369"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2d89e9ba0197a9d11d5"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163480000:1 and latest is 1353163480000:10000 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 317748 m31101| Sat Nov 17 09:44:41.722 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:41.722 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:41.723 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31101| Sat Nov 17 09:44:41.723 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:41.723 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31101| Sat Nov 17 09:44:41.723 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:41.979 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:41.980 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:44:41.992 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:44:41.992 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:42.519 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:42.519 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:42.520 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:42.520 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:42.520 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 m31100| Sat Nov 17 09:44:42.520 [rsHealthPoll] couldn't connect to domU-12-31-39-16-30-A2:31102: couldn't connect to server domU-12-31-39-16-30-A2:31102 { "ts" : { "t" : 1353163480000, "i" : 10000 }, "h" : NumberLong("8189763417175973872"), "v" : 2, "op" : "i", "ns" : "_overflow.coll", "o" : { "_id" : ObjectId("50a7a2d89e9ba0197a9d38e4"), "overflow" : "value" } } ReplSetTest await TS for connection to domU-12-31-39-16-30-A2:31101 is 1353163480000:10000 and latest is 1353163480000:10000 ReplSetTest await oplog size for connection to domU-12-31-39-16-30-A2:31101 is 317748 ReplSetTest await synced=true ReplSetTest overflow count : 317748 prev : 317748 ReplSetTest n is : connection to domU-12-31-39-16-30-A2:31102 ReplSetTest n: 2 ports: [ 31100, 31101, 31102 ] 31102 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31102, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "clusteredstale-rs0", "dbpath" : "$set-$node", "remember" : true, "restart" : true, "pathOpts" : { "node" : 2, "set" : "clusteredstale-rs0" } } ReplSetTest (Re)Starting.... Sat Nov 17 09:44:42.759 shell: started program /mnt/data/slaves/Linux_64bit_Legacy/mongo/mongod --oplogSize 40 --port 31102 --noprealloc --smallfiles --rest --replSet clusteredstale-rs0 --dbpath /data/db/clusteredstale-rs0-2 m31102| note: noprealloc may hurt performance in many applications m31102| Sat Nov 17 09:44:42.795 [initandlisten] MongoDB starting : pid=32037 port=31102 dbpath=/data/db/clusteredstale-rs0-2 64-bit host=domU-12-31-39-16-30-A2 m31102| Sat Nov 17 09:44:42.795 [initandlisten] m31102| Sat Nov 17 09:44:42.795 [initandlisten] ** NOTE: This is a development version (2.3.1-pre-) of MongoDB. m31102| Sat Nov 17 09:44:42.795 [initandlisten] ** Not recommended for production. m31102| Sat Nov 17 09:44:42.795 [initandlisten] m31102| Sat Nov 17 09:44:42.795 [initandlisten] db version v2.3.1-pre-, pdfile version 4.5 m31102| Sat Nov 17 09:44:42.795 [initandlisten] git version: 38d64e3ffb528a597fb13d8aa459cd21f0967490 m31102| Sat Nov 17 09:44:42.795 [initandlisten] build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 BOOST_LIB_VERSION=1_49 m31102| Sat Nov 17 09:44:42.795 [initandlisten] options: { dbpath: "/data/db/clusteredstale-rs0-2", noprealloc: true, oplogSize: 40, port: 31102, replSet: "clusteredstale-rs0", rest: true, smallfiles: true } m31102| Sat Nov 17 09:44:42.797 [initandlisten] journal dir=/data/db/clusteredstale-rs0-2/journal m31102| Sat Nov 17 09:44:42.797 [initandlisten] recover : no journal files present, no recovery needed m31102| Sat Nov 17 09:44:42.863 [initandlisten] waiting for connections on port 31102 m31102| Sat Nov 17 09:44:42.864 [websvr] admin web console waiting for connections on port 32102 m31102| Sat Nov 17 09:44:42.883 [rsStart] replSet I am domU-12-31-39-16-30-A2:31102 m31102| Sat Nov 17 09:44:42.883 [rsStart] replSet STARTUP2 [ connection to domU-12-31-39-16-30-A2:31100, connection to domU-12-31-39-16-30-A2:31101, connection to domU-12-31-39-16-30-A2:31102 ] ReplSetTest waitForIndicator health on connection to domU-12-31-39-16-30-A2:31102 [ 1 ] ReplSetTest waitForIndicator from node connection to domU-12-31-39-16-30-A2:31102 m31102| Sat Nov 17 09:44:42.962 [initandlisten] connection accepted from 127.0.0.1:45313 #1 (1 connection now open) ReplSetTest waitForIndicator Initial status ( timeout : 30000 ) : { "set" : "clusteredstale-rs0", "date" : ISODate("2012-11-17T14:44:42Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "domU-12-31-39-16-30-A2:31100", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 143, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "self" : true }, { "_id" : 1, "name" : "domU-12-31-39-16-30-A2:31101", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 133, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "lastHeartbeat" : ISODate("2012-11-17T14:44:42Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:44:42Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "domU-12-31-39-16-30-A2:31102", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1353163383000, "i" : 1 }, "optimeDate" : ISODate("2012-11-17T14:43:03Z"), "lastHeartbeat" : ISODate("2012-11-17T14:44:42Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } Status for : domU-12-31-39-16-30-A2:31100, checking domU-12-31-39-16-30-A2:31102/domU-12-31-39-16-30-A2:31102 Status for : domU-12-31-39-16-30-A2:31101, checking domU-12-31-39-16-30-A2:31102/domU-12-31-39-16-30-A2:31102 Status for : domU-12-31-39-16-30-A2:31102, checking domU-12-31-39-16-30-A2:31102/domU-12-31-39-16-30-A2:31102 Status : 0 target state : 1 m31102| Sat Nov 17 09:44:43.724 [initandlisten] connection accepted from 127.0.0.1:45314 #2 (2 connections now open) m31102| Sat Nov 17 09:44:43.724 [initandlisten] connection accepted from 127.0.0.1:45315 #3 (3 connections now open) m31102| Sat Nov 17 09:44:43.724 [conn2] end connection 127.0.0.1:45314 (2 connections now open) m31101| Sat Nov 17 09:44:43.725 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 thinks that we are down m31101| Sat Nov 17 09:44:43.725 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is up m31101| Sat Nov 17 09:44:43.725 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state STARTUP2 m31102| Sat Nov 17 09:44:43.884 [rsSync] replSet SECONDARY m30998| Sat Nov 17 09:44:44.502 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:44:44.503 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 ok m31102| Sat Nov 17 09:44:44.511 [initandlisten] connection accepted from 127.0.0.1:45316 #4 (3 connections now open) m31102| Sat Nov 17 09:44:44.511 [initandlisten] connection accepted from 127.0.0.1:45317 #5 (4 connections now open) m31101| Sat Nov 17 09:44:44.519 [conn19] end connection 127.0.0.1:57621 (12 connections now open) m31101| Sat Nov 17 09:44:44.519 [initandlisten] connection accepted from 127.0.0.1:57892 #20 (13 connections now open) m31102| Sat Nov 17 09:44:44.521 [initandlisten] connection accepted from 127.0.0.1:45319 #6 (5 connections now open) m31102| Sat Nov 17 09:44:44.521 [conn6] end connection 127.0.0.1:45319 (4 connections now open) m31102| Sat Nov 17 09:44:44.521 [initandlisten] connection accepted from 127.0.0.1:45320 #7 (5 connections now open) m31100| Sat Nov 17 09:44:44.521 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 thinks that we are down m31100| Sat Nov 17 09:44:44.521 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is up m31100| Sat Nov 17 09:44:44.521 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state SECONDARY m31101| Sat Nov 17 09:44:44.884 [initandlisten] connection accepted from 127.0.0.1:57895 #21 (14 connections now open) m31102| Sat Nov 17 09:44:44.885 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is up m31102| Sat Nov 17 09:44:44.885 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is now in state SECONDARY m31100| Sat Nov 17 09:44:44.885 [initandlisten] connection accepted from 127.0.0.1:60124 #38 (20 connections now open) m31102| Sat Nov 17 09:44:44.890 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31100 is up m31102| Sat Nov 17 09:44:44.890 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31100 is now in state PRIMARY ReplSetTest waitForIndicator final status: { "set" : "clusteredstale-rs0", "date" : ISODate("2012-11-17T14:44:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "domU-12-31-39-16-30-A2:31100", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 145, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "self" : true }, { "_id" : 1, "name" : "domU-12-31-39-16-30-A2:31101", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 135, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "lastHeartbeat" : ISODate("2012-11-17T14:44:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "domU-12-31-39-16-30-A2:31102", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1353163384000, "i" : 1 }, "optimeDate" : ISODate("2012-11-17T14:43:04Z"), "lastHeartbeat" : ISODate("2012-11-17T14:44:44Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:44:44Z"), "pingMs" : 0 } ], "ok" : 1 } ReplSetTest waitForIndicator state on connection to domU-12-31-39-16-30-A2:31102 [ 3 ] ReplSetTest waitForIndicator from node connection to domU-12-31-39-16-30-A2:31102 ReplSetTest waitForIndicator Initial status ( timeout : 300000 ) : { "set" : "clusteredstale-rs0", "date" : ISODate("2012-11-17T14:44:44Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "domU-12-31-39-16-30-A2:31100", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 145, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "self" : true }, { "_id" : 1, "name" : "domU-12-31-39-16-30-A2:31101", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 135, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "lastHeartbeat" : ISODate("2012-11-17T14:44:44Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "domU-12-31-39-16-30-A2:31102", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 0, "optime" : { "t" : 1353163384000, "i" : 1 }, "optimeDate" : ISODate("2012-11-17T14:43:04Z"), "lastHeartbeat" : ISODate("2012-11-17T14:44:44Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:44:44Z"), "pingMs" : 0 } ], "ok" : 1 } Status for : domU-12-31-39-16-30-A2:31100, checking domU-12-31-39-16-30-A2:31102/domU-12-31-39-16-30-A2:31102 Status for : domU-12-31-39-16-30-A2:31101, checking domU-12-31-39-16-30-A2:31102/domU-12-31-39-16-30-A2:31102 Status for : domU-12-31-39-16-30-A2:31102, checking domU-12-31-39-16-30-A2:31102/domU-12-31-39-16-30-A2:31102 Status : 2 target state : 3 m31101| Sat Nov 17 09:44:45.726 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state SECONDARY m31100| Sat Nov 17 09:44:46.521 [conn37] end connection 127.0.0.1:59875 (19 connections now open) m31100| Sat Nov 17 09:44:46.521 [initandlisten] connection accepted from 127.0.0.1:60128 #39 (20 connections now open) m31201| Sat Nov 17 09:44:46.520 [conn21] end connection 127.0.0.1:54394 (13 connections now open) m31201| Sat Nov 17 09:44:46.521 [initandlisten] connection accepted from 127.0.0.1:54653 #23 (14 connections now open) m30999| Sat Nov 17 09:44:47.191 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:47.191 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 ok m31102| Sat Nov 17 09:44:47.191 [initandlisten] connection accepted from 127.0.0.1:45327 #8 (6 connections now open) m31102| Sat Nov 17 09:44:47.192 [initandlisten] connection accepted from 127.0.0.1:45328 #9 (7 connections now open) m31200| Sat Nov 17 09:44:48.523 [conn35] end connection 127.0.0.1:46750 (21 connections now open) m31200| Sat Nov 17 09:44:48.523 [initandlisten] connection accepted from 127.0.0.1:46996 #37 (22 connections now open) m31102| Sat Nov 17 09:44:48.908 [rsBackgroundSync] replSet syncing to: domU-12-31-39-16-30-A2:31100 m31100| Sat Nov 17 09:44:48.908 [initandlisten] connection accepted from 127.0.0.1:60135 #40 (21 connections now open) m31100| Sat Nov 17 09:44:48.909 [conn40] end connection 127.0.0.1:60135 (20 connections now open) m31101| Sat Nov 17 09:44:48.909 [initandlisten] connection accepted from 127.0.0.1:57908 #22 (15 connections now open) m31102| Sat Nov 17 09:44:48.909 [rsBackgroundSync] replSet syncing to: domU-12-31-39-16-30-A2:31101 m31101| Sat Nov 17 09:44:48.909 [conn22] end connection 127.0.0.1:57908 (14 connections now open) m31102| Sat Nov 17 09:44:48.909 [rsBackgroundSync] replSet error RS102 too stale to catch up, at least from domU-12-31-39-16-30-A2:31101 m31102| Sat Nov 17 09:44:48.909 [rsBackgroundSync] replSet our last optime : Nov 17 09:43:04 50a7a278:1 m31102| Sat Nov 17 09:44:48.909 [rsBackgroundSync] replSet oldest at domU-12-31-39-16-30-A2:31101 : Nov 17 09:43:09 50a7a27d:8cd m31102| Sat Nov 17 09:44:48.909 [rsBackgroundSync] replSet See http://dochub.mongodb.org/core/resyncingaverystalereplicasetmember m31102| Sat Nov 17 09:44:48.910 [rsBackgroundSync] replSet error RS102 too stale to catch up m31102| Sat Nov 17 09:44:48.910 [rsBackgroundSync] replSet RECOVERING m31101| Sat Nov 17 09:44:49.727 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state RECOVERING m31102| Sat Nov 17 09:44:49.910 [rsSync] replSet still syncing, not yet to minValid optime 50a7a27d:8cd m31100| Sat Nov 17 09:44:50.524 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state RECOVERING m31200| Sat Nov 17 09:44:50.719 [conn36] end connection 127.0.0.1:46771 (21 connections now open) m31200| Sat Nov 17 09:44:50.721 [initandlisten] connection accepted from 127.0.0.1:47000 #38 (22 connections now open) { "set" : "clusteredstale-rs0", "date" : ISODate("2012-11-17T14:44:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "domU-12-31-39-16-30-A2:31100", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 152, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "self" : true }, { "_id" : 1, "name" : "domU-12-31-39-16-30-A2:31101", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 142, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "lastHeartbeat" : ISODate("2012-11-17T14:44:50Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:44:50Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "domU-12-31-39-16-30-A2:31102", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 7, "optime" : { "t" : 1353163384000, "i" : 1 }, "optimeDate" : ISODate("2012-11-17T14:43:04Z"), "lastHeartbeat" : ISODate("2012-11-17T14:44:50Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:44:50Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50a7a27d:8cd" } ], "ok" : 1 } Status for : domU-12-31-39-16-30-A2:31100, checking domU-12-31-39-16-30-A2:31102/domU-12-31-39-16-30-A2:31102 Status for : domU-12-31-39-16-30-A2:31101, checking domU-12-31-39-16-30-A2:31102/domU-12-31-39-16-30-A2:31102 Status for : domU-12-31-39-16-30-A2:31102, checking domU-12-31-39-16-30-A2:31102/domU-12-31-39-16-30-A2:31102 Status : 3 target state : 3 ReplSetTest waitForIndicator final status: { "set" : "clusteredstale-rs0", "date" : ISODate("2012-11-17T14:44:51Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "domU-12-31-39-16-30-A2:31100", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 152, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "self" : true }, { "_id" : 1, "name" : "domU-12-31-39-16-30-A2:31101", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 142, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "lastHeartbeat" : ISODate("2012-11-17T14:44:50Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:44:50Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "domU-12-31-39-16-30-A2:31102", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 7, "optime" : { "t" : 1353163384000, "i" : 1 }, "optimeDate" : ISODate("2012-11-17T14:43:04Z"), "lastHeartbeat" : ISODate("2012-11-17T14:44:50Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:44:50Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still syncing, not yet to minValid optime 50a7a27d:8cd" } ], "ok" : 1 } stale_clustered.js 6: stop non-overflowed secondary ReplSetTest n: 1 ports: [ 31100, 31101, 31102 ] 31101 number ReplSetTest stop *** Shutting down mongod in port 31101 *** m31102| Sat Nov 17 09:44:51.047 [conn3] end connection 127.0.0.1:45315 (6 connections now open) m30999| Sat Nov 17 09:44:51.047 [WriteBackListener-domU-12-31-39-16-30-A2:31101] DBClientCursor::init call() failed m30999| Sat Nov 17 09:44:51.047 [WriteBackListener-domU-12-31-39-16-30-A2:31101] Detecting bad connection created at 0 microSec, clearing pool for domU-12-31-39-16-30-A2:31101 m30999| Sat Nov 17 09:44:51.048 [WriteBackListener-domU-12-31-39-16-30-A2:31101] WriteBackListener exception : DBClientBase::findN: transport error: domU-12-31-39-16-30-A2:31101 ns: admin.$cmd query: { writebacklisten: ObjectId('50a7a270e10b3cbd7ae1b530') } m31100| Sat Nov 17 09:44:51.048 [conn39] end connection 127.0.0.1:60128 (19 connections now open) m31100| Sat Nov 17 09:44:51.048 [conn14] end connection 127.0.0.1:49288 (18 connections now open) m30998| Sat Nov 17 09:44:51.048 [WriteBackListener-domU-12-31-39-16-30-A2:31101] DBClientCursor::init call() failed m30998| Sat Nov 17 09:44:51.050 [WriteBackListener-domU-12-31-39-16-30-A2:31101] Detecting bad connection created at 0 microSec, clearing pool for domU-12-31-39-16-30-A2:31101 m30998| Sat Nov 17 09:44:51.051 [WriteBackListener-domU-12-31-39-16-30-A2:31101] WriteBackListener exception : DBClientBase::findN: transport error: domU-12-31-39-16-30-A2:31101 ns: admin.$cmd query: { writebacklisten: ObjectId('50a7a270e54d65b18b787921') } m31101| Sat Nov 17 09:44:51.036 got signal 15 (Terminated), will terminate after current cmd ends m31101| Sat Nov 17 09:44:51.036 [interruptThread] now exiting m31101| Sat Nov 17 09:44:51.036 dbexit: m31101| Sat Nov 17 09:44:51.036 [interruptThread] shutdown: going to close listening sockets... m31101| Sat Nov 17 09:44:51.036 [interruptThread] closing listening socket: 14 m31101| Sat Nov 17 09:44:51.036 [interruptThread] closing listening socket: 15 m31101| Sat Nov 17 09:44:51.036 [interruptThread] closing listening socket: 16 m31101| Sat Nov 17 09:44:51.036 [interruptThread] removing socket file: /tmp/mongodb-31101.sock m31101| Sat Nov 17 09:44:51.036 [interruptThread] shutdown: going to flush diaglog... m31101| Sat Nov 17 09:44:51.036 [interruptThread] shutdown: going to close sockets... m31101| Sat Nov 17 09:44:51.036 [interruptThread] shutdown: waiting for fs preallocator... m31101| Sat Nov 17 09:44:51.036 [interruptThread] shutdown: lock for final commit... m31101| Sat Nov 17 09:44:51.036 [interruptThread] shutdown: final commit... m31101| Sat Nov 17 09:44:51.048 [conn15] end connection 127.0.0.1:57122 (13 connections now open) m31101| Sat Nov 17 09:44:51.051 [conn17] end connection 127.0.0.1:57160 (13 connections now open) m31101| Sat Nov 17 09:44:51.160 [interruptThread] shutdown: closing all files... m31101| Sat Nov 17 09:44:51.179 [interruptThread] closeAllFiles() finished m31101| Sat Nov 17 09:44:51.179 [interruptThread] journalCleanup... m31101| Sat Nov 17 09:44:51.179 [interruptThread] removeJournalFiles m31101| Sat Nov 17 09:44:51.183 [interruptThread] shutdown: removing fs lock... m31101| Sat Nov 17 09:44:51.184 dbexit: really exiting now Sat Nov 17 09:44:51.982 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Sat Nov 17 09:44:51.982 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:51.984 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 ok Sat Nov 17 09:44:51.984 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Sat Nov 17 09:44:51.984 [ReplicaSetMonitorWatcher] Detecting bad connection created at 0 microSec, clearing pool for domU-12-31-39-16-30-A2:31102 m31102| Sat Nov 17 09:44:51.984 [initandlisten] connection accepted from 127.0.0.1:45340 #10 (7 connections now open) m31200| Sat Nov 17 09:44:51.995 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m31200| Sat Nov 17 09:44:51.995 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:44:51.995 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 ok m31102| Sat Nov 17 09:44:51.995 [initandlisten] connection accepted from 127.0.0.1:45341 #11 (8 connections now open) m31200| Sat Nov 17 09:44:51.995 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m31200| Sat Nov 17 09:44:51.995 [ReplicaSetMonitorWatcher] Detecting bad connection created at 0 microSec, clearing pool for domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:44:52.029 shell: stopped mongo program on port 31101 stale_clustered.js 7: check our regular and slaveOk query m31100| Sat Nov 17 09:44:52.033 [conn31] end connection 127.0.0.1:59331 (17 connections now open) m30998| Sat Nov 17 09:44:52.033 [conn1] DBClientCursor::init lazy say() failed m30998| Sat Nov 17 09:44:52.033 [conn1] DBClientCursor::init message from say() was empty m30998| Sat Nov 17 09:44:52.033 [conn1] slave no longer has secondary status: domU-12-31-39-16-30-A2:31101 m30998| Sat Nov 17 09:44:52.033 [conn1] sharded connection to clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102 not being returned to the pool m31100| Sat Nov 17 09:44:52.033 [initandlisten] connection accepted from 127.0.0.1:60144 #41 (18 connections now open) stale_clustered.js 8: restart both our secondaries clean Sat Nov 17 09:44:52.034 DBClientCursor::init call() failed ReplSetTest Could not call ismaster on node 1 ReplSetTest n: 1 ports: [ 31100, 31101, 31102 ] 31101 number ReplSetTest stop *** Shutting down mongod in port 31101 *** Sat Nov 17 09:44:52.036 No db started on port: 31101 Sat Nov 17 09:44:52.037 shell: stopped mongo program on port 31101 ReplSetTest n: 2 ports: [ 31100, 31101, 31102 ] 31102 number ReplSetTest stop *** Shutting down mongod in port 31102 *** m31102| Sat Nov 17 09:44:52.037 got signal 15 (Terminated), will terminate after current cmd ends m31102| Sat Nov 17 09:44:52.037 [interruptThread] now exiting m31102| Sat Nov 17 09:44:52.037 dbexit: m31102| Sat Nov 17 09:44:52.037 [interruptThread] shutdown: going to close listening sockets... m31102| Sat Nov 17 09:44:52.037 [interruptThread] closing listening socket: 60 m31102| Sat Nov 17 09:44:52.037 [interruptThread] closing listening socket: 61 m31102| Sat Nov 17 09:44:52.037 [interruptThread] closing listening socket: 62 m31102| Sat Nov 17 09:44:52.037 [interruptThread] removing socket file: /tmp/mongodb-31102.sock m31102| Sat Nov 17 09:44:52.038 [interruptThread] shutdown: going to flush diaglog... m31102| Sat Nov 17 09:44:52.038 [interruptThread] shutdown: going to close sockets... m31102| Sat Nov 17 09:44:52.038 [interruptThread] shutdown: waiting for fs preallocator... m31102| Sat Nov 17 09:44:52.038 [interruptThread] shutdown: lock for final commit... m31102| Sat Nov 17 09:44:52.038 [interruptThread] shutdown: final commit... m31100| Sat Nov 17 09:44:52.046 [conn38] end connection 127.0.0.1:60124 (17 connections now open) m30999| Sat Nov 17 09:44:52.045 [WriteBackListener-domU-12-31-39-16-30-A2:31102] DBClientCursor::init call() failed m30999| Sat Nov 17 09:44:52.046 [WriteBackListener-domU-12-31-39-16-30-A2:31102] Detecting bad connection created at 1353163376417793 microSec, clearing pool for domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:52.046 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : DBClientBase::findN: transport error: domU-12-31-39-16-30-A2:31102 ns: admin.$cmd query: { writebacklisten: ObjectId('50a7a270e10b3cbd7ae1b530') } m30998| Sat Nov 17 09:44:52.045 [WriteBackListener-domU-12-31-39-16-30-A2:31102] DBClientCursor::init call() failed m30998| Sat Nov 17 09:44:52.046 [WriteBackListener-domU-12-31-39-16-30-A2:31102] Detecting bad connection created at 1353163383889078 microSec, clearing pool for domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:44:52.046 [WriteBackListener-domU-12-31-39-16-30-A2:31102] WriteBackListener exception : DBClientBase::findN: transport error: domU-12-31-39-16-30-A2:31102 ns: admin.$cmd query: { writebacklisten: ObjectId('50a7a270e54d65b18b787921') } m31102| Sat Nov 17 09:44:52.046 [interruptThread] shutdown: closing all files... m31102| Sat Nov 17 09:44:52.047 [interruptThread] closeAllFiles() finished m31102| Sat Nov 17 09:44:52.047 [interruptThread] journalCleanup... m31102| Sat Nov 17 09:44:52.047 [interruptThread] removeJournalFiles m31102| Sat Nov 17 09:44:52.048 [interruptThread] shutdown: removing fs lock... m31102| Sat Nov 17 09:44:52.048 dbexit: really exiting now m30999| Sat Nov 17 09:44:52.049 [WriteBackListener-domU-12-31-39-16-30-A2:31101] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31101 m30998| Sat Nov 17 09:44:52.052 [WriteBackListener-domU-12-31-39-16-30-A2:31101] WriteBackListener exception : socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31101 m31100| Sat Nov 17 09:44:52.524 [rsHealthPoll] DBClientCursor::init call() failed m31100| Sat Nov 17 09:44:52.524 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31101 heartbeat failed, retrying m31100| Sat Nov 17 09:44:52.524 [rsHealthPoll] replSet info domU-12-31-39-16-30-A2:31101 is down (or slow to respond): m31100| Sat Nov 17 09:44:52.524 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is now in state DOWN m31100| Sat Nov 17 09:44:52.525 [rsHealthPoll] DBClientCursor::init call() failed m31100| Sat Nov 17 09:44:52.525 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying m31100| Sat Nov 17 09:44:52.525 [rsHealthPoll] replSet info domU-12-31-39-16-30-A2:31102 is down (or slow to respond): m31100| Sat Nov 17 09:44:52.525 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state DOWN m31100| Sat Nov 17 09:44:52.525 [rsMgr] can't see a majority of the set, relinquishing primary m31100| Sat Nov 17 09:44:52.525 [rsMgr] replSet relinquishing primary state m31100| Sat Nov 17 09:44:52.525 [rsMgr] replSet SECONDARY m31100| Sat Nov 17 09:44:52.525 [rsMgr] replSet closing client sockets after relinquishing primary m31202| Sat Nov 17 09:44:52.526 [conn11] end connection 127.0.0.1:38291 (12 connections now open) m31202| Sat Nov 17 09:44:52.526 [conn10] end connection 127.0.0.1:38288 (11 connections now open) m31100| Sat Nov 17 09:44:52.526 [conn33] end connection 127.0.0.1:59349 (16 connections now open) m29000| Sat Nov 17 09:44:52.525 [conn12] end connection 127.0.0.1:46337 (14 connections now open) m29000| Sat Nov 17 09:44:52.525 [conn14] end connection 127.0.0.1:46339 (13 connections now open) m29000| Sat Nov 17 09:44:52.526 [conn13] end connection 127.0.0.1:46338 (12 connections now open) m29000| Sat Nov 17 09:44:52.526 [conn16] end connection 127.0.0.1:46365 (11 connections now open) m31100| Sat Nov 17 09:44:52.527 [conn34] end connection 127.0.0.1:59387 (15 connections now open) m30999| Sat Nov 17 09:44:52.525 [WriteBackListener-domU-12-31-39-16-30-A2:31100] DBClientCursor::init call() failed m30999| Sat Nov 17 09:44:52.526 [WriteBackListener-domU-12-31-39-16-30-A2:31100] Detecting bad connection created at 0 microSec, clearing pool for domU-12-31-39-16-30-A2:31100 m30999| Sat Nov 17 09:44:52.526 [WriteBackListener-domU-12-31-39-16-30-A2:31100] WriteBackListener exception : DBClientBase::findN: transport error: domU-12-31-39-16-30-A2:31100 ns: admin.$cmd query: { writebacklisten: ObjectId('50a7a270e10b3cbd7ae1b530') } m31201| Sat Nov 17 09:44:52.526 [conn9] end connection 127.0.0.1:53827 (13 connections now open) m31201| Sat Nov 17 09:44:52.526 [conn10] end connection 127.0.0.1:53830 (12 connections now open) m30998| Sat Nov 17 09:44:52.526 [WriteBackListener-domU-12-31-39-16-30-A2:31100] DBClientCursor::init call() failed m30998| Sat Nov 17 09:44:52.527 [WriteBackListener-domU-12-31-39-16-30-A2:31100] Detecting bad connection created at 0 microSec, clearing pool for domU-12-31-39-16-30-A2:31100 m30998| Sat Nov 17 09:44:52.527 [WriteBackListener-domU-12-31-39-16-30-A2:31100] WriteBackListener exception : DBClientBase::findN: transport error: domU-12-31-39-16-30-A2:31100 ns: admin.$cmd query: { writebacklisten: ObjectId('50a7a270e54d65b18b787921') } m31200| Sat Nov 17 09:44:52.526 [conn21] end connection 127.0.0.1:46162 (21 connections now open) m31200| Sat Nov 17 09:44:52.526 [conn22] end connection 127.0.0.1:46165 (20 connections now open) m31200| Sat Nov 17 09:44:52.526 [conn24] end connection 127.0.0.1:46182 (19 connections now open) m31200| Sat Nov 17 09:44:52.526 [conn23] end connection 127.0.0.1:46168 (18 connections now open) Sat Nov 17 09:44:53.037 shell: stopped mongo program on port 31102 ReplSetTest n is : connection to domU-12-31-39-16-30-A2:31101 ReplSetTest n: 1 ports: [ 31100, 31101, 31102 ] 31101 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31101, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "clusteredstale-rs0", "dbpath" : "$set-$node", "remember" : true, "startClean" : true, "restart" : true, "pathOpts" : { "node" : 1, "set" : "clusteredstale-rs0" } } ReplSetTest (Re)Starting.... Resetting db path '/data/db/clusteredstale-rs0-1' Sat Nov 17 09:44:53.133 shell: started program /mnt/data/slaves/Linux_64bit_Legacy/mongo/mongod --oplogSize 40 --port 31101 --noprealloc --smallfiles --rest --replSet clusteredstale-rs0 --dbpath /data/db/clusteredstale-rs0-1 m31101| note: noprealloc may hurt performance in many applications m31101| Sat Nov 17 09:44:53.180 [initandlisten] MongoDB starting : pid=32139 port=31101 dbpath=/data/db/clusteredstale-rs0-1 64-bit host=domU-12-31-39-16-30-A2 m31101| Sat Nov 17 09:44:53.180 [initandlisten] m31101| Sat Nov 17 09:44:53.180 [initandlisten] ** NOTE: This is a development version (2.3.1-pre-) of MongoDB. m31101| Sat Nov 17 09:44:53.180 [initandlisten] ** Not recommended for production. m31101| Sat Nov 17 09:44:53.180 [initandlisten] m31101| Sat Nov 17 09:44:53.180 [initandlisten] db version v2.3.1-pre-, pdfile version 4.5 m31101| Sat Nov 17 09:44:53.180 [initandlisten] git version: 38d64e3ffb528a597fb13d8aa459cd21f0967490 m31101| Sat Nov 17 09:44:53.180 [initandlisten] build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 BOOST_LIB_VERSION=1_49 m31101| Sat Nov 17 09:44:53.180 [initandlisten] options: { dbpath: "/data/db/clusteredstale-rs0-1", noprealloc: true, oplogSize: 40, port: 31101, replSet: "clusteredstale-rs0", rest: true, smallfiles: true } m31101| Sat Nov 17 09:44:53.182 [initandlisten] journal dir=/data/db/clusteredstale-rs0-1/journal m31101| Sat Nov 17 09:44:53.182 [initandlisten] recover : no journal files present, no recovery needed m31101| Sat Nov 17 09:44:53.236 [initandlisten] waiting for connections on port 31101 m31101| Sat Nov 17 09:44:53.236 [websvr] admin web console waiting for connections on port 32101 m31101| Sat Nov 17 09:44:53.239 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31101| Sat Nov 17 09:44:53.239 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done [ connection to domU-12-31-39-16-30-A2:31100, connection to domU-12-31-39-16-30-A2:31101, connection to domU-12-31-39-16-30-A2:31102 ] ReplSetTest waitForIndicator health on connection to domU-12-31-39-16-30-A2:31101 [ 1 ] ReplSetTest waitForIndicator from node connection to domU-12-31-39-16-30-A2:31101 Sat Nov 17 09:44:53.336 Socket recv() errno:104 Connection reset by peer 127.0.0.1:31100 Sat Nov 17 09:44:53.336 SocketException: remote: 127.0.0.1:31100 error: 9001 socket exception [1] server [127.0.0.1:31100] Sat Nov 17 09:44:53.336 DBClientCursor::init call() failed ReplSetTest Could not call ismaster on node 0 Sat Nov 17 09:44:53.337 DBClientCursor::init call() failed ReplSetTest Could not call ismaster on node 2 ReplSetTest waitForIndicator Initial status ( timeout : 300000 ) : { "startupStatus" : 3, "info" : "run rs.initiate(...) if not yet done for the set", "errmsg" : "can't get local.system.replset config from self or any seed (EMPTYCONFIG)", "ok" : 0 } m31100| Sat Nov 17 09:44:53.336 [conn1] end connection 127.0.0.1:49214 (14 connections now open) m31101| Sat Nov 17 09:44:53.336 [initandlisten] connection accepted from 127.0.0.1:57924 #1 (1 connection now open) m31100| Sat Nov 17 09:44:53.528 [initandlisten] connection accepted from 127.0.0.1:60153 #42 (15 connections now open) m31100| Sat Nov 17 09:44:53.528 [initandlisten] connection accepted from 127.0.0.1:60154 #43 (16 connections now open) m31101| Sat Nov 17 09:44:54.052 [initandlisten] connection accepted from 127.0.0.1:57927 #2 (2 connections now open) m31101| Sat Nov 17 09:44:54.053 [initandlisten] connection accepted from 127.0.0.1:57928 #3 (3 connections now open) m31100| Sat Nov 17 09:44:54.513 [conn29] end connection 127.0.0.1:59325 (15 connections now open) m31100| Sat Nov 17 09:44:54.514 [initandlisten] connection accepted from 127.0.0.1:60158 #44 (16 connections now open) m31100| Sat Nov 17 09:44:54.514 [initandlisten] connection accepted from 127.0.0.1:60159 #45 (17 connections now open) m30998| Sat Nov 17 09:44:54.513 [ReplicaSetMonitorWatcher] Socket recv() errno:104 Connection reset by peer 127.0.0.1:31100 m30998| Sat Nov 17 09:44:54.513 [ReplicaSetMonitorWatcher] SocketException: remote: 127.0.0.1:31100 error: 9001 socket exception [1] server [127.0.0.1:31100] m30998| Sat Nov 17 09:44:54.513 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m30998| Sat Nov 17 09:44:54.513 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31100 m30998| Sat Nov 17 09:44:54.514 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31100 ok m30998| Sat Nov 17 09:44:54.514 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m30998| Sat Nov 17 09:44:54.514 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m31101| Sat Nov 17 09:44:54.525 [initandlisten] connection accepted from 127.0.0.1:57933 #4 (4 connections now open) m31100| Sat Nov 17 09:44:54.526 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is up m31100| Sat Nov 17 09:44:54.526 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes m31100| Sat Nov 17 09:44:54.526 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 heartbeat failed, retrying Sat Nov 17 09:44:55.338 trying reconnect to 127.0.0.1:31100 Sat Nov 17 09:44:55.339 reconnect 127.0.0.1:31100 ok m31100| Sat Nov 17 09:44:55.339 [initandlisten] connection accepted from 127.0.0.1:60165 #46 (18 connections now open) Sat Nov 17 09:44:55.340 trying reconnect to 127.0.0.1:31102 Sat Nov 17 09:44:55.341 reconnect 127.0.0.1:31102 failed couldn't connect to server 127.0.0.1:31102 ReplSetTest Could not call ismaster on node 2 ReplSetTest waitForIndicator final status: { "set" : "clusteredstale-rs0", "date" : ISODate("2012-11-17T14:44:55Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "domU-12-31-39-16-30-A2:31100", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 156, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "errmsg" : "not electing self, not all members up and we have been up less than 5 minutes", "self" : true }, { "_id" : 1, "name" : "domU-12-31-39-16-30-A2:31101", "health" : 1, "state" : 8, "stateStr" : "DOWN", "uptime" : 1, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "lastHeartbeat" : ISODate("2012-11-17T14:44:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still initializing" }, { "_id" : 2, "name" : "domU-12-31-39-16-30-A2:31102", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1353163384000, "i" : 1 }, "optimeDate" : ISODate("2012-11-17T14:43:04Z"), "lastHeartbeat" : ISODate("2012-11-17T14:44:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } ReplSetTest n is : connection to domU-12-31-39-16-30-A2:31102 ReplSetTest n: 2 ports: [ 31100, 31101, 31102 ] 31102 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31102, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "clusteredstale-rs0", "dbpath" : "$set-$node", "remember" : true, "startClean" : true, "restart" : true, "pathOpts" : { "node" : 2, "set" : "clusteredstale-rs0" } } ReplSetTest (Re)Starting.... Resetting db path '/data/db/clusteredstale-rs0-2' Sat Nov 17 09:44:55.418 shell: started program /mnt/data/slaves/Linux_64bit_Legacy/mongo/mongod --oplogSize 40 --port 31102 --noprealloc --smallfiles --rest --replSet clusteredstale-rs0 --dbpath /data/db/clusteredstale-rs0-2 m31102| note: noprealloc may hurt performance in many applications m31102| Sat Nov 17 09:44:55.487 [initandlisten] MongoDB starting : pid=32207 port=31102 dbpath=/data/db/clusteredstale-rs0-2 64-bit host=domU-12-31-39-16-30-A2 m31102| Sat Nov 17 09:44:55.487 [initandlisten] m31102| Sat Nov 17 09:44:55.487 [initandlisten] ** NOTE: This is a development version (2.3.1-pre-) of MongoDB. m31102| Sat Nov 17 09:44:55.487 [initandlisten] ** Not recommended for production. m31102| Sat Nov 17 09:44:55.487 [initandlisten] m31102| Sat Nov 17 09:44:55.487 [initandlisten] db version v2.3.1-pre-, pdfile version 4.5 m31102| Sat Nov 17 09:44:55.487 [initandlisten] git version: 38d64e3ffb528a597fb13d8aa459cd21f0967490 m31102| Sat Nov 17 09:44:55.487 [initandlisten] build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 BOOST_LIB_VERSION=1_49 m31102| Sat Nov 17 09:44:55.487 [initandlisten] options: { dbpath: "/data/db/clusteredstale-rs0-2", noprealloc: true, oplogSize: 40, port: 31102, replSet: "clusteredstale-rs0", rest: true, smallfiles: true } m31102| Sat Nov 17 09:44:55.493 [initandlisten] journal dir=/data/db/clusteredstale-rs0-2/journal m31102| Sat Nov 17 09:44:55.493 [initandlisten] recover : no journal files present, no recovery needed m31101| Sat Nov 17 09:44:55.516 [initandlisten] connection accepted from 127.0.0.1:57941 #5 (5 connections now open) m30998| Sat Nov 17 09:44:55.515 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: domU-12-31-39-16-30-A2:31100 { setName: "clusteredstale-rs0", ismaster: false, secondary: true, hosts: [ "domU-12-31-39-16-30-A2:31100", "domU-12-31-39-16-30-A2:31102", "domU-12-31-39-16-30-A2:31101" ], me: "domU-12-31-39-16-30-A2:31100", maxBsonObjectSize: 16777216, localTime: new Date(1353163495515), ok: 1.0 } m30998| Sat Nov 17 09:44:55.515 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31101 m30998| Sat Nov 17 09:44:55.516 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31101 ok m30998| Sat Nov 17 09:44:55.516 [ReplicaSetMonitorWatcher] node: domU-12-31-39-16-30-A2:31101 isn't a part of set: clusteredstale-rs0 ismaster: { ismaster: false, secondary: false, info: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1353163495516), ok: 1.0 } m30998| Sat Nov 17 09:44:55.516 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:44:55.516 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 failed couldn't connect to server domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:44:55.516 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception domU-12-31-39-16-30-A2:31102 socket exception [CONNECT_ERROR] for domU-12-31-39-16-30-A2:31102 m31102| Sat Nov 17 09:44:55.552 [initandlisten] waiting for connections on port 31102 m31102| Sat Nov 17 09:44:55.552 [websvr] admin web console waiting for connections on port 32102 m31102| Sat Nov 17 09:44:55.555 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31102| Sat Nov 17 09:44:55.555 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done [ connection to domU-12-31-39-16-30-A2:31100, connection to domU-12-31-39-16-30-A2:31101, connection to domU-12-31-39-16-30-A2:31102 ] ReplSetTest waitForIndicator health on connection to domU-12-31-39-16-30-A2:31102 [ 1 ] ReplSetTest waitForIndicator from node connection to domU-12-31-39-16-30-A2:31102 m31102| Sat Nov 17 09:44:55.621 [initandlisten] connection accepted from 127.0.0.1:45369 #1 (1 connection now open) ReplSetTest waitForIndicator Initial status ( timeout : 300000 ) : { "set" : "clusteredstale-rs0", "date" : ISODate("2012-11-17T14:44:55Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "domU-12-31-39-16-30-A2:31100", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 156, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "errmsg" : "not electing self, not all members up and we have been up less than 5 minutes", "self" : true }, { "_id" : 1, "name" : "domU-12-31-39-16-30-A2:31101", "health" : 1, "state" : 8, "stateStr" : "DOWN", "uptime" : 1, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "lastHeartbeat" : ISODate("2012-11-17T14:44:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still initializing" }, { "_id" : 2, "name" : "domU-12-31-39-16-30-A2:31102", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : { "t" : 1353163384000, "i" : 1 }, "optimeDate" : ISODate("2012-11-17T14:43:04Z"), "lastHeartbeat" : ISODate("2012-11-17T14:44:54Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0 } ], "ok" : 1 } Status for : domU-12-31-39-16-30-A2:31100, checking domU-12-31-39-16-30-A2:31102/domU-12-31-39-16-30-A2:31102 Status for : domU-12-31-39-16-30-A2:31101, checking domU-12-31-39-16-30-A2:31102/domU-12-31-39-16-30-A2:31102 Status for : domU-12-31-39-16-30-A2:31102, checking domU-12-31-39-16-30-A2:31102/domU-12-31-39-16-30-A2:31102 Status : 0 target state : 1 m31100| Sat Nov 17 09:44:55.831 [conn11] SocketException handling request, closing client connection: 9001 socket exception [2] server [127.0.0.1:49285] m30998| Sat Nov 17 09:44:56.517 [ReplicaSetMonitorWatcher] No primary detected for set clusteredstale-rs0 m31102| Sat Nov 17 09:44:56.527 [initandlisten] connection accepted from 127.0.0.1:45370 #2 (2 connections now open) m31100| Sat Nov 17 09:44:56.527 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is up m31100| Sat Nov 17 09:44:56.528 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes m31100| Sat Nov 17 09:44:57.195 [conn20] end connection 127.0.0.1:49337 (16 connections now open) m31100| Sat Nov 17 09:44:57.195 [initandlisten] connection accepted from 127.0.0.1:60173 #47 (17 connections now open) m30999| Sat Nov 17 09:44:57.195 [ReplicaSetMonitorWatcher] Socket recv() errno:104 Connection reset by peer 127.0.0.1:31100 m30999| Sat Nov 17 09:44:57.195 [ReplicaSetMonitorWatcher] SocketException: remote: 127.0.0.1:31100 error: 9001 socket exception [1] server [127.0.0.1:31100] m30999| Sat Nov 17 09:44:57.195 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m30999| Sat Nov 17 09:44:57.195 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31100 m30999| Sat Nov 17 09:44:57.195 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31100 ok m31100| Sat Nov 17 09:44:57.196 [initandlisten] connection accepted from 127.0.0.1:60174 #48 (18 connections now open) m30999| Sat Nov 17 09:44:57.196 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m30999| Sat Nov 17 09:44:57.196 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed ReplSetTest waitForIndicator final status: { "set" : "clusteredstale-rs0", "date" : ISODate("2012-11-17T14:44:57Z"), "myState" : 2, "members" : [ { "_id" : 0, "name" : "domU-12-31-39-16-30-A2:31100", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 158, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "errmsg" : "not electing self, not all members up and we have been up less than 5 minutes", "self" : true }, { "_id" : 1, "name" : "domU-12-31-39-16-30-A2:31101", "health" : 1, "state" : 8, "stateStr" : "DOWN", "uptime" : 3, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "lastHeartbeat" : ISODate("2012-11-17T14:44:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still initializing" }, { "_id" : 2, "name" : "domU-12-31-39-16-30-A2:31102", "health" : 1, "state" : 8, "stateStr" : "DOWN", "uptime" : 1, "optime" : { "t" : 1353163384000, "i" : 1 }, "optimeDate" : ISODate("2012-11-17T14:43:04Z"), "lastHeartbeat" : ISODate("2012-11-17T14:44:56Z"), "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"), "pingMs" : 0, "lastHeartbeatMessage" : "still initializing" } ], "ok" : 1 } stale_clustered.js 9: wait for recovery m30999| Sat Nov 17 09:44:58.197 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: domU-12-31-39-16-30-A2:31100 { setName: "clusteredstale-rs0", ismaster: false, secondary: true, hosts: [ "domU-12-31-39-16-30-A2:31100", "domU-12-31-39-16-30-A2:31102", "domU-12-31-39-16-30-A2:31101" ], me: "domU-12-31-39-16-30-A2:31100", maxBsonObjectSize: 16777216, localTime: new Date(1353163498197), ok: 1.0 } m30999| Sat Nov 17 09:44:58.197 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31101 m30999| Sat Nov 17 09:44:58.197 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31101 ok m30999| Sat Nov 17 09:44:58.197 [ReplicaSetMonitorWatcher] node: domU-12-31-39-16-30-A2:31101 isn't a part of set: clusteredstale-rs0 ismaster: { ismaster: false, secondary: false, info: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1353163498197), ok: 1.0 } m30999| Sat Nov 17 09:44:58.197 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30999| Sat Nov 17 09:44:58.198 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 ok m31102| Sat Nov 17 09:44:58.198 [initandlisten] connection accepted from 127.0.0.1:45376 #3 (3 connections now open) m30999| Sat Nov 17 09:44:58.198 [ReplicaSetMonitorWatcher] node: domU-12-31-39-16-30-A2:31102 isn't a part of set: clusteredstale-rs0 ismaster: { ismaster: false, secondary: false, info: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1353163498198), ok: 1.0 } m31101| Sat Nov 17 09:44:58.197 [initandlisten] connection accepted from 127.0.0.1:57949 #6 (6 connections now open) m31102| Sat Nov 17 09:44:58.528 [conn2] end connection 127.0.0.1:45370 (2 connections now open) m31102| Sat Nov 17 09:44:58.528 [initandlisten] connection accepted from 127.0.0.1:45378 #4 (3 connections now open) m30999| Sat Nov 17 09:44:59.198 [ReplicaSetMonitorWatcher] No primary detected for set clusteredstale-rs0 m31202| Sat Nov 17 09:45:00.526 [conn20] end connection 127.0.0.1:38976 (10 connections now open) m31202| Sat Nov 17 09:45:00.527 [initandlisten] connection accepted from 127.0.0.1:39170 #22 (11 connections now open) m31102| Sat Nov 17 09:45:01.046 [initandlisten] connection accepted from 127.0.0.1:45383 #5 (4 connections now open) m31102| Sat Nov 17 09:45:01.046 [initandlisten] connection accepted from 127.0.0.1:45384 #6 (5 connections now open) m31100| Sat Nov 17 09:45:01.810 [LockPinger] Socket say send() errno:9 Bad file descriptor 127.0.0.1:29000 m31100| Sat Nov 17 09:45:01.810 [LockPinger] Detecting bad connection created at 0 microSec, clearing pool for domU-12-31-39-16-30-A2:29000 m31100| Sat Nov 17 09:45:01.810 [LockPinger] distributed lock pinger 'domU-12-31-39-16-30-A2:29000/domU-12-31-39-16-30-A2:31100:1353163381:745512520' detected an exception while pinging. :: caused by :: socket exception [SEND_ERROR] for 127.0.0.1:29000 m31100| Sat Nov 17 09:45:01.983 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 127.0.0.1:31200 m31100| Sat Nov 17 09:45:01.983 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31200 m31100| Sat Nov 17 09:45:01.984 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31200 ok m31100| Sat Nov 17 09:45:01.984 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 127.0.0.1:31200 m31100| Sat Nov 17 09:45:01.984 [ReplicaSetMonitorWatcher] Detecting bad connection created at 0 microSec, clearing pool for domU-12-31-39-16-30-A2:31200 Sat Nov 17 09:45:01.985 [ReplicaSetMonitorWatcher] Socket recv() errno:104 Connection reset by peer 127.0.0.1:31100 Sat Nov 17 09:45:01.986 [ReplicaSetMonitorWatcher] SocketException: remote: 127.0.0.1:31100 error: 9001 socket exception [1] server [127.0.0.1:31100] Sat Nov 17 09:45:01.986 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Sat Nov 17 09:45:01.986 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31100 Sat Nov 17 09:45:01.986 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31100 ok Sat Nov 17 09:45:01.986 [ReplicaSetMonitorWatcher] Socket recv() errno:104 Connection reset by peer 127.0.0.1:31100 Sat Nov 17 09:45:01.986 [ReplicaSetMonitorWatcher] SocketException: remote: 127.0.0.1:31100 error: 9001 socket exception [1] server [127.0.0.1:31100] Sat Nov 17 09:45:01.986 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Sat Nov 17 09:45:01.987 [ReplicaSetMonitorWatcher] Detecting bad connection created at 0 microSec, clearing pool for domU-12-31-39-16-30-A2:31100 Sat Nov 17 09:45:01.987 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31101 Sat Nov 17 09:45:01.987 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31101 ok Sat Nov 17 09:45:01.987 [ReplicaSetMonitorWatcher] node: domU-12-31-39-16-30-A2:31101 isn't a part of set: clusteredstale-rs0 ismaster: { ismaster: false, secondary: false, info: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1353163501987), ok: 1.0 } Sat Nov 17 09:45:01.987 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m31100| Sat Nov 17 09:45:01.984 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 127.0.0.1:31201 m31100| Sat Nov 17 09:45:01.984 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 127.0.0.1:31202 m31100| Sat Nov 17 09:45:01.985 [conn17] end connection 127.0.0.1:49298 (17 connections now open) m31200| Sat Nov 17 09:45:01.983 [initandlisten] connection accepted from 127.0.0.1:47053 #39 (19 connections now open) m31200| Sat Nov 17 09:45:01.984 [initandlisten] connection accepted from 127.0.0.1:47054 #40 (20 connections now open) m31101| Sat Nov 17 09:45:01.987 [initandlisten] connection accepted from 127.0.0.1:57966 #7 (7 connections now open) m31100| Sat Nov 17 09:45:01.986 [initandlisten] connection accepted from 127.0.0.1:60193 #49 (18 connections now open) m31100| Sat Nov 17 09:45:01.986 [conn18] end connection 127.0.0.1:49301 (17 connections now open) m31100| Sat Nov 17 09:45:01.996 [conn25] end connection 127.0.0.1:59308 (16 connections now open) m31100| Sat Nov 17 09:45:02.004 [initandlisten] connection accepted from 127.0.0.1:60195 #50 (17 connections now open) m31100| Sat Nov 17 09:45:02.005 [conn26] end connection 127.0.0.1:59311 (16 connections now open) m31101| Sat Nov 17 09:45:02.005 [initandlisten] connection accepted from 127.0.0.1:57968 #8 (8 connections now open) m31200| Sat Nov 17 09:45:01.996 [ReplicaSetMonitorWatcher] Socket recv() errno:104 Connection reset by peer 127.0.0.1:31100 m31200| Sat Nov 17 09:45:01.996 [ReplicaSetMonitorWatcher] SocketException: remote: 127.0.0.1:31100 error: 9001 socket exception [1] server [127.0.0.1:31100] m31200| Sat Nov 17 09:45:01.996 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m31200| Sat Nov 17 09:45:01.996 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31100 m31200| Sat Nov 17 09:45:01.996 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31100 ok m31200| Sat Nov 17 09:45:02.005 [ReplicaSetMonitorWatcher] Socket recv() errno:104 Connection reset by peer 127.0.0.1:31100 m31200| Sat Nov 17 09:45:02.005 [ReplicaSetMonitorWatcher] SocketException: remote: 127.0.0.1:31100 error: 9001 socket exception [1] server [127.0.0.1:31100] m31200| Sat Nov 17 09:45:02.005 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m31200| Sat Nov 17 09:45:02.005 [ReplicaSetMonitorWatcher] Detecting bad connection created at 0 microSec, clearing pool for domU-12-31-39-16-30-A2:31100 m31200| Sat Nov 17 09:45:02.005 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31101 m31200| Sat Nov 17 09:45:02.005 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31101 ok m31200| Sat Nov 17 09:45:02.007 [ReplicaSetMonitorWatcher] node: domU-12-31-39-16-30-A2:31101 isn't a part of set: clusteredstale-rs0 ismaster: { ismaster: false, secondary: false, info: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1353163502007), ok: 1.0 } m31200| Sat Nov 17 09:45:02.007 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m31100| Sat Nov 17 09:45:02.528 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes m31202| Sat Nov 17 09:45:02.530 [conn21] end connection 127.0.0.1:38998 (10 connections now open) m31202| Sat Nov 17 09:45:02.530 [initandlisten] connection accepted from 127.0.0.1:39183 #23 (11 connections now open) Sat Nov 17 09:45:02.988 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: domU-12-31-39-16-30-A2:31100 { setName: "clusteredstale-rs0", ismaster: false, secondary: true, hosts: [ "domU-12-31-39-16-30-A2:31100", "domU-12-31-39-16-30-A2:31102", "domU-12-31-39-16-30-A2:31101" ], me: "domU-12-31-39-16-30-A2:31100", maxBsonObjectSize: 16777216, localTime: new Date(1353163502988), ok: 1.0 } m31100| Sat Nov 17 09:45:02.990 [initandlisten] connection accepted from 127.0.0.1:60199 #51 (17 connections now open) Sat Nov 17 09:45:02.990 [ReplicaSetMonitorWatcher] node: domU-12-31-39-16-30-A2:31101 isn't a part of set: clusteredstale-rs0 ismaster: { ismaster: false, secondary: false, info: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1353163502990), ok: 1.0 } Sat Nov 17 09:45:02.990 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:45:02.991 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 ok Sat Nov 17 09:45:02.992 [ReplicaSetMonitorWatcher] node: domU-12-31-39-16-30-A2:31102 isn't a part of set: clusteredstale-rs0 ismaster: { ismaster: false, secondary: false, info: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1353163502992), ok: 1.0 } m31102| Sat Nov 17 09:45:02.991 [initandlisten] connection accepted from 127.0.0.1:45398 #7 (6 connections now open) m31200| Sat Nov 17 09:45:03.008 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: domU-12-31-39-16-30-A2:31100 { setName: "clusteredstale-rs0", ismaster: false, secondary: true, hosts: [ "domU-12-31-39-16-30-A2:31100", "domU-12-31-39-16-30-A2:31102", "domU-12-31-39-16-30-A2:31101" ], me: "domU-12-31-39-16-30-A2:31100", maxBsonObjectSize: 16777216, localTime: new Date(1353163503008), ok: 1.0 } m31100| Sat Nov 17 09:45:03.008 [initandlisten] connection accepted from 127.0.0.1:60201 #52 (18 connections now open) m31200| Sat Nov 17 09:45:03.008 [ReplicaSetMonitorWatcher] node: domU-12-31-39-16-30-A2:31101 isn't a part of set: clusteredstale-rs0 ismaster: { ismaster: false, secondary: false, info: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1353163503008), ok: 1.0 } m31200| Sat Nov 17 09:45:03.008 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m31200| Sat Nov 17 09:45:03.009 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 ok m31200| Sat Nov 17 09:45:03.009 [ReplicaSetMonitorWatcher] node: domU-12-31-39-16-30-A2:31102 isn't a part of set: clusteredstale-rs0 ismaster: { ismaster: false, secondary: false, info: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)", isreplicaset: true, maxBsonObjectSize: 16777216, localTime: new Date(1353163503009), ok: 1.0 } m31102| Sat Nov 17 09:45:03.009 [initandlisten] connection accepted from 127.0.0.1:45400 #8 (7 connections now open) m31201| Sat Nov 17 09:45:03.080 [conn22] end connection 127.0.0.1:54550 (11 connections now open) m31201| Sat Nov 17 09:45:03.080 [initandlisten] connection accepted from 127.0.0.1:54729 #24 (12 connections now open) m31101| Sat Nov 17 09:45:03.240 [rsStart] trying to contact domU-12-31-39-16-30-A2:31100 m31100| Sat Nov 17 09:45:03.241 [initandlisten] connection accepted from 127.0.0.1:60205 #53 (19 connections now open) m31101| Sat Nov 17 09:45:03.244 [rsStart] replSet I am domU-12-31-39-16-30-A2:31101 m31101| Sat Nov 17 09:45:03.244 [rsStart] replSet got config version 1 from a remote, saving locally m31101| Sat Nov 17 09:45:03.244 [rsStart] replSet info saving a newer config version to local.system.replset m31101| Sat Nov 17 09:45:03.244 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/local.ns, filling with zeroes... m31101| Sat Nov 17 09:45:03.244 [FileAllocator] creating directory /data/db/clusteredstale-rs0-1/_tmp m31101| Sat Nov 17 09:45:03.449 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/local.ns, size: 16MB, took 0.202 secs m31101| Sat Nov 17 09:45:03.450 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/local.0, filling with zeroes... m31101| Sat Nov 17 09:45:03.667 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/local.0, size: 16MB, took 0.216 secs m31101| Sat Nov 17 09:45:03.688 [rsStart] replSet saveConfigLocally done m31101| Sat Nov 17 09:45:03.689 [rsStart] replSet STARTUP2 m31101| Sat Nov 17 09:45:03.689 [rsSync] ****** m31101| Sat Nov 17 09:45:03.689 [rsSync] creating replication oplog of size: 40MB... m31101| Sat Nov 17 09:45:03.691 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/local.1, filling with zeroes... Sat Nov 17 09:45:03.993 [ReplicaSetMonitorWatcher] No primary detected for set clusteredstale-rs0 m31200| Sat Nov 17 09:45:04.010 [ReplicaSetMonitorWatcher] No primary detected for set clusteredstale-rs0 m31100| Sat Nov 17 09:45:04.529 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31101 thinks that we are down m31100| Sat Nov 17 09:45:04.529 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is now in state STARTUP2 m31100| Sat Nov 17 09:45:04.530 [rsMgr] not electing self, domU-12-31-39-16-30-A2:31101 would veto m31101| Sat Nov 17 09:45:04.816 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/local.1, size: 64MB, took 1.124 secs m31101| Sat Nov 17 09:45:04.865 [rsSync] ****** m31101| Sat Nov 17 09:45:04.865 [rsSync] replSet initial sync pending m31101| Sat Nov 17 09:45:04.865 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31101| Sat Nov 17 09:45:05.246 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31100 is up m31101| Sat Nov 17 09:45:05.246 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31100 is now in state SECONDARY m31101| Sat Nov 17 09:45:05.247 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is up m31102| Sat Nov 17 09:45:05.246 [initandlisten] connection accepted from 127.0.0.1:45412 #9 (8 connections now open) m31102| Sat Nov 17 09:45:05.556 [rsStart] trying to contact domU-12-31-39-16-30-A2:31100 m31100| Sat Nov 17 09:45:05.556 [initandlisten] connection accepted from 127.0.0.1:60217 #54 (20 connections now open) m31102| Sat Nov 17 09:45:05.557 [rsStart] trying to contact domU-12-31-39-16-30-A2:31101 m31101| Sat Nov 17 09:45:05.557 [initandlisten] connection accepted from 127.0.0.1:57990 #9 (9 connections now open) m31102| Sat Nov 17 09:45:05.558 [rsStart] replSet I am domU-12-31-39-16-30-A2:31102 m31102| Sat Nov 17 09:45:05.558 [rsStart] replSet got config version 1 from a remote, saving locally m31102| Sat Nov 17 09:45:05.558 [rsStart] replSet info saving a newer config version to local.system.replset m31102| Sat Nov 17 09:45:05.558 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/local.ns, filling with zeroes... m31102| Sat Nov 17 09:45:05.558 [FileAllocator] creating directory /data/db/clusteredstale-rs0-2/_tmp m31102| Sat Nov 17 09:45:05.759 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/local.ns, size: 16MB, took 0.198 secs m31102| Sat Nov 17 09:45:05.760 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/local.0, filling with zeroes... m31102| Sat Nov 17 09:45:06.011 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/local.0, size: 16MB, took 0.251 secs m31102| Sat Nov 17 09:45:06.035 [rsStart] replSet saveConfigLocally done m31102| Sat Nov 17 09:45:06.036 [rsStart] replSet STARTUP2 m31102| Sat Nov 17 09:45:06.036 [rsSync] ****** m31102| Sat Nov 17 09:45:06.036 [rsSync] creating replication oplog of size: 40MB... m31102| Sat Nov 17 09:45:06.038 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/local.1, filling with zeroes... m31101| Sat Nov 17 09:45:06.519 [initandlisten] connection accepted from 127.0.0.1:57991 #10 (10 connections now open) m30998| Sat Nov 17 09:45:06.519 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:45:06.520 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31102 ok m31102| Sat Nov 17 09:45:06.520 [initandlisten] connection accepted from 127.0.0.1:45418 #10 (9 connections now open) m31102| Sat Nov 17 09:45:06.520 [initandlisten] connection accepted from 127.0.0.1:45419 #11 (10 connections now open) m31100| Sat Nov 17 09:45:06.531 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 thinks that we are down m31100| Sat Nov 17 09:45:06.531 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state STARTUP2 m31100| Sat Nov 17 09:45:06.531 [rsMgr] not electing self, domU-12-31-39-16-30-A2:31102 would veto m31102| Sat Nov 17 09:45:06.949 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/local.1, size: 64MB, took 0.91 secs m31102| Sat Nov 17 09:45:06.990 [rsSync] ****** m31102| Sat Nov 17 09:45:06.991 [rsSync] replSet initial sync pending m31102| Sat Nov 17 09:45:06.991 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31101| Sat Nov 17 09:45:07.247 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31102 thinks that we are down m31101| Sat Nov 17 09:45:07.248 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state STARTUP2 m30998| Sat Nov 17 09:45:07.521 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: domU-12-31-39-16-30-A2:31100 { setName: "clusteredstale-rs0", ismaster: false, secondary: true, hosts: [ "domU-12-31-39-16-30-A2:31100", "domU-12-31-39-16-30-A2:31102", "domU-12-31-39-16-30-A2:31101" ], me: "domU-12-31-39-16-30-A2:31100", maxBsonObjectSize: 16777216, localTime: new Date(1353163507521), ok: 1.0 } m30998| Sat Nov 17 09:45:07.522 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: domU-12-31-39-16-30-A2:31101 { setName: "clusteredstale-rs0", ismaster: false, secondary: false, hosts: [ "domU-12-31-39-16-30-A2:31101", "domU-12-31-39-16-30-A2:31102", "domU-12-31-39-16-30-A2:31100" ], me: "domU-12-31-39-16-30-A2:31101", maxBsonObjectSize: 16777216, localTime: new Date(1353163507521), ok: 1.0 } m30998| Sat Nov 17 09:45:07.522 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: domU-12-31-39-16-30-A2:31102 { setName: "clusteredstale-rs0", ismaster: false, secondary: false, hosts: [ "domU-12-31-39-16-30-A2:31102", "domU-12-31-39-16-30-A2:31101", "domU-12-31-39-16-30-A2:31100" ], me: "domU-12-31-39-16-30-A2:31102", maxBsonObjectSize: 16777216, localTime: new Date(1353163507522), ok: 1.0 } m31102| Sat Nov 17 09:45:07.561 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is up m31102| Sat Nov 17 09:45:07.561 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is now in state STARTUP2 m31102| Sat Nov 17 09:45:07.561 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31100 is up m31102| Sat Nov 17 09:45:07.561 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31100 is now in state SECONDARY m30998| Sat Nov 17 09:45:08.522 [ReplicaSetMonitorWatcher] No primary detected for set clusteredstale-rs0 m30998| Sat Nov 17 09:45:08.522 [ReplicaSetMonitorWatcher] All nodes for set clusteredstale-rs0 are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks m31101| Sat Nov 17 09:45:09.201 [initandlisten] connection accepted from 127.0.0.1:57996 #11 (11 connections now open) m31102| Sat Nov 17 09:45:09.201 [initandlisten] connection accepted from 127.0.0.1:45423 #12 (11 connections now open) m30999| Sat Nov 17 09:45:10.202 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: domU-12-31-39-16-30-A2:31100 { setName: "clusteredstale-rs0", ismaster: false, secondary: true, hosts: [ "domU-12-31-39-16-30-A2:31100", "domU-12-31-39-16-30-A2:31102", "domU-12-31-39-16-30-A2:31101" ], me: "domU-12-31-39-16-30-A2:31100", maxBsonObjectSize: 16777216, localTime: new Date(1353163510202), ok: 1.0 } m30999| Sat Nov 17 09:45:10.202 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: domU-12-31-39-16-30-A2:31101 { setName: "clusteredstale-rs0", ismaster: false, secondary: false, hosts: [ "domU-12-31-39-16-30-A2:31101", "domU-12-31-39-16-30-A2:31102", "domU-12-31-39-16-30-A2:31100" ], me: "domU-12-31-39-16-30-A2:31101", maxBsonObjectSize: 16777216, localTime: new Date(1353163510202), ok: 1.0 } m30999| Sat Nov 17 09:45:10.202 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: domU-12-31-39-16-30-A2:31102 { setName: "clusteredstale-rs0", ismaster: false, secondary: false, hosts: [ "domU-12-31-39-16-30-A2:31102", "domU-12-31-39-16-30-A2:31101", "domU-12-31-39-16-30-A2:31100" ], me: "domU-12-31-39-16-30-A2:31102", maxBsonObjectSize: 16777216, localTime: new Date(1353163510202), ok: 1.0 } m30999| Sat Nov 17 09:45:11.203 [ReplicaSetMonitorWatcher] No primary detected for set clusteredstale-rs0 m31100| Sat Nov 17 09:45:11.986 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31201 m31100| Sat Nov 17 09:45:11.986 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31201 ok m31100| Sat Nov 17 09:45:11.987 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 127.0.0.1:31201 m31100| Sat Nov 17 09:45:11.987 [ReplicaSetMonitorWatcher] Detecting bad connection created at 0 microSec, clearing pool for domU-12-31-39-16-30-A2:31201 m31100| Sat Nov 17 09:45:11.987 [ReplicaSetMonitorWatcher] trying reconnect to domU-12-31-39-16-30-A2:31202 m31100| Sat Nov 17 09:45:11.987 [ReplicaSetMonitorWatcher] reconnect domU-12-31-39-16-30-A2:31202 ok m31100| Sat Nov 17 09:45:11.987 [ReplicaSetMonitorWatcher] Socket say send() errno:9 Bad file descriptor 127.0.0.1:31202 m31100| Sat Nov 17 09:45:11.987 [ReplicaSetMonitorWatcher] Detecting bad connection created at 0 microSec, clearing pool for domU-12-31-39-16-30-A2:31202 m31202| Sat Nov 17 09:45:11.987 [initandlisten] connection accepted from 127.0.0.1:39216 #24 (12 connections now open) m31201| Sat Nov 17 09:45:11.986 [initandlisten] connection accepted from 127.0.0.1:54755 #25 (13 connections now open) m31100| Sat Nov 17 09:45:12.532 [rsMgr] replSet info electSelf 0 m31100| Sat Nov 17 09:45:12.533 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state RECOVERING m31102| Sat Nov 17 09:45:12.532 [conn4] replSet RECOVERING m31102| Sat Nov 17 09:45:12.532 [conn4] replSet info voting yea for domU-12-31-39-16-30-A2:31100 (0) m31101| Sat Nov 17 09:45:12.533 [conn4] replSet RECOVERING m31101| Sat Nov 17 09:45:12.533 [conn4] replSet info voting yea for domU-12-31-39-16-30-A2:31100 (0) m31100| Sat Nov 17 09:45:12.533 [rsMgr] replSet PRIMARY m31101| Sat Nov 17 09:45:13.249 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31100 is now in state PRIMARY m31101| Sat Nov 17 09:45:13.250 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state RECOVERING m31102| Sat Nov 17 09:45:13.572 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is now in state RECOVERING m31102| Sat Nov 17 09:45:13.573 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31100 is now in state PRIMARY ReplSetTest waitForIndicator state on connection to domU-12-31-39-16-30-A2:31101 [ 2 ] ReplSetTest waitForIndicator from node connection to domU-12-31-39-16-30-A2:31101 ReplSetTest waitForIndicator Initial status ( timeout : 300000 ) : { "set" : "clusteredstale-rs0", "date" : ISODate("2012-11-17T14:45:13Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "domU-12-31-39-16-30-A2:31100", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 174, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "self" : true }, { "_id" : 1, "name" : "domU-12-31-39-16-30-A2:31101", "health" : 1, "state" : 5, "stateStr" : "STARTUP2", "uptime" : 19, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2012-11-17T14:45:12Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:45:13Z"), "pingMs" : 0, "lastHeartbeatMessage" : "initial sync need a member to be primary or secondary to do our initial sync" }, { "_id" : 2, "name" : "domU-12-31-39-16-30-A2:31102", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 17, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2012-11-17T14:45:12Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:45:13Z"), "pingMs" : 0, "lastHeartbeatMessage" : "initial sync need a member to be primary or secondary to do our initial sync" } ], "ok" : 1 } Status for : domU-12-31-39-16-30-A2:31100, checking domU-12-31-39-16-30-A2:31101/domU-12-31-39-16-30-A2:31101 Status for : domU-12-31-39-16-30-A2:31101, checking domU-12-31-39-16-30-A2:31101/domU-12-31-39-16-30-A2:31101 Status : 5 target state : 2 Status for : domU-12-31-39-16-30-A2:31102, checking domU-12-31-39-16-30-A2:31101/domU-12-31-39-16-30-A2:31101 Sat Nov 17 09:45:14.007 [ReplicaSetMonitorWatcher] Primary for replica set clusteredstale-rs0 changed to domU-12-31-39-16-30-A2:31100 Sat Nov 17 09:45:14.007 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Sat Nov 17 09:45:14.007 [ReplicaSetMonitorWatcher] Detecting bad connection created at 0 microSec, clearing pool for domU-12-31-39-16-30-A2:31101 m31102| Sat Nov 17 09:45:14.008 [initandlisten] connection accepted from 127.0.0.1:45431 #13 (12 connections now open) m31102| Sat Nov 17 09:45:14.015 [initandlisten] connection accepted from 127.0.0.1:45432 #14 (13 connections now open) m31200| Sat Nov 17 09:45:14.015 [ReplicaSetMonitorWatcher] Primary for replica set clusteredstale-rs0 changed to domU-12-31-39-16-30-A2:31100 m31200| Sat Nov 17 09:45:14.015 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m31200| Sat Nov 17 09:45:14.015 [ReplicaSetMonitorWatcher] Detecting bad connection created at 0 microSec, clearing pool for domU-12-31-39-16-30-A2:31101 m31101| Sat Nov 17 09:45:14.532 [conn4] end connection 127.0.0.1:57933 (10 connections now open) m31101| Sat Nov 17 09:45:14.533 [initandlisten] connection accepted from 127.0.0.1:58007 #12 (11 connections now open) m31100| Sat Nov 17 09:45:14.533 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is now in state RECOVERING m31201| Sat Nov 17 09:45:16.534 [conn23] end connection 127.0.0.1:54653 (12 connections now open) m31201| Sat Nov 17 09:45:16.535 [initandlisten] connection accepted from 127.0.0.1:54763 #26 (13 connections now open) m30998| Sat Nov 17 09:45:18.524 [ReplicaSetMonitorWatcher] Primary for replica set clusteredstale-rs0 changed to domU-12-31-39-16-30-A2:31100 m31200| Sat Nov 17 09:45:18.536 [conn37] end connection 127.0.0.1:46996 (19 connections now open) m31200| Sat Nov 17 09:45:18.536 [initandlisten] connection accepted from 127.0.0.1:47103 #41 (20 connections now open) m31100| Sat Nov 17 09:45:19.251 [conn53] end connection 127.0.0.1:60205 (19 connections now open) m31100| Sat Nov 17 09:45:19.252 [initandlisten] connection accepted from 127.0.0.1:60244 #55 (20 connections now open) { "set" : "clusteredstale-rs0", "date" : ISODate("2012-11-17T14:45:19Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "domU-12-31-39-16-30-A2:31100", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 180, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "self" : true }, { "_id" : 1, "name" : "domU-12-31-39-16-30-A2:31101", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 25, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2012-11-17T14:45:18Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:45:19Z"), "pingMs" : 0, "lastHeartbeatMessage" : "initial sync need a member to be primary or secondary to do our initial sync" }, { "_id" : 2, "name" : "domU-12-31-39-16-30-A2:31102", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 23, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2012-11-17T14:45:18Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:45:19Z"), "pingMs" : 0, "lastHeartbeatMessage" : "initial sync need a member to be primary or secondary to do our initial sync" } ], "ok" : 1 } Status for : domU-12-31-39-16-30-A2:31100, checking domU-12-31-39-16-30-A2:31101/domU-12-31-39-16-30-A2:31101 Status for : domU-12-31-39-16-30-A2:31101, checking domU-12-31-39-16-30-A2:31101/domU-12-31-39-16-30-A2:31101 Status : 3 target state : 2 Status for : domU-12-31-39-16-30-A2:31102, checking domU-12-31-39-16-30-A2:31101/domU-12-31-39-16-30-A2:31101 m31200| Sat Nov 17 09:45:20.757 [conn38] end connection 127.0.0.1:47000 (19 connections now open) m31200| Sat Nov 17 09:45:20.757 [initandlisten] connection accepted from 127.0.0.1:47109 #42 (20 connections now open) m31101| Sat Nov 17 09:45:20.866 [rsSync] replSet initial sync pending m31101| Sat Nov 17 09:45:20.866 [rsSync] replSet syncing to: domU-12-31-39-16-30-A2:31100 m31100| Sat Nov 17 09:45:20.867 [initandlisten] connection accepted from 127.0.0.1:60248 #56 (21 connections now open) m31101| Sat Nov 17 09:45:20.868 [rsSync] build index local.me { _id: 1 } m31101| Sat Nov 17 09:45:20.869 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Sat Nov 17 09:45:20.872 [rsSync] replSet initial sync drop all databases m31101| Sat Nov 17 09:45:20.872 [rsSync] dropAllDatabasesExceptLocal 1 m31101| Sat Nov 17 09:45:20.872 [rsSync] replSet initial sync clone all databases m31101| Sat Nov 17 09:45:20.873 [rsSync] replSet initial sync cloning db: _overflow m31100| Sat Nov 17 09:45:20.873 [initandlisten] connection accepted from 127.0.0.1:60249 #57 (22 connections now open) m31101| Sat Nov 17 09:45:20.873 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/_overflow.ns, filling with zeroes... m31101| Sat Nov 17 09:45:21.109 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/_overflow.ns, size: 16MB, took 0.235 secs m31101| Sat Nov 17 09:45:21.109 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/_overflow.0, filling with zeroes... m30999| Sat Nov 17 09:45:21.217 [ReplicaSetMonitorWatcher] Primary for replica set clusteredstale-rs0 changed to domU-12-31-39-16-30-A2:31100 m31101| Sat Nov 17 09:45:21.325 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/_overflow.0, size: 16MB, took 0.215 secs m31100| Sat Nov 17 09:45:21.592 [conn54] end connection 127.0.0.1:60217 (21 connections now open) m31100| Sat Nov 17 09:45:21.597 [initandlisten] connection accepted from 127.0.0.1:60250 #58 (22 connections now open) m31202| Sat Nov 17 09:45:22.009 [initandlisten] connection accepted from 127.0.0.1:39239 #25 (13 connections now open) m31201| Sat Nov 17 09:45:22.009 [initandlisten] connection accepted from 127.0.0.1:54778 #27 (14 connections now open) m31101| Sat Nov 17 09:45:22.069 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/_overflow.1, filling with zeroes... m31101| Sat Nov 17 09:45:22.492 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/_overflow.1, size: 32MB, took 0.423 secs m31102| Sat Nov 17 09:45:23.049 [rsSync] replSet initial sync pending m31102| Sat Nov 17 09:45:23.049 [rsSync] replSet syncing to: domU-12-31-39-16-30-A2:31100 m31102| Sat Nov 17 09:45:23.052 [rsSync] build index local.me { _id: 1 } m31100| Sat Nov 17 09:45:23.050 [initandlisten] connection accepted from 127.0.0.1:60256 #59 (23 connections now open) m31102| Sat Nov 17 09:45:23.053 [rsSync] build index done. scanned 0 total records. 0 secs m31102| Sat Nov 17 09:45:23.064 [rsSync] replSet initial sync drop all databases m31102| Sat Nov 17 09:45:23.065 [rsSync] dropAllDatabasesExceptLocal 1 m31102| Sat Nov 17 09:45:23.065 [rsSync] replSet initial sync clone all databases m31102| Sat Nov 17 09:45:23.065 [rsSync] replSet initial sync cloning db: _overflow m31100| Sat Nov 17 09:45:23.066 [initandlisten] connection accepted from 127.0.0.1:60258 #60 (24 connections now open) m31102| Sat Nov 17 09:45:23.067 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/_overflow.ns, filling with zeroes... m31101| Sat Nov 17 09:45:23.160 [rsSync] build index _overflow.coll { _id: 1 } m31102| Sat Nov 17 09:45:23.299 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/_overflow.ns, size: 16MB, took 0.221 secs m31102| Sat Nov 17 09:45:23.300 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/_overflow.0, filling with zeroes... m31102| Sat Nov 17 09:45:23.606 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/_overflow.0, size: 16MB, took 0.306 secs m31100| Sat Nov 17 09:45:24.318 [conn57] end connection 127.0.0.1:60249 (23 connections now open) m31102| Sat Nov 17 09:45:24.318 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/_overflow.1, filling with zeroes... m31101| Sat Nov 17 09:45:24.052 [initandlisten] connection accepted from 127.0.0.1:58032 #13 (12 connections now open) m31101| Sat Nov 17 09:45:24.052 [initandlisten] connection accepted from 127.0.0.1:58033 #14 (13 connections now open) m31101| Sat Nov 17 09:45:24.259 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Sat Nov 17 09:45:24.306 [rsSync] build index done. scanned 330001 total records. 1.145 secs m31101| Sat Nov 17 09:45:24.438 [rsSync] replSet initial sync cloning db: test m31100| Sat Nov 17 09:45:24.780 [initandlisten] connection accepted from 127.0.0.1:60262 #61 (24 connections now open) m31101| Sat Nov 17 09:45:24.781 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/test.ns, filling with zeroes... m31102| Sat Nov 17 09:45:24.752 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/_overflow.1, size: 32MB, took 0.433 secs m31101| Sat Nov 17 09:45:25.009 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/test.ns, size: 16MB, took 0.227 secs m31101| Sat Nov 17 09:45:25.009 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/test.0, filling with zeroes... m31100| Sat Nov 17 09:45:25.363 [conn61] end connection 127.0.0.1:60262 (23 connections now open) m31100| Sat Nov 17 09:45:25.363 [initandlisten] connection accepted from 127.0.0.1:60264 #62 (24 connections now open) m31101| Sat Nov 17 09:45:25.359 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/test.0, size: 16MB, took 0.306 secs m31101| Sat Nov 17 09:45:25.362 [rsSync] build index test.foo { _id: 1 } m31101| Sat Nov 17 09:45:25.362 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Sat Nov 17 09:45:25.362 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Sat Nov 17 09:45:25.363 [rsSync] replSet initial sync cloning db: test_a m31101| Sat Nov 17 09:45:25.363 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/test_a.ns, filling with zeroes... { "set" : "clusteredstale-rs0", "date" : ISODate("2012-11-17T14:45:25Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "domU-12-31-39-16-30-A2:31100", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 186, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "self" : true }, { "_id" : 1, "name" : "domU-12-31-39-16-30-A2:31101", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 31, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2012-11-17T14:45:24Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:45:25Z"), "pingMs" : 0, "lastHeartbeatMessage" : "initial sync cloning db: test" }, { "_id" : 2, "name" : "domU-12-31-39-16-30-A2:31102", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 29, "optime" : { "t" : 0, "i" : 0 }, "optimeDate" : ISODate("1970-01-01T00:00:00Z"), "lastHeartbeat" : ISODate("2012-11-17T14:45:24Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:45:25Z"), "pingMs" : 0, "lastHeartbeatMessage" : "initial sync cloning db: _overflow" } ], "ok" : 1 } Status for : domU-12-31-39-16-30-A2:31100, checking domU-12-31-39-16-30-A2:31101/domU-12-31-39-16-30-A2:31101 Status for : domU-12-31-39-16-30-A2:31101, checking domU-12-31-39-16-30-A2:31101/domU-12-31-39-16-30-A2:31101 Status : 3 target state : 2 Status for : domU-12-31-39-16-30-A2:31102, checking domU-12-31-39-16-30-A2:31101/domU-12-31-39-16-30-A2:31101 m31101| Sat Nov 17 09:45:25.874 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/test_a.ns, size: 16MB, took 0.51 secs m31101| Sat Nov 17 09:45:25.874 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/test_a.0, filling with zeroes... m31101| Sat Nov 17 09:45:26.145 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/test_a.0, size: 16MB, took 0.27 secs m31102| Sat Nov 17 09:45:26.151 [rsSync] build index _overflow.coll { _id: 1 } m31101| Sat Nov 17 09:45:26.171 [rsSync] build index test_a.dummy { _id: 1 } m31101| Sat Nov 17 09:45:26.172 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Sat Nov 17 09:45:26.172 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Sat Nov 17 09:45:26.172 [rsSync] replSet initial sync cloning db: admin m31101| Sat Nov 17 09:45:26.173 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/admin.ns, filling with zeroes... m31100| Sat Nov 17 09:45:26.172 [conn62] end connection 127.0.0.1:60264 (23 connections now open) m31100| Sat Nov 17 09:45:26.172 [initandlisten] connection accepted from 127.0.0.1:60267 #63 (24 connections now open) m31101| Sat Nov 17 09:45:26.444 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/admin.ns, size: 16MB, took 0.27 secs m31101| Sat Nov 17 09:45:26.444 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-1/admin.0, filling with zeroes... m31101| Sat Nov 17 09:45:26.709 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-1/admin.0, size: 16MB, took 0.265 secs m29000| Sat Nov 17 09:45:26.712 [conn17] update config.lockpings query: { _id: "domU-12-31-39-16-30-A2:30998:1353163376:1804289383" } update: { $set: { ping: new Date(1353163526567) } } nscanned:1 nupdated:1 keyUpdates:1 locks(micros) w:104 144ms m31101| Sat Nov 17 09:45:26.726 [rsSync] build index admin.foo { _id: 1 } m31101| Sat Nov 17 09:45:26.726 [rsSync] fastBuildIndex dupsToDrop:0 m31101| Sat Nov 17 09:45:26.727 [rsSync] build index done. scanned 1 total records. 0 secs m31101| Sat Nov 17 09:45:26.727 [rsSync] replSet initial sync data copy, starting syncup m31101| Sat Nov 17 09:45:26.727 [rsSync] oplog sync 1 of 3 m31101| Sat Nov 17 09:45:26.727 [rsSync] oplog sync 2 of 3 m31101| Sat Nov 17 09:45:26.727 [rsSync] replSet initial sync building indexes m31101| Sat Nov 17 09:45:26.727 [rsSync] replSet initial sync cloning indexes for : _overflow m31101| Sat Nov 17 09:45:26.728 [rsSync] replSet initial sync cloning indexes for : test m31101| Sat Nov 17 09:45:26.728 [rsSync] replSet initial sync cloning indexes for : test_a m31101| Sat Nov 17 09:45:26.729 [rsSync] replSet initial sync cloning indexes for : admin m31101| Sat Nov 17 09:45:26.729 [rsSync] oplog sync 3 of 3 m31101| Sat Nov 17 09:45:26.729 [rsSync] replSet initial sync finishing up m31100| Sat Nov 17 09:45:26.727 [conn63] end connection 127.0.0.1:60267 (23 connections now open) m31100| Sat Nov 17 09:45:26.728 [initandlisten] connection accepted from 127.0.0.1:60271 #64 (24 connections now open) m31100| Sat Nov 17 09:45:26.728 [conn64] end connection 127.0.0.1:60271 (23 connections now open) m31100| Sat Nov 17 09:45:26.728 [initandlisten] connection accepted from 127.0.0.1:60272 #65 (24 connections now open) m31100| Sat Nov 17 09:45:26.729 [conn65] end connection 127.0.0.1:60272 (23 connections now open) m31100| Sat Nov 17 09:45:26.729 [initandlisten] connection accepted from 127.0.0.1:60273 #66 (24 connections now open) m31100| Sat Nov 17 09:45:26.729 [conn66] end connection 127.0.0.1:60273 (23 connections now open) m31100| Sat Nov 17 09:45:26.729 [initandlisten] connection accepted from 127.0.0.1:60274 #67 (24 connections now open) m31100| Sat Nov 17 09:45:26.729 [conn67] end connection 127.0.0.1:60274 (23 connections now open) m31101| Sat Nov 17 09:45:26.732 [rsSync] replSet set minValid=50a7a2d8:2710 m31101| Sat Nov 17 09:45:26.733 [rsSync] build index local.replset.minvalid { _id: 1 } m31101| Sat Nov 17 09:45:26.734 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Sat Nov 17 09:45:26.734 [rsSync] replSet initial sync done m31100| Sat Nov 17 09:45:26.734 [conn56] end connection 127.0.0.1:60248 (22 connections now open) m31101| Sat Nov 17 09:45:26.917 [rsBackgroundSync] replSet syncing to: domU-12-31-39-16-30-A2:31100 m31100| Sat Nov 17 09:45:26.923 [initandlisten] connection accepted from 127.0.0.1:60275 #68 (23 connections now open) m31102| Sat Nov 17 09:45:27.165 [rsSync] fastBuildIndex dupsToDrop:0 m31102| Sat Nov 17 09:45:27.212 [rsSync] build index done. scanned 330001 total records. 1.06 secs m31100| Sat Nov 17 09:45:27.212 [conn60] end connection 127.0.0.1:60258 (22 connections now open) m31102| Sat Nov 17 09:45:27.212 [rsSync] replSet initial sync cloning db: test m31100| Sat Nov 17 09:45:27.365 [initandlisten] connection accepted from 127.0.0.1:60276 #69 (23 connections now open) m31102| Sat Nov 17 09:45:27.365 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/test.ns, filling with zeroes... m31102| Sat Nov 17 09:45:27.583 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/test.ns, size: 16MB, took 0.218 secs m31102| Sat Nov 17 09:45:27.584 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/test.0, filling with zeroes... m31101| Sat Nov 17 09:45:27.728 [rsSyncNotifier] replset setting oplog notifier to domU-12-31-39-16-30-A2:31100 m31100| Sat Nov 17 09:45:27.728 [initandlisten] connection accepted from 127.0.0.1:60277 #70 (24 connections now open) m31102| Sat Nov 17 09:45:27.848 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/test.0, size: 16MB, took 0.264 secs m31102| Sat Nov 17 09:45:27.872 [rsSync] build index test.foo { _id: 1 } m31102| Sat Nov 17 09:45:27.873 [rsSync] fastBuildIndex dupsToDrop:0 m31102| Sat Nov 17 09:45:27.873 [rsSync] build index done. scanned 1 total records. 0 secs m31100| Sat Nov 17 09:45:27.873 [conn69] end connection 127.0.0.1:60276 (23 connections now open) m31100| Sat Nov 17 09:45:27.873 [initandlisten] connection accepted from 127.0.0.1:60278 #71 (24 connections now open) m31102| Sat Nov 17 09:45:27.873 [rsSync] replSet initial sync cloning db: test_a m31102| Sat Nov 17 09:45:27.877 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/test_a.ns, filling with zeroes... m31102| Sat Nov 17 09:45:28.113 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/test_a.ns, size: 16MB, took 0.235 secs m31102| Sat Nov 17 09:45:28.114 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/test_a.0, filling with zeroes... m31102| Sat Nov 17 09:45:28.360 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/test_a.0, size: 16MB, took 0.246 secs m31102| Sat Nov 17 09:45:28.384 [rsSync] build index test_a.dummy { _id: 1 } m31102| Sat Nov 17 09:45:28.385 [rsSync] fastBuildIndex dupsToDrop:0 m31102| Sat Nov 17 09:45:28.385 [rsSync] build index done. scanned 1 total records. 0 secs m31100| Sat Nov 17 09:45:28.385 [conn71] end connection 127.0.0.1:60278 (23 connections now open) m31102| Sat Nov 17 09:45:28.385 [rsSync] replSet initial sync cloning db: admin m31100| Sat Nov 17 09:45:28.388 [initandlisten] connection accepted from 127.0.0.1:60281 #72 (24 connections now open) m31102| Sat Nov 17 09:45:28.388 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/admin.ns, filling with zeroes... m31102| Sat Nov 17 09:45:28.541 [conn4] end connection 127.0.0.1:45378 (12 connections now open) m31102| Sat Nov 17 09:45:28.541 [initandlisten] connection accepted from 127.0.0.1:45480 #15 (13 connections now open) m31102| Sat Nov 17 09:45:28.603 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/admin.ns, size: 16MB, took 0.215 secs m31102| Sat Nov 17 09:45:28.603 [FileAllocator] allocating new datafile /data/db/clusteredstale-rs0-2/admin.0, filling with zeroes... m31101| Sat Nov 17 09:45:28.736 [rsSync] replSet SECONDARY m31102| Sat Nov 17 09:45:28.768 [FileAllocator] done allocating datafile /data/db/clusteredstale-rs0-2/admin.0, size: 16MB, took 0.163 secs m31102| Sat Nov 17 09:45:28.772 [rsSync] build index admin.foo { _id: 1 } m31102| Sat Nov 17 09:45:28.773 [rsSync] fastBuildIndex dupsToDrop:0 m31102| Sat Nov 17 09:45:28.773 [rsSync] build index done. scanned 1 total records. 0 secs m31100| Sat Nov 17 09:45:28.773 [conn72] end connection 127.0.0.1:60281 (23 connections now open) m31102| Sat Nov 17 09:45:28.773 [rsSync] replSet initial sync data copy, starting syncup m31102| Sat Nov 17 09:45:28.773 [rsSync] oplog sync 1 of 3 m31102| Sat Nov 17 09:45:28.773 [rsSync] oplog sync 2 of 3 m31102| Sat Nov 17 09:45:28.773 [rsSync] replSet initial sync building indexes m31102| Sat Nov 17 09:45:28.773 [rsSync] replSet initial sync cloning indexes for : _overflow m31100| Sat Nov 17 09:45:28.774 [initandlisten] connection accepted from 127.0.0.1:60284 #73 (24 connections now open) m31100| Sat Nov 17 09:45:28.774 [conn73] end connection 127.0.0.1:60284 (23 connections now open) m31102| Sat Nov 17 09:45:28.774 [rsSync] replSet initial sync cloning indexes for : test m31100| Sat Nov 17 09:45:28.776 [initandlisten] connection accepted from 127.0.0.1:60285 #74 (24 connections now open) m31100| Sat Nov 17 09:45:28.776 [conn74] end connection 127.0.0.1:60285 (23 connections now open) m31102| Sat Nov 17 09:45:28.776 [rsSync] replSet initial sync cloning indexes for : test_a m31100| Sat Nov 17 09:45:28.776 [initandlisten] connection accepted from 127.0.0.1:60286 #75 (24 connections now open) m31100| Sat Nov 17 09:45:28.777 [conn75] end connection 127.0.0.1:60286 (23 connections now open) m31102| Sat Nov 17 09:45:28.777 [rsSync] replSet initial sync cloning indexes for : admin m31100| Sat Nov 17 09:45:28.777 [initandlisten] connection accepted from 127.0.0.1:60287 #76 (24 connections now open) m31100| Sat Nov 17 09:45:28.777 [conn76] end connection 127.0.0.1:60287 (23 connections now open) m31102| Sat Nov 17 09:45:28.777 [rsSync] oplog sync 3 of 3 m31102| Sat Nov 17 09:45:28.777 [rsSync] replSet initial sync finishing up m31102| Sat Nov 17 09:45:28.780 [rsSync] replSet set minValid=50a7a2d8:2710 m31102| Sat Nov 17 09:45:28.781 [rsSync] build index local.replset.minvalid { _id: 1 } m31102| Sat Nov 17 09:45:28.781 [rsSync] build index done. scanned 0 total records. 0 secs m31102| Sat Nov 17 09:45:28.781 [rsSync] replSet initial sync done m31100| Sat Nov 17 09:45:28.782 [conn59] end connection 127.0.0.1:60256 (22 connections now open) m31102| Sat Nov 17 09:45:29.442 [rsBackgroundSync] replSet syncing to: domU-12-31-39-16-30-A2:31100 m31100| Sat Nov 17 09:45:29.442 [initandlisten] connection accepted from 127.0.0.1:60289 #77 (23 connections now open) m31102| Sat Nov 17 09:45:29.606 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is now in state SECONDARY m31102| Sat Nov 17 09:45:29.775 [rsSyncNotifier] replset setting oplog notifier to domU-12-31-39-16-30-A2:31100 m31100| Sat Nov 17 09:45:29.777 [initandlisten] connection accepted from 127.0.0.1:60290 #78 (24 connections now open) m31202| Sat Nov 17 09:45:30.553 [conn22] end connection 127.0.0.1:39170 (12 connections now open) m31202| Sat Nov 17 09:45:30.554 [initandlisten] connection accepted from 127.0.0.1:39278 #26 (13 connections now open) m31100| Sat Nov 17 09:45:30.553 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31101 is now in state SECONDARY m31102| Sat Nov 17 09:45:30.784 [rsSync] replSet SECONDARY m31101| Sat Nov 17 09:45:31.296 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state SECONDARY { "set" : "clusteredstale-rs0", "date" : ISODate("2012-11-17T14:45:31Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "domU-12-31-39-16-30-A2:31100", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 192, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "self" : true }, { "_id" : 1, "name" : "domU-12-31-39-16-30-A2:31101", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 37, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "lastHeartbeat" : ISODate("2012-11-17T14:45:30Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:45:31Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "domU-12-31-39-16-30-A2:31102", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 35, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "lastHeartbeat" : ISODate("2012-11-17T14:45:30Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:45:31Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: domU-12-31-39-16-30-A2:31100" } ], "ok" : 1 } Status for : domU-12-31-39-16-30-A2:31100, checking domU-12-31-39-16-30-A2:31101/domU-12-31-39-16-30-A2:31101 Status for : domU-12-31-39-16-30-A2:31101, checking domU-12-31-39-16-30-A2:31101/domU-12-31-39-16-30-A2:31101 Status : 2 target state : 2 ReplSetTest waitForIndicator final status: { "set" : "clusteredstale-rs0", "date" : ISODate("2012-11-17T14:45:31Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "domU-12-31-39-16-30-A2:31100", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 192, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "self" : true }, { "_id" : 1, "name" : "domU-12-31-39-16-30-A2:31101", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 37, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "lastHeartbeat" : ISODate("2012-11-17T14:45:30Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:45:31Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "domU-12-31-39-16-30-A2:31102", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 35, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "lastHeartbeat" : ISODate("2012-11-17T14:45:30Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:45:31Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: domU-12-31-39-16-30-A2:31100" } ], "ok" : 1 } ReplSetTest waitForIndicator state on connection to domU-12-31-39-16-30-A2:31102 [ 2 ] ReplSetTest waitForIndicator from node connection to domU-12-31-39-16-30-A2:31102 ReplSetTest waitForIndicator Initial status ( timeout : 300000 ) : { "set" : "clusteredstale-rs0", "date" : ISODate("2012-11-17T14:45:31Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "domU-12-31-39-16-30-A2:31100", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 192, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "self" : true }, { "_id" : 1, "name" : "domU-12-31-39-16-30-A2:31101", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 37, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "lastHeartbeat" : ISODate("2012-11-17T14:45:30Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:45:31Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "domU-12-31-39-16-30-A2:31102", "health" : 1, "state" : 3, "stateStr" : "RECOVERING", "uptime" : 35, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "lastHeartbeat" : ISODate("2012-11-17T14:45:30Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:45:31Z"), "pingMs" : 0, "lastHeartbeatMessage" : "syncing to: domU-12-31-39-16-30-A2:31100" } ], "ok" : 1 } Status for : domU-12-31-39-16-30-A2:31100, checking domU-12-31-39-16-30-A2:31102/domU-12-31-39-16-30-A2:31102 Status for : domU-12-31-39-16-30-A2:31101, checking domU-12-31-39-16-30-A2:31102/domU-12-31-39-16-30-A2:31102 Status for : domU-12-31-39-16-30-A2:31102, checking domU-12-31-39-16-30-A2:31102/domU-12-31-39-16-30-A2:31102 Status : 3 target state : 2 m29000| Sat Nov 17 09:45:31.811 [initandlisten] connection accepted from 127.0.0.1:47343 #18 (12 connections now open) m31100| Sat Nov 17 09:45:32.553 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31102 is now in state SECONDARY m31202| Sat Nov 17 09:45:32.555 [conn23] end connection 127.0.0.1:39183 (12 connections now open) m31202| Sat Nov 17 09:45:32.555 [initandlisten] connection accepted from 127.0.0.1:39288 #27 (13 connections now open) m31201| Sat Nov 17 09:45:33.094 [conn24] end connection 127.0.0.1:54729 (13 connections now open) m31201| Sat Nov 17 09:45:33.095 [initandlisten] connection accepted from 127.0.0.1:54829 #28 (14 connections now open) m31102| Sat Nov 17 09:45:33.297 [conn9] end connection 127.0.0.1:45412 (12 connections now open) m31102| Sat Nov 17 09:45:33.297 [initandlisten] connection accepted from 127.0.0.1:45502 #16 (13 connections now open) ReplSetTest waitForIndicator final status: { "set" : "clusteredstale-rs0", "date" : ISODate("2012-11-17T14:45:33Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "domU-12-31-39-16-30-A2:31100", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 194, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "self" : true }, { "_id" : 1, "name" : "domU-12-31-39-16-30-A2:31101", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 39, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "lastHeartbeat" : ISODate("2012-11-17T14:45:32Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:45:33Z"), "pingMs" : 0 }, { "_id" : 2, "name" : "domU-12-31-39-16-30-A2:31102", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 37, "optime" : { "t" : 1353163480000, "i" : 10000 }, "optimeDate" : ISODate("2012-11-17T14:44:40Z"), "lastHeartbeat" : ISODate("2012-11-17T14:45:32Z"), "lastHeartbeatRecv" : ISODate("2012-11-17T14:45:33Z"), "pingMs" : 0 } ], "ok" : 1 } stale_clustered.js 10: check our regular and slaveOk query Awaiting domU-12-31-39-16-30-A2:31100 to be { "ok" : true } for connection to domU-12-31-39-16-30-A2:30999 (rs: undefined) { "clusteredstale-rs0" : { "hosts" : [ { "addr" : "domU-12-31-39-16-30-A2:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 3 }, { "addr" : "domU-12-31-39-16-30-A2:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 3 }, { "addr" : "domU-12-31-39-16-30-A2:31102", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 }, "clusteredstale-rs1" : { "hosts" : [ { "addr" : "domU-12-31-39-16-30-A2:31200", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "domU-12-31-39-16-30-A2:31201", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "domU-12-31-39-16-30-A2:31202", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } Awaiting domU-12-31-39-16-30-A2:31101 to be { "ok" : true } for connection to domU-12-31-39-16-30-A2:30999 (rs: undefined) { "clusteredstale-rs0" : { "hosts" : [ { "addr" : "domU-12-31-39-16-30-A2:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 3 }, { "addr" : "domU-12-31-39-16-30-A2:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 3 }, { "addr" : "domU-12-31-39-16-30-A2:31102", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 }, "clusteredstale-rs1" : { "hosts" : [ { "addr" : "domU-12-31-39-16-30-A2:31200", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "domU-12-31-39-16-30-A2:31201", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "domU-12-31-39-16-30-A2:31202", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } Awaiting domU-12-31-39-16-30-A2:31102 to be { "ok" : true } for connection to domU-12-31-39-16-30-A2:30999 (rs: undefined) { "clusteredstale-rs0" : { "hosts" : [ { "addr" : "domU-12-31-39-16-30-A2:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 3 }, { "addr" : "domU-12-31-39-16-30-A2:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 3 }, { "addr" : "domU-12-31-39-16-30-A2:31102", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 }, "clusteredstale-rs1" : { "hosts" : [ { "addr" : "domU-12-31-39-16-30-A2:31200", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "domU-12-31-39-16-30-A2:31201", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "domU-12-31-39-16-30-A2:31202", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } Awaiting domU-12-31-39-16-30-A2:31200 to be { "ok" : true } for connection to domU-12-31-39-16-30-A2:30999 (rs: undefined) { "clusteredstale-rs0" : { "hosts" : [ { "addr" : "domU-12-31-39-16-30-A2:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 3 }, { "addr" : "domU-12-31-39-16-30-A2:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 3 }, { "addr" : "domU-12-31-39-16-30-A2:31102", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 }, "clusteredstale-rs1" : { "hosts" : [ { "addr" : "domU-12-31-39-16-30-A2:31200", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "domU-12-31-39-16-30-A2:31201", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "domU-12-31-39-16-30-A2:31202", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } Awaiting domU-12-31-39-16-30-A2:31201 to be { "ok" : true } for connection to domU-12-31-39-16-30-A2:30999 (rs: undefined) { "clusteredstale-rs0" : { "hosts" : [ { "addr" : "domU-12-31-39-16-30-A2:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 3 }, { "addr" : "domU-12-31-39-16-30-A2:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 3 }, { "addr" : "domU-12-31-39-16-30-A2:31102", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 }, "clusteredstale-rs1" : { "hosts" : [ { "addr" : "domU-12-31-39-16-30-A2:31200", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "domU-12-31-39-16-30-A2:31201", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "domU-12-31-39-16-30-A2:31202", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } Awaiting domU-12-31-39-16-30-A2:31202 to be { "ok" : true } for connection to domU-12-31-39-16-30-A2:30999 (rs: undefined) { "clusteredstale-rs0" : { "hosts" : [ { "addr" : "domU-12-31-39-16-30-A2:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 3 }, { "addr" : "domU-12-31-39-16-30-A2:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 3 }, { "addr" : "domU-12-31-39-16-30-A2:31102", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 }, "clusteredstale-rs1" : { "hosts" : [ { "addr" : "domU-12-31-39-16-30-A2:31200", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "domU-12-31-39-16-30-A2:31201", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "domU-12-31-39-16-30-A2:31202", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } stale_clustered.js SlaveOK Query... m30998| Sat Nov 17 09:45:33.721 [conn1] Socket recv() errno:104 Connection reset by peer 127.0.0.1:31100 m30998| Sat Nov 17 09:45:33.721 [conn1] SocketException: remote: 127.0.0.1:31100 error: 9001 socket exception [1] server [127.0.0.1:31100] m30998| Sat Nov 17 09:45:33.721 [conn1] DBClientCursor::init lazy say() failed m30998| Sat Nov 17 09:45:33.721 [conn1] DBClientCursor::init message from say() was empty m30998| Sat Nov 17 09:45:33.721 [conn1] slave no longer has secondary status: domU-12-31-39-16-30-A2:31100 m31100| Sat Nov 17 09:45:33.721 [conn41] end connection 127.0.0.1:60144 (23 connections now open) m30998| Sat Nov 17 09:45:33.721 [conn1] sharded connection to clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102 not being returned to the pool m30998| Sat Nov 17 09:45:33.741 [conn1] socket exception when initializing on clusteredstale-rs0:clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102, current connection state is { state: { conn: "", vinfo: "test.foo @ 2|1||50a7a275e10b3cbd7ae1b532", cursor: "(none)", count: 0, done: false }, retryNext: true, init: false, finish: false, errored: false } :: caused by :: 11002 socket exception [6] server [clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102] mongos shardconnection connectionpool error: connect failed to replica set clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102 m30998| Sat Nov 17 09:45:33.741 [conn1] DBException in process: socket exception [CONNECT_ERROR] for clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102 Sat Nov 17 09:45:33.742 exec error: src/mongo/shell/query.js:128 error: { "$err" : "socket exception [CONNECT_ERROR] for clusteredstale-rs0/domU-12-31-39-16-30-A2:31100,domU-12-31-39-16-30-A2:31101,domU-12-31-39-16-30-A2:31102", "code" : 11002, "shard" : "clusteredstale-rs0" } throw "error: " + tojson( ret ); ^ failed to load: /mnt/data/slaves/Linux_64bit_Legacy/mongo/jstests/replsets/stale_clustered.js m29000| Sat Nov 17 09:45:33.743 got signal 15 (Terminated), will terminate after current cmd ends m29000| Sat Nov 17 09:45:33.743 [interruptThread] now exiting m29000| Sat Nov 17 09:45:33.743 dbexit: m29000| Sat Nov 17 09:45:33.743 [interruptThread] shutdown: going to close listening sockets... m29000| Sat Nov 17 09:45:33.743 [interruptThread] closing listening socket: 41 m29000| Sat Nov 17 09:45:33.743 [interruptThread] closing listening socket: 42 m29000| Sat Nov 17 09:45:33.743 [interruptThread] closing listening socket: 43 m29000| Sat Nov 17 09:45:33.743 [interruptThread] removing socket file: /tmp/mongodb-29000.sock m29000| Sat Nov 17 09:45:33.743 [interruptThread] shutdown: going to flush diaglog... m29000| Sat Nov 17 09:45:33.743 [interruptThread] shutdown: going to close sockets... m29000| Sat Nov 17 09:45:33.743 [interruptThread] shutdown: waiting for fs preallocator... m29000| Sat Nov 17 09:45:33.743 [interruptThread] shutdown: lock for final commit... m29000| Sat Nov 17 09:45:33.743 [interruptThread] shutdown: final commit... m29000| Sat Nov 17 09:45:33.744 [interruptThread] shutdown: closing all files... m29000| Sat Nov 17 09:45:33.748 [interruptThread] closeAllFiles() finished m29000| Sat Nov 17 09:45:33.748 [interruptThread] journalCleanup... m29000| Sat Nov 17 09:45:33.748 [interruptThread] removeJournalFiles m29000| Sat Nov 17 09:45:33.750 [interruptThread] shutdown: removing fs lock... m29000| Sat Nov 17 09:45:33.750 dbexit: really exiting now m30998| Sat Nov 17 09:45:34.743 [mongosMain] dbexit: received signal 15 rc:0 received signal 15 m31100| Sat Nov 17 09:45:34.745 [conn44] end connection 127.0.0.1:60158 (22 connections now open) m31101| Sat Nov 17 09:45:34.746 [conn5] end connection 127.0.0.1:57941 (12 connections now open) m31100| Sat Nov 17 09:45:34.746 [conn45] end connection 127.0.0.1:60159 (21 connections now open) m31200| Sat Nov 17 09:45:34.746 [conn26] end connection 127.0.0.1:46195 (19 connections now open) m31201| Sat Nov 17 09:45:34.746 [conn12] end connection 127.0.0.1:53860 (13 connections now open) m31200| Sat Nov 17 09:45:34.746 [conn28] end connection 127.0.0.1:46201 (18 connections now open) m31200| Sat Nov 17 09:45:34.747 [conn30] end connection 127.0.0.1:46251 (17 connections now open) m31101| Sat Nov 17 09:45:34.746 [conn10] end connection 127.0.0.1:57991 (11 connections now open) m31201| Sat Nov 17 09:45:34.746 [conn14] end connection 127.0.0.1:53867 (12 connections now open) m31201| Sat Nov 17 09:45:34.747 [conn16] end connection 127.0.0.1:53916 (11 connections now open) m31102| Sat Nov 17 09:45:34.747 [conn10] end connection 127.0.0.1:45418 (12 connections now open) m31202| Sat Nov 17 09:45:34.746 [conn12] end connection 127.0.0.1:38321 (12 connections now open) m31202| Sat Nov 17 09:45:34.747 [conn15] end connection 127.0.0.1:38377 (11 connections now open) m31102| Sat Nov 17 09:45:34.747 [conn11] end connection 127.0.0.1:45419 (11 connections now open) m31101| Sat Nov 17 09:45:35.608 [conn9] end connection 127.0.0.1:57990 (10 connections now open) m31101| Sat Nov 17 09:45:35.609 [initandlisten] connection accepted from 127.0.0.1:58083 #15 (11 connections now open) m31100| Sat Nov 17 09:45:35.746 [conn22] end connection 127.0.0.1:49343 (20 connections now open) m31100| Sat Nov 17 09:45:35.746 [conn47] end connection 127.0.0.1:60173 (19 connections now open) m31100| Sat Nov 17 09:45:35.747 [conn48] end connection 127.0.0.1:60174 (18 connections now open) m31100| Sat Nov 17 09:45:35.747 [conn23] end connection 127.0.0.1:49352 (17 connections now open) m31200| Sat Nov 17 09:45:35.746 [conn18] end connection 127.0.0.1:34198 (16 connections now open) m31200| Sat Nov 17 09:45:35.747 [conn29] end connection 127.0.0.1:46213 (15 connections now open) m31200| Sat Nov 17 09:45:35.747 [conn16] end connection 127.0.0.1:34192 (14 connections now open) m31200| Sat Nov 17 09:45:35.747 [conn19] end connection 127.0.0.1:34204 (13 connections now open) m31101| Sat Nov 17 09:45:35.747 [conn6] end connection 127.0.0.1:57949 (10 connections now open) m31101| Sat Nov 17 09:45:35.747 [conn11] end connection 127.0.0.1:57996 (9 connections now open) m31202| Sat Nov 17 09:45:35.747 [conn6] end connection 127.0.0.1:60922 (10 connections now open) m31202| Sat Nov 17 09:45:35.747 [conn14] end connection 127.0.0.1:38339 (9 connections now open) m30999| Sat Nov 17 09:45:35.744 [mongosMain] dbexit: received signal 15 rc:0 received signal 15 m31102| Sat Nov 17 09:45:35.747 [conn3] end connection 127.0.0.1:45376 (10 connections now open) m31102| Sat Nov 17 09:45:35.748 [conn12] end connection 127.0.0.1:45423 (9 connections now open) m31201| Sat Nov 17 09:45:35.747 [conn7] end connection 127.0.0.1:36398 (10 connections now open) m31201| Sat Nov 17 09:45:35.747 [conn15] end connection 127.0.0.1:53878 (9 connections now open) m31100| Sat Nov 17 09:45:36.745 got signal 15 (Terminated), will terminate after current cmd ends m31100| Sat Nov 17 09:45:36.745 [interruptThread] now exiting m31100| Sat Nov 17 09:45:36.745 dbexit: m31100| Sat Nov 17 09:45:36.745 [interruptThread] shutdown: going to close listening sockets... m31100| Sat Nov 17 09:45:36.745 [interruptThread] closing listening socket: 11 m31100| Sat Nov 17 09:45:36.745 [interruptThread] closing listening socket: 12 m31100| Sat Nov 17 09:45:36.745 [interruptThread] closing listening socket: 13 m31100| Sat Nov 17 09:45:36.745 [interruptThread] removing socket file: /tmp/mongodb-31100.sock m31100| Sat Nov 17 09:45:36.745 [interruptThread] shutdown: going to flush diaglog... m31100| Sat Nov 17 09:45:36.745 [interruptThread] shutdown: going to close sockets... m31100| Sat Nov 17 09:45:36.745 [interruptThread] shutdown: waiting for fs preallocator... m31100| Sat Nov 17 09:45:36.745 [interruptThread] shutdown: lock for final commit... m31100| Sat Nov 17 09:45:36.745 [interruptThread] shutdown: final commit... m31102| Sat Nov 17 09:45:36.745 [conn15] end connection 127.0.0.1:45480 (8 connections now open) m31101| Sat Nov 17 09:45:36.746 [conn12] end connection 127.0.0.1:58007 (8 connections now open) m31200| Sat Nov 17 09:45:36.746 [conn40] end connection 127.0.0.1:47054 (12 connections now open) m31202| Sat Nov 17 09:45:36.746 [conn25] end connection 127.0.0.1:39239 (8 connections now open) m31100| Sat Nov 17 09:45:36.746 [interruptThread] shutdown: closing all files... m31201| Sat Nov 17 09:45:36.746 [conn27] end connection 127.0.0.1:54778 (8 connections now open) m31102| Sat Nov 17 09:45:36.752 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: domU-12-31-39-16-30-A2:31100 m31200| Sat Nov 17 09:45:36.763 [conn39] end connection 127.0.0.1:47053 (11 connections now open) m31201| Sat Nov 17 09:45:36.764 [conn25] end connection 127.0.0.1:54755 (7 connections now open) m31202| Sat Nov 17 09:45:36.763 [conn24] end connection 127.0.0.1:39216 (7 connections now open) m31101| Sat Nov 17 09:45:36.763 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: domU-12-31-39-16-30-A2:31100 m31100| Sat Nov 17 09:45:36.783 [interruptThread] closeAllFiles() finished m31100| Sat Nov 17 09:45:36.783 [interruptThread] journalCleanup... m31100| Sat Nov 17 09:45:36.783 [interruptThread] removeJournalFiles m31100| Sat Nov 17 09:45:36.786 [interruptThread] shutdown: removing fs lock... m31100| Sat Nov 17 09:45:36.786 dbexit: really exiting now m31101| Sat Nov 17 09:45:37.298 [rsHealthPoll] DBClientCursor::init call() failed m31101| Sat Nov 17 09:45:37.298 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31100 heartbeat failed, retrying m31101| Sat Nov 17 09:45:37.298 [rsHealthPoll] replSet info domU-12-31-39-16-30-A2:31100 is down (or slow to respond): m31101| Sat Nov 17 09:45:37.298 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31100 is now in state DOWN m31101| Sat Nov 17 09:45:37.298 [rsMgr] not electing self, domU-12-31-39-16-30-A2:31102 would veto m31102| Sat Nov 17 09:45:37.609 [rsHealthPoll] DBClientCursor::init call() failed m31102| Sat Nov 17 09:45:37.609 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31100 heartbeat failed, retrying m31102| Sat Nov 17 09:45:37.609 [rsHealthPoll] replSet info domU-12-31-39-16-30-A2:31100 is down (or slow to respond): m31102| Sat Nov 17 09:45:37.609 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31100 is now in state DOWN m31101| Sat Nov 17 09:45:37.746 got signal 15 (Terminated), will terminate after current cmd ends m31101| Sat Nov 17 09:45:37.746 [interruptThread] now exiting m31101| Sat Nov 17 09:45:37.746 dbexit: m31101| Sat Nov 17 09:45:37.746 [interruptThread] shutdown: going to close listening sockets... m31101| Sat Nov 17 09:45:37.746 [interruptThread] closing listening socket: 49 m31101| Sat Nov 17 09:45:37.746 [interruptThread] closing listening socket: 50 m31101| Sat Nov 17 09:45:37.746 [interruptThread] closing listening socket: 51 m31101| Sat Nov 17 09:45:37.746 [interruptThread] removing socket file: /tmp/mongodb-31101.sock m31101| Sat Nov 17 09:45:37.746 [interruptThread] shutdown: going to flush diaglog... m31101| Sat Nov 17 09:45:37.746 [interruptThread] shutdown: going to close sockets... m31101| Sat Nov 17 09:45:37.746 [interruptThread] shutdown: waiting for fs preallocator... m31101| Sat Nov 17 09:45:37.746 [interruptThread] shutdown: lock for final commit... m31101| Sat Nov 17 09:45:37.746 [interruptThread] shutdown: final commit... m31102| Sat Nov 17 09:45:37.746 [conn16] end connection 127.0.0.1:45502 (7 connections now open) m31101| Sat Nov 17 09:45:37.822 [conn15] end connection 127.0.0.1:58083 (7 connections now open) m31102| Sat Nov 17 09:45:37.822 [MultiCommandJob] DBClientCursor::init call() failed m31102| Sat Nov 17 09:45:37.822 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes m31101| Sat Nov 17 09:45:38.058 [interruptThread] shutdown: closing all files... m31101| Sat Nov 17 09:45:38.067 [interruptThread] closeAllFiles() finished m31101| Sat Nov 17 09:45:38.067 [interruptThread] journalCleanup... m31101| Sat Nov 17 09:45:38.067 [interruptThread] removeJournalFiles m31101| Sat Nov 17 09:45:38.069 [interruptThread] shutdown: removing fs lock... m31101| Sat Nov 17 09:45:38.071 dbexit: really exiting now m31102| Sat Nov 17 09:45:38.746 got signal 15 (Terminated), will terminate after current cmd ends m31102| Sat Nov 17 09:45:38.747 [interruptThread] now exiting m31102| Sat Nov 17 09:45:38.747 dbexit: m31102| Sat Nov 17 09:45:38.747 [interruptThread] shutdown: going to close listening sockets... m31102| Sat Nov 17 09:45:38.747 [interruptThread] closing listening socket: 51 m31102| Sat Nov 17 09:45:38.747 [interruptThread] closing listening socket: 52 m31102| Sat Nov 17 09:45:38.747 [interruptThread] closing listening socket: 53 m31102| Sat Nov 17 09:45:38.747 [interruptThread] removing socket file: /tmp/mongodb-31102.sock m31102| Sat Nov 17 09:45:38.747 [interruptThread] shutdown: going to flush diaglog... m31102| Sat Nov 17 09:45:38.747 [interruptThread] shutdown: going to close sockets... m31102| Sat Nov 17 09:45:38.747 [interruptThread] shutdown: waiting for fs preallocator... m31102| Sat Nov 17 09:45:38.747 [interruptThread] shutdown: lock for final commit... m31102| Sat Nov 17 09:45:38.747 [interruptThread] shutdown: final commit... m31102| Sat Nov 17 09:45:39.096 [interruptThread] shutdown: closing all files... m31102| Sat Nov 17 09:45:39.104 [interruptThread] closeAllFiles() finished m31102| Sat Nov 17 09:45:39.104 [interruptThread] journalCleanup... m31102| Sat Nov 17 09:45:39.104 [interruptThread] removeJournalFiles m31102| Sat Nov 17 09:45:39.105 [interruptThread] shutdown: removing fs lock... m31102| Sat Nov 17 09:45:39.105 dbexit: really exiting now m31200| Sat Nov 17 09:45:39.747 got signal 15 (Terminated), will terminate after current cmd ends m31200| Sat Nov 17 09:45:39.747 [interruptThread] now exiting m31200| Sat Nov 17 09:45:39.747 dbexit: m31200| Sat Nov 17 09:45:39.747 [interruptThread] shutdown: going to close listening sockets... m31200| Sat Nov 17 09:45:39.747 [interruptThread] closing listening socket: 20 m31200| Sat Nov 17 09:45:39.748 [interruptThread] closing listening socket: 21 m31200| Sat Nov 17 09:45:39.748 [interruptThread] closing listening socket: 22 m31200| Sat Nov 17 09:45:39.748 [interruptThread] removing socket file: /tmp/mongodb-31200.sock m31200| Sat Nov 17 09:45:39.748 [interruptThread] shutdown: going to flush diaglog... m31200| Sat Nov 17 09:45:39.748 [interruptThread] shutdown: going to close sockets... m31200| Sat Nov 17 09:45:39.748 [interruptThread] shutdown: waiting for fs preallocator... m31200| Sat Nov 17 09:45:39.748 [interruptThread] shutdown: lock for final commit... m31200| Sat Nov 17 09:45:39.748 [interruptThread] shutdown: final commit... m31200| Sat Nov 17 09:45:39.748 [interruptThread] shutdown: closing all files... m31200| Sat Nov 17 09:45:39.748 [interruptThread] closeAllFiles() finished m31200| Sat Nov 17 09:45:39.748 [interruptThread] journalCleanup... m31200| Sat Nov 17 09:45:39.748 [interruptThread] removeJournalFiles m31200| Sat Nov 17 09:45:39.750 [interruptThread] shutdown: removing fs lock... m31200| Sat Nov 17 09:45:39.750 dbexit: really exiting now m31202| Sat Nov 17 09:45:39.749 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: domU-12-31-39-16-30-A2:31200 m31202| Sat Nov 17 09:45:39.749 [conn26] end connection 127.0.0.1:39278 (6 connections now open) m31201| Sat Nov 17 09:45:39.749 [conn26] end connection 127.0.0.1:54763 (6 connections now open) m31201| Sat Nov 17 09:45:39.749 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: domU-12-31-39-16-30-A2:31200 m31201| Sat Nov 17 09:45:40.545 [rsHealthPoll] DBClientCursor::init call() failed m31201| Sat Nov 17 09:45:40.545 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31200 heartbeat failed, retrying m31201| Sat Nov 17 09:45:40.546 [rsHealthPoll] replSet info domU-12-31-39-16-30-A2:31200 is down (or slow to respond): m31201| Sat Nov 17 09:45:40.546 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31200 is now in state DOWN m31201| Sat Nov 17 09:45:40.546 [rsMgr] not electing self, domU-12-31-39-16-30-A2:31202 would veto m31201| Sat Nov 17 09:45:40.748 got signal 15 (Terminated), will terminate after current cmd ends m31201| Sat Nov 17 09:45:40.748 [interruptThread] now exiting m31201| Sat Nov 17 09:45:40.748 dbexit: m31201| Sat Nov 17 09:45:40.748 [interruptThread] shutdown: going to close listening sockets... m31201| Sat Nov 17 09:45:40.748 [interruptThread] closing listening socket: 23 m31201| Sat Nov 17 09:45:40.748 [interruptThread] closing listening socket: 24 m31201| Sat Nov 17 09:45:40.748 [interruptThread] closing listening socket: 25 m31201| Sat Nov 17 09:45:40.748 [interruptThread] removing socket file: /tmp/mongodb-31201.sock m31201| Sat Nov 17 09:45:40.748 [interruptThread] shutdown: going to flush diaglog... m31201| Sat Nov 17 09:45:40.748 [interruptThread] shutdown: going to close sockets... m31201| Sat Nov 17 09:45:40.748 [interruptThread] shutdown: waiting for fs preallocator... m31201| Sat Nov 17 09:45:40.748 [interruptThread] shutdown: lock for final commit... m31202| Sat Nov 17 09:45:40.748 [conn27] end connection 127.0.0.1:39288 (5 connections now open) m31201| Sat Nov 17 09:45:40.748 [interruptThread] shutdown: final commit... m31201| Sat Nov 17 09:45:40.750 [interruptThread] shutdown: closing all files... m31201| Sat Nov 17 09:45:40.751 [interruptThread] closeAllFiles() finished m31201| Sat Nov 17 09:45:40.751 [interruptThread] journalCleanup... m31201| Sat Nov 17 09:45:40.751 [interruptThread] removeJournalFiles m31201| Sat Nov 17 09:45:40.762 [interruptThread] shutdown: removing fs lock... m31201| Sat Nov 17 09:45:40.762 dbexit: really exiting now m31202| Sat Nov 17 09:45:40.798 [rsHealthPoll] DBClientCursor::init call() failed m31202| Sat Nov 17 09:45:40.798 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31200 heartbeat failed, retrying m31202| Sat Nov 17 09:45:40.799 [rsHealthPoll] replSet info domU-12-31-39-16-30-A2:31200 is down (or slow to respond): m31202| Sat Nov 17 09:45:40.799 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31200 is now in state DOWN m31202| Sat Nov 17 09:45:40.799 [MultiCommandJob] DBClientCursor::init call() failed m31202| Sat Nov 17 09:45:40.799 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes m31202| Sat Nov 17 09:45:41.098 [rsHealthPoll] replset info domU-12-31-39-16-30-A2:31201 heartbeat failed, retrying m31202| Sat Nov 17 09:45:41.098 [rsHealthPoll] replSet info domU-12-31-39-16-30-A2:31201 is down (or slow to respond): m31202| Sat Nov 17 09:45:41.098 [rsHealthPoll] replSet member domU-12-31-39-16-30-A2:31201 is now in state DOWN m31202| Sat Nov 17 09:45:41.098 [rsMgr] replSet can't see a majority, will not try to elect self m31202| Sat Nov 17 09:45:41.749 got signal 15 (Terminated), will terminate after current cmd ends m31202| Sat Nov 17 09:45:41.749 [interruptThread] now exiting m31202| Sat Nov 17 09:45:41.749 dbexit: m31202| Sat Nov 17 09:45:41.749 [interruptThread] shutdown: going to close listening sockets... m31202| Sat Nov 17 09:45:41.749 [interruptThread] closing listening socket: 26 m31202| Sat Nov 17 09:45:41.751 [interruptThread] closing listening socket: 27 m31202| Sat Nov 17 09:45:41.751 [interruptThread] closing listening socket: 28 m31202| Sat Nov 17 09:45:41.751 [interruptThread] removing socket file: /tmp/mongodb-31202.sock m31202| Sat Nov 17 09:45:41.751 [interruptThread] shutdown: going to flush diaglog... m31202| Sat Nov 17 09:45:41.751 [interruptThread] shutdown: going to close sockets... m31202| Sat Nov 17 09:45:41.751 [interruptThread] shutdown: waiting for fs preallocator... m31202| Sat Nov 17 09:45:41.751 [interruptThread] shutdown: lock for final commit... m31202| Sat Nov 17 09:45:41.751 [interruptThread] shutdown: final commit... m31202| Sat Nov 17 09:45:41.752 [interruptThread] shutdown: closing all files... m31202| Sat Nov 17 09:45:41.752 [interruptThread] closeAllFiles() finished m31202| Sat Nov 17 09:45:41.752 [interruptThread] journalCleanup Sat Nov 17 09:45:42.866 got signal 15 (Terminated), will terminate after current cmd ends Sat Nov 17 09:45:42.866 [interruptThread] now exiting Sat Nov 17 09:45:42.866 dbexit: Sat Nov 17 09:45:42.866 [interruptThread] shutdown: going to close listening sockets... Sat Nov 17 09:45:42.866 [interruptThread] closing listening socket: 8 Sat Nov 17 09:45:42.866 [interruptThread] closing listening socket: 9 Sat Nov 17 09:45:42.866 [interruptThread] closing listening socket: 10 Sat Nov 17 09:45:42.866 [interruptThread] removing socket file: /tmp/mongodb-27999.sock Sat Nov 17 09:45:42.866 [interruptThread] shutdown: going to flush diaglog... Sat Nov 17 09:45:42.866 [interruptThread] shutdown: going to close sockets... Sat Nov 17 09:45:42.866 [interruptThread] shutdown: waiting for fs preallocator... Sat Nov 17 09:45:42.866 [interruptThread] shutdown: lock for final commit... Sat Nov 17 09:45:42.866 [interruptThread] shutdown: final commit... Sat Nov 17 09:45:42.866 [interruptThread] shutdown: closing all files... Sat Nov 17 09:45:42.866 [interruptThread] closeAllFiles() finished Sat Nov 17 09:45:42.866 [interruptThread] journalCleanup... Sat Nov 17 09:45:42.866 [interruptThread] removeJournalFiles Sat Nov 17 09:45:42.867 [interruptThread] shutdown: removing fs lock... Sat Nov 17 09:45:42.867 dbexit: really exiting now