Fri May 10 08:19:06.237 [conn106] end connection 127.0.0.1:53171 (0 connections now open) Fri May 10 08:19:21.114 [initandlisten] connection accepted from 127.0.0.1:53251 #107 (1 connection now open) MongoDB shell version: 2.5.0-pre- MONGODB-CR Replica set test! ReplSetTest Starting Set ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 31100, 31101 ] 31100 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31100, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "test-rs0", "dbpath" : "$set-$node", "useHostname" : true, "noJournalPrealloc" : undefined, "pathOpts" : { "testName" : "test", "shard" : 0, "node" : 0, "set" : "test-rs0" }, "restart" : undefined } ReplSetTest Starting.... Resetting db path '/data/db/test-rs0-0' Fri May 10 08:19:21.846 shell: started program mongod.exe --oplogSize 40 --port 31100 --noprealloc --smallfiles --rest --replSet test-rs0 --dbpath /data/db/test-rs0-0 --setParameter enableTestCommands=1 m31100| note: noprealloc may hurt performance in many applications m31100| Fri May 10 08:19:21.870 m31100| Fri May 10 08:19:21.871 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability. m31100| Fri May 10 08:19:21.871 m31100| Fri May 10 08:19:21.879 [initandlisten] MongoDB starting : pid=3720 port=31100 dbpath=/data/db/test-rs0-0 32-bit host=ip-0A420969 m31100| Fri May 10 08:19:21.879 [initandlisten] m31100| Fri May 10 08:19:21.879 [initandlisten] ** NOTE: This is a development version (2.5.0-pre-) of MongoDB. m31100| Fri May 10 08:19:21.879 [initandlisten] ** Not recommended for production. m31100| Fri May 10 08:19:21.879 [initandlisten] m31100| Fri May 10 08:19:21.879 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary. m31100| Fri May 10 08:19:21.879 [initandlisten] ** 32 bit builds are limited to less than 2GB of data (or less with --journal). m31100| Fri May 10 08:19:21.879 [initandlisten] ** Note that journaling defaults to off for 32 bit and is currently off. m31100| Fri May 10 08:19:21.879 [initandlisten] ** See http://dochub.mongodb.org/core/32bit m31100| Fri May 10 08:19:21.879 [initandlisten] m31100| Fri May 10 08:19:21.879 [initandlisten] db version v2.5.0-pre- m31100| Fri May 10 08:19:21.879 [initandlisten] git version: e54e86b4d3c36f6762ff63c03d824b1eaec0c2df m31100| Fri May 10 08:19:21.879 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=0, build=6002, platform=2, service_pack='Service Pack 2') BOOST_LIB_VERSION=1_49 m31100| Fri May 10 08:19:21.879 [initandlisten] allocator: system m31100| Fri May 10 08:19:21.879 [initandlisten] options: { dbpath: "/data/db/test-rs0-0", noprealloc: true, oplogSize: 40, port: 31100, replSet: "test-rs0", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31100| Fri May 10 08:19:21.887 [FileAllocator] allocating new datafile /data/db/test-rs0-0\local.ns, filling with zeroes... m31100| Fri May 10 08:19:21.888 [FileAllocator] creating directory /data/db/test-rs0-0\_tmp m31100| Fri May 10 08:19:21.943 [FileAllocator] done allocating datafile /data/db/test-rs0-0\local.ns, size: 16MB, took 0.054 secs m31100| Fri May 10 08:19:21.943 [FileAllocator] allocating new datafile /data/db/test-rs0-0\local.0, filling with zeroes... m31100| Fri May 10 08:19:21.998 [FileAllocator] done allocating datafile /data/db/test-rs0-0\local.0, size: 16MB, took 0.054 secs m31100| Fri May 10 08:19:21.999 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 111ms m31100| Fri May 10 08:19:22.000 [initandlisten] waiting for connections on port 31100 m31100| Fri May 10 08:19:22.000 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31100| Fri May 10 08:19:22.000 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31100| Fri May 10 08:19:22.000 [websvr] admin web console waiting for connections on port 32100 m31100| Fri May 10 08:19:22.348 [initandlisten] connection accepted from 127.0.0.1:53260 #1 (1 connection now open) [ connection to ip-0A420969:31100 ] ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 31100, 31101 ] 31101 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : undefined, "port" : 31101, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "test-rs0", "dbpath" : "$set-$node", "useHostname" : true, "noJournalPrealloc" : undefined, "pathOpts" : { "testName" : "test", "shard" : 0, "node" : 1, "set" : "test-rs0" }, "restart" : undefined } ReplSetTest Starting.... Resetting db path '/data/db/test-rs0-1' Fri May 10 08:19:22.351 shell: started program mongod.exe --oplogSize 40 --port 31101 --noprealloc --smallfiles --rest --replSet test-rs0 --dbpath /data/db/test-rs0-1 --setParameter enableTestCommands=1 m31101| note: noprealloc may hurt performance in many applications m31101| Fri May 10 08:19:22.376 m31101| Fri May 10 08:19:22.377 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability. m31101| Fri May 10 08:19:22.377 m31101| Fri May 10 08:19:22.385 [initandlisten] MongoDB starting : pid=2724 port=31101 dbpath=/data/db/test-rs0-1 32-bit host=ip-0A420969 m31101| Fri May 10 08:19:22.385 [initandlisten] m31101| Fri May 10 08:19:22.385 [initandlisten] ** NOTE: This is a development version (2.5.0-pre-) of MongoDB. m31101| Fri May 10 08:19:22.385 [initandlisten] ** Not recommended for production. m31101| Fri May 10 08:19:22.385 [initandlisten] m31101| Fri May 10 08:19:22.385 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary. m31101| Fri May 10 08:19:22.385 [initandlisten] ** 32 bit builds are limited to less than 2GB of data (or less with --journal). m31101| Fri May 10 08:19:22.385 [initandlisten] ** Note that journaling defaults to off for 32 bit and is currently off. m31101| Fri May 10 08:19:22.385 [initandlisten] ** See http://dochub.mongodb.org/core/32bit m31101| Fri May 10 08:19:22.385 [initandlisten] m31101| Fri May 10 08:19:22.385 [initandlisten] db version v2.5.0-pre- m31101| Fri May 10 08:19:22.385 [initandlisten] git version: e54e86b4d3c36f6762ff63c03d824b1eaec0c2df m31101| Fri May 10 08:19:22.385 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=0, build=6002, platform=2, service_pack='Service Pack 2') BOOST_LIB_VERSION=1_49 m31101| Fri May 10 08:19:22.385 [initandlisten] allocator: system m31101| Fri May 10 08:19:22.385 [initandlisten] options: { dbpath: "/data/db/test-rs0-1", noprealloc: true, oplogSize: 40, port: 31101, replSet: "test-rs0", rest: true, setParameter: [ "enableTestCommands=1" ], smallfiles: true } m31101| Fri May 10 08:19:22.395 [FileAllocator] allocating new datafile /data/db/test-rs0-1\local.ns, filling with zeroes... m31101| Fri May 10 08:19:22.395 [FileAllocator] creating directory /data/db/test-rs0-1\_tmp m31101| Fri May 10 08:19:22.444 [FileAllocator] done allocating datafile /data/db/test-rs0-1\local.ns, size: 16MB, took 0.048 secs m31101| Fri May 10 08:19:22.444 [FileAllocator] allocating new datafile /data/db/test-rs0-1\local.0, filling with zeroes... m31101| Fri May 10 08:19:22.493 [FileAllocator] done allocating datafile /data/db/test-rs0-1\local.0, size: 16MB, took 0.048 secs m31101| Fri May 10 08:19:22.495 [initandlisten] waiting for connections on port 31101 m31101| Fri May 10 08:19:22.495 [websvr] admin web console waiting for connections on port 32101 m31101| Fri May 10 08:19:22.495 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31101| Fri May 10 08:19:22.495 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31101| Fri May 10 08:19:22.847 [initandlisten] connection accepted from 127.0.0.1:53261 #1 (1 connection now open) [ connection to ip-0A420969:31100, connection to ip-0A420969:31101 ] { "replSetInitiate" : { "_id" : "test-rs0", "members" : [ { "_id" : 0, "host" : "ip-0A420969:31100" }, { "_id" : 1, "host" : "ip-0A420969:31101" } ] } } m31100| Fri May 10 08:19:22.850 [conn1] replSet replSetInitiate admin command received from client m31100| Fri May 10 08:19:22.850 [conn1] replSet replSetInitiate config object parses ok, 2 members specified m31100| Fri May 10 08:19:22.852 [initandlisten] connection accepted from 10.66.9.105:53262 #2 (2 connections now open) m31100| Fri May 10 08:19:22.852 [conn2] end connection 10.66.9.105:53262 (1 connection now open) m31101| Fri May 10 08:19:22.853 [initandlisten] connection accepted from 10.66.9.105:53263 #2 (2 connections now open) m31100| Fri May 10 08:19:22.854 [conn1] replSet replSetInitiate all members seem up m31100| Fri May 10 08:19:22.854 [conn1] ****** m31100| Fri May 10 08:19:22.854 [conn1] creating replication oplog of size: 40MB... m31100| Fri May 10 08:19:22.854 [FileAllocator] allocating new datafile /data/db/test-rs0-0\local.1, filling with zeroes... m31100| Fri May 10 08:19:23.047 [FileAllocator] done allocating datafile /data/db/test-rs0-0\local.1, size: 64MB, took 0.192 secs m31101| Fri May 10 08:19:23.495 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31101| Fri May 10 08:19:24.495 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31101| Fri May 10 08:19:25.495 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31100| Fri May 10 08:19:26.255 [conn1] ****** m31100| Fri May 10 08:19:26.255 [conn1] replSet info saving a newer config version to local.system.replset m31100| Fri May 10 08:19:26.266 [conn1] replSet saveConfigLocally done m31100| Fri May 10 08:19:26.266 [conn1] replSet replSetInitiate config now saved locally. Should come online in about a minute. m31100| Fri May 10 08:19:26.266 [conn1] command admin.$cmd command: { replSetInitiate: { _id: "test-rs0", members: [ { _id: 0.0, host: "ip-0A420969:31100" }, { _id: 1.0, host: "ip-0A420969:31101" } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:3412682 reslen:112 3416ms m31100| Fri May 10 08:19:26.267 [rsStart] replSet I am ip-0A420969:31100 m31100| Fri May 10 08:19:26.267 [rsStart] replSet STARTUP2 { "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1 } m31100| Fri May 10 08:19:26.268 [rsHealthPoll] replSet member ip-0A420969:31101 is up m31100| Fri May 10 08:19:26.269 [rsSync] replSet SECONDARY m31100| Fri May 10 08:19:26.270 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote m31100| Fri May 10 08:19:26.270 [rsMgr] replSet can't see a majority, will not try to elect self m31100| Fri May 10 08:19:26.271 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes m31101| Fri May 10 08:19:26.495 [rsStart] trying to contact ip-0A420969:31100 m31100| Fri May 10 08:19:26.496 [initandlisten] connection accepted from 10.66.9.105:53269 #3 (2 connections now open) m31101| Fri May 10 08:19:26.498 [initandlisten] connection accepted from 10.66.9.105:53270 #3 (3 connections now open) m31101| Fri May 10 08:19:26.498 [rsStart] replSet I am ip-0A420969:31101 m31101| Fri May 10 08:19:26.499 [conn3] end connection 10.66.9.105:53270 (2 connections now open) m31101| Fri May 10 08:19:26.499 [rsStart] replSet got config version 1 from a remote, saving locally m31101| Fri May 10 08:19:26.499 [rsStart] replSet info saving a newer config version to local.system.replset m31101| Fri May 10 08:19:26.509 [rsStart] replSet saveConfigLocally done m31101| Fri May 10 08:19:26.509 [rsStart] replSet STARTUP2 m31101| Fri May 10 08:19:26.509 [rsMgr] replSet total number of votes is even - add arbiter or give one member an extra vote m31101| Fri May 10 08:19:26.510 [rsSync] ****** m31101| Fri May 10 08:19:26.510 [rsSync] creating replication oplog of size: 40MB... m31101| Fri May 10 08:19:26.510 [FileAllocator] allocating new datafile /data/db/test-rs0-1\local.1, filling with zeroes... m31101| Fri May 10 08:19:26.703 [FileAllocator] done allocating datafile /data/db/test-rs0-1\local.1, size: 64MB, took 0.193 secs m31100| Fri May 10 08:19:28.268 [rsHealthPoll] replset info ip-0A420969:31101 thinks that we are down m31100| Fri May 10 08:19:28.268 [rsHealthPoll] replSet member ip-0A420969:31101 is now in state STARTUP2 m31100| Fri May 10 08:19:28.269 [rsMgr] not electing self, ip-0A420969:31101 would veto with 'I don't think ip-0A420969:31100 is electable' m31101| Fri May 10 08:19:28.338 [rsSync] ****** m31101| Fri May 10 08:19:28.338 [rsSync] replSet initial sync pending m31101| Fri May 10 08:19:28.338 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31101| Fri May 10 08:19:28.499 [rsHealthPoll] replSet member ip-0A420969:31100 is up m31101| Fri May 10 08:19:28.499 [rsHealthPoll] replSet member ip-0A420969:31100 is now in state SECONDARY m31100| Fri May 10 08:19:34.268 [rsMgr] replSet info electSelf 0 m31101| Fri May 10 08:19:34.269 [conn2] replSet RECOVERING m31101| Fri May 10 08:19:34.269 [conn2] replSet info voting yea for ip-0A420969:31100 (0) m31100| Fri May 10 08:19:34.270 [rsMgr] replSet PRIMARY m31101| Fri May 10 08:19:34.499 [rsHealthPoll] replSet member ip-0A420969:31100 is now in state PRIMARY m31100| Fri May 10 08:19:36.268 [rsHealthPoll] replSet member ip-0A420969:31101 is now in state RECOVERING m31101| Fri May 10 08:19:44.338 [rsSync] replSet initial sync pending m31101| Fri May 10 08:19:44.338 [rsSync] replSet syncing to: ip-0A420969:31100 m31100| Fri May 10 08:19:44.339 [initandlisten] connection accepted from 10.66.9.105:53274 #4 (3 connections now open) m31101| Fri May 10 08:19:44.340 [rsSync] build index local.me { _id: 1 } m31101| Fri May 10 08:19:44.341 [rsSync] build index done. scanned 0 total records. 0.001 secs m31101| Fri May 10 08:19:44.342 [rsSync] build index local.replset.minvalid { _id: 1 } m31101| Fri May 10 08:19:44.342 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Fri May 10 08:19:44.343 [rsSync] replSet initial sync drop all databases m31101| Fri May 10 08:19:44.343 [rsSync] dropAllDatabasesExceptLocal 1 m31101| Fri May 10 08:19:44.343 [rsSync] replSet initial sync clone all databases m31101| Fri May 10 08:19:44.344 [rsSync] replSet initial sync data copy, starting syncup m31101| Fri May 10 08:19:44.344 [rsSync] oplog sync 1 of 3 m31101| Fri May 10 08:19:44.344 [rsSync] oplog sync 2 of 3 m31101| Fri May 10 08:19:44.344 [rsSync] replSet initial sync building indexes m31101| Fri May 10 08:19:44.344 [rsSync] oplog sync 3 of 3 m31101| Fri May 10 08:19:44.345 [rsSync] replSet initial sync finishing up m31101| Fri May 10 08:19:44.360 [rsSync] replSet set minValid=518ce5ce:1 m31101| Fri May 10 08:19:44.364 [rsSync] replSet initial sync done m31100| Fri May 10 08:19:44.364 [conn4] end connection 10.66.9.105:53274 (2 connections now open) m31101| Fri May 10 08:19:44.510 [rsBackgroundSync] replSet syncing to: ip-0A420969:31100 m31100| Fri May 10 08:19:44.511 [initandlisten] connection accepted from 10.66.9.105:53277 #5 (3 connections now open) m31101| Fri May 10 08:19:45.344 [rsSyncNotifier] replset setting oplog notifier to ip-0A420969:31100 m31100| Fri May 10 08:19:45.345 [initandlisten] connection accepted from 10.66.9.105:53281 #6 (4 connections now open) m31101| Fri May 10 08:19:45.364 [rsSync] replSet SECONDARY m31100| Fri May 10 08:19:45.550 [FileAllocator] allocating new datafile /data/db/test-rs0-0\admin.ns, filling with zeroes... m31100| Fri May 10 08:19:45.600 [FileAllocator] done allocating datafile /data/db/test-rs0-0\admin.ns, size: 16MB, took 0.048 secs m31100| Fri May 10 08:19:45.600 [FileAllocator] allocating new datafile /data/db/test-rs0-0\admin.0, filling with zeroes... m31100| Fri May 10 08:19:45.649 [FileAllocator] done allocating datafile /data/db/test-rs0-0\admin.0, size: 16MB, took 0.048 secs m31100| Fri May 10 08:19:45.650 [conn1] build index admin.foo { _id: 1 } m31100| Fri May 10 08:19:45.651 [conn1] build index done. scanned 0 total records. 0 secs m31100| Fri May 10 08:19:45.651 [conn1] insert admin.foo ninserted:1 keyUpdates:0 101ms m31101| Fri May 10 08:19:45.653 [FileAllocator] allocating new datafile /data/db/test-rs0-1\admin.ns, filling with zeroes... ReplSetTest awaitReplication: starting: timestamp for primary, ip-0A420969:31100, is Timestamp(1368188385, 1) ReplSetTest awaitReplication: checking secondaries against timestamp Timestamp(1368188385, 1) ReplSetTest awaitReplication: checking secondary #1: ip-0A420969:31101 m31101| Fri May 10 08:19:45.703 [FileAllocator] done allocating datafile /data/db/test-rs0-1\admin.ns, size: 16MB, took 0.049 secs m31101| Fri May 10 08:19:45.704 [FileAllocator] allocating new datafile /data/db/test-rs0-1\admin.0, filling with zeroes... m31101| Fri May 10 08:19:45.753 [FileAllocator] done allocating datafile /data/db/test-rs0-1\admin.0, size: 16MB, took 0.049 secs m31101| Fri May 10 08:19:45.754 [repl writer worker 1] build index admin.foo { _id: 1 } m31101| Fri May 10 08:19:45.755 [repl writer worker 1] build index done. scanned 0 total records. 0 secs ReplSetTest awaitReplication: secondary #1, ip-0A420969:31101, is synced ReplSetTest awaitReplication: finished: all 1 secondaries synced at timestamp Timestamp(1368188385, 1) Fri May 10 08:19:45.760 starting new replica set monitor for replica set test-rs0 with seed of ip-0A420969:31100,ip-0A420969:31101 Fri May 10 08:19:45.761 successfully connected to seed ip-0A420969:31100 for replica set test-rs0 m31100| Fri May 10 08:19:45.762 [initandlisten] connection accepted from 10.66.9.105:53282 #7 (5 connections now open) Fri May 10 08:19:45.762 changing hosts to { 0: "ip-0A420969:31100", 1: "ip-0A420969:31101" } from test-rs0/ Fri May 10 08:19:45.762 trying to add new host ip-0A420969:31100 to replica set test-rs0 Fri May 10 08:19:45.763 successfully connected to new host ip-0A420969:31100 in replica set test-rs0 Fri May 10 08:19:45.763 trying to add new host ip-0A420969:31101 to replica set test-rs0 m31100| Fri May 10 08:19:45.763 [initandlisten] connection accepted from 10.66.9.105:53283 #8 (6 connections now open) Fri May 10 08:19:45.764 successfully connected to new host ip-0A420969:31101 in replica set test-rs0 m31101| Fri May 10 08:19:45.764 [initandlisten] connection accepted from 10.66.9.105:53284 #4 (3 connections now open) Fri May 10 08:19:45.764 Primary for replica set test-rs0 changed to ip-0A420969:31100 m31100| Fri May 10 08:19:45.764 [conn7] end connection 10.66.9.105:53282 (5 connections now open) Fri May 10 08:19:45.765 replica set monitor for replica set test-rs0 started, address is test-rs0/ip-0A420969:31100,ip-0A420969:31101 Fri May 10 08:19:45.765 [ReplicaSetMonitorWatcher] starting Resetting db path '/data/db/test-config0' Fri May 10 08:19:45.767 shell: started program mongod.exe --port 29000 --dbpath /data/db/test-config0 --configsvr --setParameter enableTestCommands=1 m29000| Fri May 10 08:19:45.792 m29000| Fri May 10 08:19:45.792 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability. m29000| Fri May 10 08:19:45.792 m29000| Fri May 10 08:19:45.800 [initandlisten] MongoDB starting : pid=3204 port=29000 dbpath=/data/db/test-config0 master=1 32-bit host=ip-0A420969 m29000| Fri May 10 08:19:45.800 [initandlisten] m29000| Fri May 10 08:19:45.800 [initandlisten] ** NOTE: This is a development version (2.5.0-pre-) of MongoDB. m29000| Fri May 10 08:19:45.800 [initandlisten] ** Not recommended for production. m29000| Fri May 10 08:19:45.800 [initandlisten] m29000| Fri May 10 08:19:45.800 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary. m29000| Fri May 10 08:19:45.800 [initandlisten] ** 32 bit builds are limited to less than 2GB of data (or less with --journal). m29000| Fri May 10 08:19:45.800 [initandlisten] ** See http://dochub.mongodb.org/core/32bit m29000| Fri May 10 08:19:45.800 [initandlisten] m29000| Fri May 10 08:19:45.800 [initandlisten] db version v2.5.0-pre- m29000| Fri May 10 08:19:45.800 [initandlisten] git version: e54e86b4d3c36f6762ff63c03d824b1eaec0c2df m29000| Fri May 10 08:19:45.800 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=0, build=6002, platform=2, service_pack='Service Pack 2') BOOST_LIB_VERSION=1_49 m29000| Fri May 10 08:19:45.800 [initandlisten] allocator: system m29000| Fri May 10 08:19:45.800 [initandlisten] options: { configsvr: true, dbpath: "/data/db/test-config0", port: 29000, setParameter: [ "enableTestCommands=1" ] } m29000| Fri May 10 08:19:45.813 [initandlisten] journal dir=/data/db/test-config0\journal m29000| Fri May 10 08:19:45.813 [initandlisten] recover : no journal files present, no recovery needed m29000| Fri May 10 08:19:46.052 [FileAllocator] allocating new datafile /data/db/test-config0\local.ns, filling with zeroes... m29000| Fri May 10 08:19:46.052 [FileAllocator] creating directory /data/db/test-config0\_tmp m29000| Fri May 10 08:19:46.101 [FileAllocator] done allocating datafile /data/db/test-config0\local.ns, size: 16MB, took 0.048 secs m29000| Fri May 10 08:19:46.101 [FileAllocator] allocating new datafile /data/db/test-config0\local.0, filling with zeroes... m29000| Fri May 10 08:19:46.151 [FileAllocator] done allocating datafile /data/db/test-config0\local.0, size: 16MB, took 0.049 secs m29000| Fri May 10 08:19:46.152 [initandlisten] ****** m29000| Fri May 10 08:19:46.152 [initandlisten] creating replication oplog of size: 5MB... m31100| Fri May 10 08:19:46.268 [rsHealthPoll] replSet member ip-0A420969:31101 is now in state SECONDARY m31100| Fri May 10 08:19:46.347 [slaveTracking] build index local.slaves { _id: 1 } m31100| Fri May 10 08:19:46.348 [slaveTracking] build index done. scanned 0 total records. 0 secs m29000| Fri May 10 08:19:47.428 [initandlisten] ****** m29000| Fri May 10 08:19:47.429 [initandlisten] waiting for connections on port 29000 m29000| Fri May 10 08:19:47.429 [websvr] admin web console waiting for connections on port 30000 m29000| Fri May 10 08:19:47.468 [initandlisten] connection accepted from 127.0.0.1:53286 #1 (1 connection now open) Resetting db path '/data/db/test-config1' Fri May 10 08:19:47.471 shell: started program mongod.exe --port 29001 --dbpath /data/db/test-config1 --configsvr --setParameter enableTestCommands=1 m29001| Fri May 10 08:19:47.495 m29001| Fri May 10 08:19:47.496 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability. m29001| Fri May 10 08:19:47.496 m29001| Fri May 10 08:19:47.504 [initandlisten] MongoDB starting : pid=5384 port=29001 dbpath=/data/db/test-config1 master=1 32-bit host=ip-0A420969 m29001| Fri May 10 08:19:47.504 [initandlisten] m29001| Fri May 10 08:19:47.504 [initandlisten] ** NOTE: This is a development version (2.5.0-pre-) of MongoDB. m29001| Fri May 10 08:19:47.504 [initandlisten] ** Not recommended for production. m29001| Fri May 10 08:19:47.504 [initandlisten] m29001| Fri May 10 08:19:47.504 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary. m29001| Fri May 10 08:19:47.504 [initandlisten] ** 32 bit builds are limited to less than 2GB of data (or less with --journal). m29001| Fri May 10 08:19:47.504 [initandlisten] ** See http://dochub.mongodb.org/core/32bit m29001| Fri May 10 08:19:47.504 [initandlisten] m29001| Fri May 10 08:19:47.504 [initandlisten] db version v2.5.0-pre- m29001| Fri May 10 08:19:47.504 [initandlisten] git version: e54e86b4d3c36f6762ff63c03d824b1eaec0c2df m29001| Fri May 10 08:19:47.504 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=0, build=6002, platform=2, service_pack='Service Pack 2') BOOST_LIB_VERSION=1_49 m29001| Fri May 10 08:19:47.504 [initandlisten] allocator: system m29001| Fri May 10 08:19:47.504 [initandlisten] options: { configsvr: true, dbpath: "/data/db/test-config1", port: 29001, setParameter: [ "enableTestCommands=1" ] } m29001| Fri May 10 08:19:47.781 [initandlisten] journal dir=/data/db/test-config1\journal m29001| Fri May 10 08:19:47.782 [initandlisten] recover : no journal files present, no recovery needed m29001| Fri May 10 08:19:48.019 [FileAllocator] allocating new datafile /data/db/test-config1\local.ns, filling with zeroes... m29001| Fri May 10 08:19:48.019 [FileAllocator] creating directory /data/db/test-config1\_tmp m29001| Fri May 10 08:19:48.068 [FileAllocator] done allocating datafile /data/db/test-config1\local.ns, size: 16MB, took 0.048 secs m29001| Fri May 10 08:19:48.068 [FileAllocator] allocating new datafile /data/db/test-config1\local.0, filling with zeroes... m29001| Fri May 10 08:19:48.120 [FileAllocator] done allocating datafile /data/db/test-config1\local.0, size: 16MB, took 0.05 secs m29001| Fri May 10 08:19:48.121 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 101ms m29001| Fri May 10 08:19:48.121 [initandlisten] ****** m29001| Fri May 10 08:19:48.121 [initandlisten] creating replication oplog of size: 5MB... m29001| Fri May 10 08:19:49.394 [initandlisten] ****** m29001| Fri May 10 08:19:49.396 [initandlisten] waiting for connections on port 29001 m29001| Fri May 10 08:19:49.396 [websvr] admin web console waiting for connections on port 30001 m29001| Fri May 10 08:19:49.668 [initandlisten] connection accepted from 127.0.0.1:53290 #1 (1 connection now open) Resetting db path '/data/db/test-config2' Fri May 10 08:19:49.670 shell: started program mongod.exe --port 29002 --dbpath /data/db/test-config2 --configsvr --setParameter enableTestCommands=1 m29002| Fri May 10 08:19:49.695 m29002| Fri May 10 08:19:49.696 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability. m29002| Fri May 10 08:19:49.696 m29002| Fri May 10 08:19:49.704 [initandlisten] MongoDB starting : pid=4024 port=29002 dbpath=/data/db/test-config2 master=1 32-bit host=ip-0A420969 m29002| Fri May 10 08:19:49.704 [initandlisten] m29002| Fri May 10 08:19:49.704 [initandlisten] ** NOTE: This is a development version (2.5.0-pre-) of MongoDB. m29002| Fri May 10 08:19:49.704 [initandlisten] ** Not recommended for production. m29002| Fri May 10 08:19:49.704 [initandlisten] m29002| Fri May 10 08:19:49.704 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary. m29002| Fri May 10 08:19:49.704 [initandlisten] ** 32 bit builds are limited to less than 2GB of data (or less with --journal). m29002| Fri May 10 08:19:49.704 [initandlisten] ** See http://dochub.mongodb.org/core/32bit m29002| Fri May 10 08:19:49.704 [initandlisten] m29002| Fri May 10 08:19:49.704 [initandlisten] db version v2.5.0-pre- m29002| Fri May 10 08:19:49.704 [initandlisten] git version: e54e86b4d3c36f6762ff63c03d824b1eaec0c2df m29002| Fri May 10 08:19:49.704 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=0, build=6002, platform=2, service_pack='Service Pack 2') BOOST_LIB_VERSION=1_49 m29002| Fri May 10 08:19:49.704 [initandlisten] allocator: system m29002| Fri May 10 08:19:49.704 [initandlisten] options: { configsvr: true, dbpath: "/data/db/test-config2", port: 29002, setParameter: [ "enableTestCommands=1" ] } m29002| Fri May 10 08:19:49.747 [initandlisten] journal dir=/data/db/test-config2\journal m29002| Fri May 10 08:19:49.747 [initandlisten] recover : no journal files present, no recovery needed m29002| Fri May 10 08:19:49.984 [FileAllocator] allocating new datafile /data/db/test-config2\local.ns, filling with zeroes... m29002| Fri May 10 08:19:49.984 [FileAllocator] creating directory /data/db/test-config2\_tmp m29002| Fri May 10 08:19:50.033 [FileAllocator] done allocating datafile /data/db/test-config2\local.ns, size: 16MB, took 0.048 secs m29002| Fri May 10 08:19:50.034 [FileAllocator] allocating new datafile /data/db/test-config2\local.0, filling with zeroes... m29002| Fri May 10 08:19:50.143 [FileAllocator] done allocating datafile /data/db/test-config2\local.0, size: 16MB, took 0.108 secs m29002| Fri May 10 08:19:50.144 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 160ms m29002| Fri May 10 08:19:50.144 [initandlisten] ****** m29002| Fri May 10 08:19:50.144 [initandlisten] creating replication oplog of size: 5MB... m29002| Fri May 10 08:19:50.982 [initandlisten] ****** m29002| Fri May 10 08:19:50.983 [initandlisten] waiting for connections on port 29002 m29002| Fri May 10 08:19:50.983 [websvr] admin web console waiting for connections on port 30002 m29002| Fri May 10 08:19:51.372 [initandlisten] connection accepted from 127.0.0.1:53294 #1 (1 connection now open) "ip-0A420969:29000,ip-0A420969:29001,ip-0A420969:29002" Fri May 10 08:19:51.373 SyncClusterConnection connecting to [ip-0A420969:29000] m29000| Fri May 10 08:19:51.374 [initandlisten] connection accepted from 10.66.9.105:53296 #2 (2 connections now open) Fri May 10 08:19:51.374 SyncClusterConnection connecting to [ip-0A420969:29001] m29001| Fri May 10 08:19:51.375 [initandlisten] connection accepted from 10.66.9.105:53297 #2 (2 connections now open) Fri May 10 08:19:51.375 SyncClusterConnection connecting to [ip-0A420969:29002] m29002| Fri May 10 08:19:51.376 [initandlisten] connection accepted from 10.66.9.105:53298 #2 (2 connections now open) ShardingTest test : { "config" : "ip-0A420969:29000,ip-0A420969:29001,ip-0A420969:29002", "shards" : [ connection to test-rs0/ip-0A420969:31100,ip-0A420969:31101 ] } Fri May 10 08:19:51.378 shell: started program mongos.exe --port 30999 --configdb ip-0A420969:29000,ip-0A420969:29001,ip-0A420969:29002 --chunkSize 50 --setParameter enableTestCommands=1 m30999| Fri May 10 08:19:51.393 [mongosMain] MongoS version 2.5.0-pre- starting: pid=5320 port=30999 32-bit host=ip-0A420969 (--help for usage) m30999| Fri May 10 08:19:51.394 [mongosMain] git version: e54e86b4d3c36f6762ff63c03d824b1eaec0c2df m30999| Fri May 10 08:19:51.394 [mongosMain] build info: windows sys.getwindowsversion(major=6, minor=0, build=6002, platform=2, service_pack='Service Pack 2') BOOST_LIB_VERSION=1_49 m30999| Fri May 10 08:19:51.394 [mongosMain] options: { chunkSize: 50, configdb: "ip-0A420969:29000,ip-0A420969:29001,ip-0A420969:29002", port: 30999, setParameter: [ "enableTestCommands=1" ] } m29000| Fri May 10 08:19:51.397 [initandlisten] connection accepted from 10.66.9.105:53300 #3 (3 connections now open) m29001| Fri May 10 08:19:51.399 [initandlisten] connection accepted from 10.66.9.105:53301 #3 (3 connections now open) m29002| Fri May 10 08:19:51.401 [initandlisten] connection accepted from 10.66.9.105:53302 #3 (3 connections now open) m30999| Fri May 10 08:19:51.402 [mongosMain] SyncClusterConnection connecting to [ip-0A420969:29000] m30999| Fri May 10 08:19:51.403 [mongosMain] SyncClusterConnection connecting to [ip-0A420969:29001] m29000| Fri May 10 08:19:51.403 [initandlisten] connection accepted from 10.66.9.105:53303 #4 (4 connections now open) m29001| Fri May 10 08:19:51.404 [initandlisten] connection accepted from 10.66.9.105:53304 #4 (4 connections now open) m30999| Fri May 10 08:19:51.404 [mongosMain] SyncClusterConnection connecting to [ip-0A420969:29002] m29002| Fri May 10 08:19:51.404 [initandlisten] connection accepted from 10.66.9.105:53305 #4 (4 connections now open) m29000| Fri May 10 08:19:51.406 [conn4] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:51.417 [conn4] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:51.425 [conn4] CMD fsync: sync:1 lock:0 m30999| Fri May 10 08:19:51.435 [mongosMain] scoped connection to ip-0A420969:29000,ip-0A420969:29001,ip-0A420969:29002 not being returned to the pool m29000| Fri May 10 08:19:51.436 [conn4] end connection 10.66.9.105:53303 (3 connections now open) m29002| Fri May 10 08:19:51.436 [conn4] end connection 10.66.9.105:53305 (3 connections now open) m29001| Fri May 10 08:19:51.436 [conn4] end connection 10.66.9.105:53304 (3 connections now open) m29000| Fri May 10 08:19:51.437 [initandlisten] connection accepted from 10.66.9.105:53306 #5 (4 connections now open) m29001| Fri May 10 08:19:51.438 [initandlisten] connection accepted from 10.66.9.105:53307 #5 (4 connections now open) m29002| Fri May 10 08:19:51.440 [initandlisten] connection accepted from 10.66.9.105:53308 #5 (4 connections now open) m30999| Fri May 10 08:19:51.443 [mongosMain] SyncClusterConnection connecting to [ip-0A420969:29000] m30999| Fri May 10 08:19:51.443 [LockPinger] creating distributed lock ping thread for ip-0A420969:29000,ip-0A420969:29001,ip-0A420969:29002 and process ip-0A420969:30999:1368188391:41 (sleeping for 30000ms) m30999| Fri May 10 08:19:51.443 [LockPinger] SyncClusterConnection connecting to [ip-0A420969:29000] m30999| Fri May 10 08:19:51.444 [mongosMain] SyncClusterConnection connecting to [ip-0A420969:29001] m29000| Fri May 10 08:19:51.444 [initandlisten] connection accepted from 10.66.9.105:53309 #6 (5 connections now open) m30999| Fri May 10 08:19:51.445 [LockPinger] SyncClusterConnection connecting to [ip-0A420969:29001] m29000| Fri May 10 08:19:51.445 [initandlisten] connection accepted from 10.66.9.105:53310 #7 (6 connections now open) m29001| Fri May 10 08:19:51.445 [initandlisten] connection accepted from 10.66.9.105:53311 #6 (5 connections now open) m30999| Fri May 10 08:19:51.445 [mongosMain] SyncClusterConnection connecting to [ip-0A420969:29002] m29001| Fri May 10 08:19:51.446 [initandlisten] connection accepted from 10.66.9.105:53312 #7 (6 connections now open) m30999| Fri May 10 08:19:51.446 [LockPinger] SyncClusterConnection connecting to [ip-0A420969:29002] m29002| Fri May 10 08:19:51.446 [initandlisten] connection accepted from 10.66.9.105:53313 #6 (5 connections now open) m29000| Fri May 10 08:19:51.447 [conn6] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:51.448 [conn6] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:51.449 [conn6] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:51.449 [initandlisten] connection accepted from 10.66.9.105:53314 #7 (6 connections now open) m29000| Fri May 10 08:19:51.450 [FileAllocator] allocating new datafile /data/db/test-config0\config.ns, filling with zeroes... m29001| Fri May 10 08:19:51.450 [FileAllocator] allocating new datafile /data/db/test-config1\config.ns, filling with zeroes... m29000| Fri May 10 08:19:51.450 [conn7] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:51.451 [FileAllocator] allocating new datafile /data/db/test-config2\config.ns, filling with zeroes... m29000| Fri May 10 08:19:51.508 [FileAllocator] done allocating datafile /data/db/test-config0\config.ns, size: 16MB, took 0.057 secs m29000| Fri May 10 08:19:51.509 [FileAllocator] allocating new datafile /data/db/test-config0\config.0, filling with zeroes... m29001| Fri May 10 08:19:51.540 [FileAllocator] done allocating datafile /data/db/test-config1\config.ns, size: 16MB, took 0.089 secs m29001| Fri May 10 08:19:51.541 [FileAllocator] allocating new datafile /data/db/test-config1\config.0, filling with zeroes... m29000| Fri May 10 08:19:51.567 [FileAllocator] done allocating datafile /data/db/test-config0\config.0, size: 16MB, took 0.057 secs m29000| Fri May 10 08:19:51.568 [conn6] build index config.locks { _id: 1 } m29000| Fri May 10 08:19:51.569 [conn6] build index done. scanned 0 total records. 0 secs m29000| Fri May 10 08:19:51.570 [conn6] insert config.locks ninserted:1 keyUpdates:0 locks(micros) w:120325 120ms m29000| Fri May 10 08:19:51.570 [FileAllocator] allocating new datafile /data/db/test-config0\config.1, filling with zeroes... m29002| Fri May 10 08:19:51.573 [FileAllocator] done allocating datafile /data/db/test-config2\config.ns, size: 16MB, took 0.122 secs m29002| Fri May 10 08:19:51.574 [FileAllocator] allocating new datafile /data/db/test-config2\config.0, filling with zeroes... m29001| Fri May 10 08:19:51.668 [FileAllocator] done allocating datafile /data/db/test-config1\config.0, size: 16MB, took 0.126 secs m29001| Fri May 10 08:19:51.669 [FileAllocator] allocating new datafile /data/db/test-config1\config.1, filling with zeroes... m29001| Fri May 10 08:19:51.670 [conn6] build index config.locks { _id: 1 } m29001| Fri May 10 08:19:51.670 [conn6] build index done. scanned 0 total records. 0 secs m29001| Fri May 10 08:19:51.671 [conn6] insert config.locks ninserted:1 keyUpdates:0 locks(micros) w:221523 221ms m29002| Fri May 10 08:19:51.696 [FileAllocator] done allocating datafile /data/db/test-config2\config.0, size: 16MB, took 0.121 secs m29002| Fri May 10 08:19:51.698 [conn6] build index config.locks { _id: 1 } m29002| Fri May 10 08:19:51.699 [conn6] build index done. scanned 0 total records. 0 secs m29002| Fri May 10 08:19:51.699 [conn6] insert config.locks ninserted:1 keyUpdates:0 locks(micros) w:249517 249ms m29002| Fri May 10 08:19:51.699 [FileAllocator] allocating new datafile /data/db/test-config2\config.1, filling with zeroes... m29000| Fri May 10 08:19:51.704 [FileAllocator] done allocating datafile /data/db/test-config0\config.1, size: 32MB, took 0.133 secs m29001| Fri May 10 08:19:51.788 [conn6] command admin.$cmd command: { getlasterror: 1, fsync: 1 } ntoreturn:1 keyUpdates:0 reslen:95 117ms m29001| Fri May 10 08:19:51.798 [FileAllocator] done allocating datafile /data/db/test-config1\config.1, size: 32MB, took 0.128 secs m29002| Fri May 10 08:19:51.814 [FileAllocator] done allocating datafile /data/db/test-config2\config.1, size: 32MB, took 0.113 secs m29000| Fri May 10 08:19:51.821 [conn6] CMD fsync: sync:1 lock:0 m29000| Fri May 10 08:19:52.822 [conn7] command admin.$cmd command: { fsync: 1 } ntoreturn:1 keyUpdates:0 locks(micros) W:5996 reslen:51 1372ms m29001| Fri May 10 08:19:52.823 [conn7] CMD fsync: sync:1 lock:0 m29000| Fri May 10 08:19:52.825 [conn6] command admin.$cmd command: { fsync: 1 } ntoreturn:1 keyUpdates:0 locks(micros) W:138 reslen:51 1004ms m29001| Fri May 10 08:19:52.826 [conn6] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:53.248 [conn7] command admin.$cmd command: { fsync: 1 } ntoreturn:1 keyUpdates:0 locks(micros) W:450 reslen:51 425ms m29002| Fri May 10 08:19:53.249 [conn7] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:53.250 [conn6] command admin.$cmd command: { fsync: 1 } ntoreturn:1 keyUpdates:0 locks(micros) W:422862 reslen:51 424ms m29002| Fri May 10 08:19:53.250 [conn6] CMD fsync: sync:1 lock:0 m31101| Fri May 10 08:19:54.268 [conn2] end connection 10.66.9.105:53263 (2 connections now open) m31101| Fri May 10 08:19:54.269 [initandlisten] connection accepted from 10.66.9.105:53319 #5 (3 connections now open) m29002| Fri May 10 08:19:54.517 [conn7] command admin.$cmd command: { fsync: 1 } ntoreturn:1 keyUpdates:0 locks(micros) W:187 reslen:51 1267ms m29001| Fri May 10 08:19:54.519 [conn7] build index config.lockpings { _id: 1 } m29001| Fri May 10 08:19:54.520 [conn7] build index done. scanned 0 total records. 0.001 secs m29002| Fri May 10 08:19:54.518 [conn6] command admin.$cmd command: { fsync: 1 } ntoreturn:1 keyUpdates:0 locks(micros) W:1266116 reslen:51 1267ms m29002| Fri May 10 08:19:54.521 [conn7] build index config.lockpings { _id: 1 } m29000| Fri May 10 08:19:54.519 [conn7] build index config.lockpings { _id: 1 } m29000| Fri May 10 08:19:54.520 [conn7] build index done. scanned 0 total records. 0.001 secs m29002| Fri May 10 08:19:54.522 [conn7] build index done. scanned 0 total records. 0 secs m29000| Fri May 10 08:19:54.860 [conn6] command admin.$cmd command: { getlasterror: 1, fsync: 1 } ntoreturn:1 keyUpdates:0 reslen:113 338ms m29000| Fri May 10 08:19:54.860 [conn7] command admin.$cmd command: { getlasterror: 1, fsync: 1 } ntoreturn:1 keyUpdates:0 reslen:113 340ms m29000| Fri May 10 08:19:54.898 [conn7] CMD fsync: sync:1 lock:0 m29000| Fri May 10 08:19:54.898 [conn6] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:54.920 [conn7] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:54.921 [conn6] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:54.938 [conn7] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:54.940 [conn6] CMD fsync: sync:1 lock:0 m29000| Fri May 10 08:19:55.004 [conn7] CMD fsync: sync:1 lock:0 m30999| Fri May 10 08:19:55.005 [mongosMain] distributed lock 'configUpgrade/ip-0A420969:30999:1368188391:41' acquired, ts : 518ce5e7a1fe09096bd5fd97 m30999| Fri May 10 08:19:55.005 [mongosMain] SyncClusterConnection connecting to [ip-0A420969:29000] m30999| Fri May 10 08:19:55.006 [mongosMain] SyncClusterConnection connecting to [ip-0A420969:29001] m29000| Fri May 10 08:19:55.006 [initandlisten] connection accepted from 10.66.9.105:53321 #8 (7 connections now open) m29001| Fri May 10 08:19:55.007 [initandlisten] connection accepted from 10.66.9.105:53322 #8 (7 connections now open) m30999| Fri May 10 08:19:55.007 [mongosMain] SyncClusterConnection connecting to [ip-0A420969:29002] m29002| Fri May 10 08:19:55.007 [initandlisten] connection accepted from 10.66.9.105:53323 #8 (7 connections now open) m30999| Fri May 10 08:19:55.010 [mongosMain] starting upgrade of config server from v0 to v4 m30999| Fri May 10 08:19:55.010 [mongosMain] starting next upgrade step from v0 to v4 m30999| Fri May 10 08:19:55.010 [mongosMain] about to log new metadata event: { _id: "ip-0A420969-2013-05-10T12:19:55-518ce5eba1fe09096bd5fd98", server: "ip-0A420969", clientAddr: "N/A", time: new Date(1368188395010), what: "starting upgrade of config database", ns: "config.version", details: { from: 0, to: 4 } } m29000| Fri May 10 08:19:55.011 [conn8] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:55.011 [conn7] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:55.013 [conn8] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:55.017 [conn7] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:55.019 [conn8] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:55.023 [conn7] build index config.lockpings { ping: 1 } m29001| Fri May 10 08:19:55.024 [conn7] build index done. scanned 1 total records. 0.001 secs m29002| Fri May 10 08:19:55.025 [conn7] build index config.lockpings { ping: 1 } m29000| Fri May 10 08:19:55.023 [conn7] build index config.lockpings { ping: 1 } m29000| Fri May 10 08:19:55.024 [conn7] build index done. scanned 1 total records. 0.001 secs m29002| Fri May 10 08:19:55.026 [conn7] build index done. scanned 1 total records. 0.001 secs m29000| Fri May 10 08:19:55.031 [conn8] build index config.changelog { _id: 1 } m29000| Fri May 10 08:19:55.032 [conn8] build index done. scanned 0 total records. 0 secs m29001| Fri May 10 08:19:55.033 [conn8] build index config.changelog { _id: 1 } m29001| Fri May 10 08:19:55.033 [conn8] build index done. scanned 0 total records. 0 secs m29002| Fri May 10 08:19:55.034 [conn8] build index config.changelog { _id: 1 } m29002| Fri May 10 08:19:55.035 [conn8] build index done. scanned 0 total records. 0 secs m29000| Fri May 10 08:19:55.141 [conn8] command admin.$cmd command: { getlasterror: 1, fsync: 1 } ntoreturn:1 keyUpdates:0 reslen:95 105ms m29000| Fri May 10 08:19:55.177 [conn8] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:55.201 [conn8] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:55.223 [conn8] CMD fsync: sync:1 lock:0 m30999| Fri May 10 08:19:55.317 [mongosMain] writing initial config version at v4 m29000| Fri May 10 08:19:55.317 [conn8] CMD fsync: sync:1 lock:0 m29000| Fri May 10 08:19:55.713 [conn8] command admin.$cmd command: { fsync: 1 } ntoreturn:1 keyUpdates:0 locks(micros) W:95 reslen:51 395ms m29001| Fri May 10 08:19:55.713 [conn8] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:55.722 [conn8] CMD fsync: sync:1 lock:0 m29000| Fri May 10 08:19:55.731 [conn8] build index config.version { _id: 1 } m29000| Fri May 10 08:19:55.732 [conn8] build index done. scanned 0 total records. 0.001 secs m29001| Fri May 10 08:19:55.731 [conn8] build index config.version { _id: 1 } m29001| Fri May 10 08:19:55.732 [conn8] build index done. scanned 0 total records. 0.001 secs m29002| Fri May 10 08:19:55.733 [conn8] build index config.version { _id: 1 } m29002| Fri May 10 08:19:55.734 [conn8] build index done. scanned 0 total records. 0 secs m30999| Fri May 10 08:19:55.799 [mongosMain] about to log new metadata event: { _id: "ip-0A420969-2013-05-10T12:19:55-518ce5eba1fe09096bd5fd9a", server: "ip-0A420969", clientAddr: "N/A", time: new Date(1368188395798), what: "finished upgrade of config database", ns: "config.version", details: { from: 0, to: 4 } } m29000| Fri May 10 08:19:55.799 [conn8] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:55.815 [conn8] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:55.830 [conn8] CMD fsync: sync:1 lock:0 m30999| Fri May 10 08:19:55.903 [mongosMain] upgrade of config server to v4 successful m29000| Fri May 10 08:19:55.903 [conn6] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:55.913 [conn6] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:55.921 [conn6] CMD fsync: sync:1 lock:0 m30999| Fri May 10 08:19:55.975 [mongosMain] distributed lock 'configUpgrade/ip-0A420969:30999:1368188391:41' unlocked. m29000| Fri May 10 08:19:55.976 [conn8] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:55.984 [conn8] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:55.989 [conn8] CMD fsync: sync:1 lock:0 m29000| Fri May 10 08:19:55.995 [conn8] build index config.settings { _id: 1 } m29000| Fri May 10 08:19:55.996 [conn8] build index done. scanned 0 total records. 0.001 secs m29001| Fri May 10 08:19:55.996 [conn8] build index config.settings { _id: 1 } m29001| Fri May 10 08:19:55.997 [conn8] build index done. scanned 0 total records. 0.001 secs m29002| Fri May 10 08:19:55.997 [conn8] build index config.settings { _id: 1 } m29002| Fri May 10 08:19:55.998 [conn8] build index done. scanned 0 total records. 0 secs m29000| Fri May 10 08:19:56.046 [conn8] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:56.063 [conn8] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:56.078 [conn8] CMD fsync: sync:1 lock:0 m29000| Fri May 10 08:19:56.094 [conn8] build index config.chunks { _id: 1 } m29000| Fri May 10 08:19:56.095 [conn8] build index done. scanned 0 total records. 0 secs m29000| Fri May 10 08:19:56.095 [conn8] info: creating collection config.chunks on add index m29000| Fri May 10 08:19:56.095 [conn8] build index config.chunks { ns: 1, min: 1 } m29000| Fri May 10 08:19:56.096 [conn8] build index done. scanned 0 total records. 0 secs m29002| Fri May 10 08:19:56.097 [conn8] build index config.chunks { _id: 1 } m29002| Fri May 10 08:19:56.098 [conn8] build index done. scanned 0 total records. 0 secs m29002| Fri May 10 08:19:56.098 [conn8] info: creating collection config.chunks on add index m29002| Fri May 10 08:19:56.098 [conn8] build index config.chunks { ns: 1, min: 1 } m29001| Fri May 10 08:19:56.094 [conn8] build index config.chunks { _id: 1 } m29001| Fri May 10 08:19:56.095 [conn8] build index done. scanned 0 total records. 0.001 secs m29001| Fri May 10 08:19:56.095 [conn8] info: creating collection config.chunks on add index m29001| Fri May 10 08:19:56.095 [conn8] build index config.chunks { ns: 1, min: 1 } m29001| Fri May 10 08:19:56.096 [conn8] build index done. scanned 0 total records. 0 secs m29002| Fri May 10 08:19:56.099 [conn8] build index done. scanned 0 total records. 0 secs m29000| Fri May 10 08:19:56.152 [conn8] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:56.172 [conn8] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:56.190 [conn8] CMD fsync: sync:1 lock:0 m29000| Fri May 10 08:19:56.208 [conn8] build index config.chunks { ns: 1, shard: 1, min: 1 } m29000| Fri May 10 08:19:56.209 [conn8] build index done. scanned 0 total records. 0.001 secs m29001| Fri May 10 08:19:56.209 [conn8] build index config.chunks { ns: 1, shard: 1, min: 1 } m29001| Fri May 10 08:19:56.210 [conn8] build index done. scanned 0 total records. 0.001 secs m29002| Fri May 10 08:19:56.210 [conn8] build index config.chunks { ns: 1, shard: 1, min: 1 } m29002| Fri May 10 08:19:56.211 [conn8] build index done. scanned 0 total records. 0 secs m29000| Fri May 10 08:19:56.257 [conn8] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:56.274 [conn8] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:56.288 [conn8] CMD fsync: sync:1 lock:0 m29000| Fri May 10 08:19:56.305 [conn8] build index config.chunks { ns: 1, lastmod: 1 } m29001| Fri May 10 08:19:56.305 [conn8] build index config.chunks { ns: 1, lastmod: 1 } m29002| Fri May 10 08:19:56.306 [conn8] build index config.chunks { ns: 1, lastmod: 1 } m29002| Fri May 10 08:19:56.307 [conn8] build index done. scanned 0 total records. 0 secs m29000| Fri May 10 08:19:56.307 [conn8] build index done. scanned 0 total records. 0.001 secs m29001| Fri May 10 08:19:56.308 [conn8] build index done. scanned 0 total records. 0.002 secs m29000| Fri May 10 08:19:56.469 [conn8] CMD fsync: sync:1 lock:0 m31100| Fri May 10 08:19:56.500 [conn3] end connection 10.66.9.105:53269 (4 connections now open) m31100| Fri May 10 08:19:56.501 [initandlisten] connection accepted from 10.66.9.105:53330 #9 (5 connections now open) m29000| Fri May 10 08:19:56.721 [conn8] command admin.$cmd command: { fsync: 1 } ntoreturn:1 keyUpdates:0 locks(micros) W:109 reslen:51 252ms m29001| Fri May 10 08:19:56.722 [conn8] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:56.736 [conn8] CMD fsync: sync:1 lock:0 m29000| Fri May 10 08:19:56.751 [conn8] build index config.shards { _id: 1 } m29000| Fri May 10 08:19:56.752 [conn8] build index done. scanned 0 total records. 0 secs m29000| Fri May 10 08:19:56.752 [conn8] info: creating collection config.shards on add index m29000| Fri May 10 08:19:56.752 [conn8] build index config.shards { host: 1 } m29000| Fri May 10 08:19:56.753 [conn8] build index done. scanned 0 total records. 0 secs m29002| Fri May 10 08:19:56.754 [conn8] build index config.shards { _id: 1 } m29001| Fri May 10 08:19:56.751 [conn8] build index config.shards { _id: 1 } m29001| Fri May 10 08:19:56.752 [conn8] build index done. scanned 0 total records. 0.001 secs m29001| Fri May 10 08:19:56.752 [conn8] info: creating collection config.shards on add index m29001| Fri May 10 08:19:56.752 [conn8] build index config.shards { host: 1 } m29001| Fri May 10 08:19:56.753 [conn8] build index done. scanned 0 total records. 0.001 secs m29002| Fri May 10 08:19:56.755 [conn8] build index done. scanned 0 total records. 0.001 secs m29002| Fri May 10 08:19:56.755 [conn8] info: creating collection config.shards on add index m29002| Fri May 10 08:19:56.755 [conn8] build index config.shards { host: 1 } m29002| Fri May 10 08:19:56.756 [conn8] build index done. scanned 0 total records. 0.001 secs m30999| Fri May 10 08:19:56.847 [websvr] admin web console waiting for connections on port 31999 m30999| Fri May 10 08:19:56.847 [mongosMain] waiting for connections on port 30999 m30999| Fri May 10 08:19:56.847 [Balancer] about to contact config servers and shards m30999| Fri May 10 08:19:56.848 [Balancer] config servers and shards contacted successfully m30999| Fri May 10 08:19:56.848 [Balancer] balancer id: ip-0A420969:30999 started at May 10 08:19:56 m29000| Fri May 10 08:19:56.849 [conn8] build index config.mongos { _id: 1 } m29000| Fri May 10 08:19:56.850 [conn8] build index done. scanned 0 total records. 0.001 secs m29001| Fri May 10 08:19:56.849 [conn8] build index config.mongos { _id: 1 } m29001| Fri May 10 08:19:56.850 [conn8] build index done. scanned 0 total records. 0.001 secs m29002| Fri May 10 08:19:56.851 [conn8] build index config.mongos { _id: 1 } m29000| Fri May 10 08:19:56.851 [conn6] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:56.852 [conn8] build index done. scanned 0 total records. 0 secs m29001| Fri May 10 08:19:56.882 [conn6] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:56.911 [conn6] CMD fsync: sync:1 lock:0 m29000| Fri May 10 08:19:57.021 [conn6] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:57.032 [conn6] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:57.040 [conn6] CMD fsync: sync:1 lock:0 m29000| Fri May 10 08:19:57.128 [conn6] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:57.137 [conn6] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:57.145 [conn6] CMD fsync: sync:1 lock:0 m30999| Fri May 10 08:19:57.173 [mongosMain] connection accepted from 127.0.0.1:53329 #1 (1 connection now open) ShardingTest undefined going to add shard : test-rs0/ip-0A420969:31100,ip-0A420969:31101 m30999| Fri May 10 08:19:57.175 [conn1] couldn't find database [admin] in config db m29000| Fri May 10 08:19:57.176 [conn7] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:57.184 [conn7] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:57.194 [conn7] CMD fsync: sync:1 lock:0 m30999| Fri May 10 08:19:57.197 [Balancer] distributed lock 'balancer/ip-0A420969:30999:1368188391:41' acquired, ts : 518ce5eda1fe09096bd5fd9c m29000| Fri May 10 08:19:57.198 [conn6] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:57.203 [conn6] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:57.204 [conn6] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:57.207 [conn7] build index config.databases { _id: 1 } m29001| Fri May 10 08:19:57.208 [conn7] build index done. scanned 0 total records. 0.001 secs m29000| Fri May 10 08:19:57.206 [conn7] build index config.databases { _id: 1 } m29000| Fri May 10 08:19:57.208 [conn7] build index done. scanned 0 total records. 0.001 secs m29002| Fri May 10 08:19:57.209 [conn7] build index config.databases { _id: 1 } m29002| Fri May 10 08:19:57.210 [conn7] build index done. scanned 0 total records. 0.001 secs m30999| Fri May 10 08:19:57.303 [Balancer] distributed lock 'balancer/ip-0A420969:30999:1368188391:41' unlocked. m30999| Fri May 10 08:19:57.303 [conn1] put [admin] on: config:ip-0A420969:29000,ip-0A420969:29001,ip-0A420969:29002 m30999| Fri May 10 08:19:57.303 [conn1] starting new replica set monitor for replica set test-rs0 with seed of ip-0A420969:31100,ip-0A420969:31101 m30999| Fri May 10 08:19:57.304 [conn1] successfully connected to seed ip-0A420969:31100 for replica set test-rs0 m31100| Fri May 10 08:19:57.305 [initandlisten] connection accepted from 10.66.9.105:53333 #10 (6 connections now open) m30999| Fri May 10 08:19:57.305 [conn1] changing hosts to { 0: "ip-0A420969:31100", 1: "ip-0A420969:31101" } from test-rs0/ m30999| Fri May 10 08:19:57.305 [conn1] trying to add new host ip-0A420969:31100 to replica set test-rs0 m30999| Fri May 10 08:19:57.306 [conn1] successfully connected to new host ip-0A420969:31100 in replica set test-rs0 m31100| Fri May 10 08:19:57.306 [initandlisten] connection accepted from 10.66.9.105:53334 #11 (7 connections now open) m30999| Fri May 10 08:19:57.306 [conn1] trying to add new host ip-0A420969:31101 to replica set test-rs0 m30999| Fri May 10 08:19:57.307 [conn1] successfully connected to new host ip-0A420969:31101 in replica set test-rs0 m31101| Fri May 10 08:19:57.307 [initandlisten] connection accepted from 10.66.9.105:53335 #6 (4 connections now open) m31100| Fri May 10 08:19:57.307 [conn10] end connection 10.66.9.105:53333 (6 connections now open) m30999| Fri May 10 08:19:57.307 [conn1] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:19:57.308 [conn1] replica set monitor for replica set test-rs0 started, address is test-rs0/ip-0A420969:31100,ip-0A420969:31101 m30999| Fri May 10 08:19:57.308 [ReplicaSetMonitorWatcher] starting m31100| Fri May 10 08:19:57.309 [initandlisten] connection accepted from 10.66.9.105:53336 #12 (7 connections now open) m30999| Fri May 10 08:19:57.312 [conn1] going to add shard: { _id: "test-rs0", host: "test-rs0/ip-0A420969:31100,ip-0A420969:31101" } m29000| Fri May 10 08:19:57.312 [conn8] CMD fsync: sync:1 lock:0 m29000| Fri May 10 08:19:57.741 [conn8] command admin.$cmd command: { fsync: 1 } ntoreturn:1 keyUpdates:0 locks(micros) W:19064 reslen:51 429ms m29001| Fri May 10 08:19:57.741 [conn8] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:57.758 [conn8] CMD fsync: sync:1 lock:0 { "shardAdded" : "test-rs0", "ok" : 1 } m30999| Fri May 10 08:19:58.540 [conn1] creating WriteBackListener for: ip-0A420969:31100 serverID: 518ce5eca1fe09096bd5fd9b m30999| Fri May 10 08:19:58.540 [conn1] creating WriteBackListener for: ip-0A420969:31101 serverID: 518ce5eca1fe09096bd5fd9b m31100| Fri May 10 08:19:58.542 [initandlisten] connection accepted from 10.66.9.105:53337 #13 (8 connections now open) m31101| Fri May 10 08:19:58.542 [initandlisten] connection accepted from 10.66.9.105:53338 #7 (5 connections now open) m31100| Fri May 10 08:19:58.542 [initandlisten] connection accepted from 10.66.9.105:53339 #14 (9 connections now open) m31100| Fri May 10 08:19:58.542 [conn13] first cluster operation detected, adding sharding hook to enable versioning and authentication to remote servers m30999| Fri May 10 08:19:58.543 [conn1] SyncClusterConnection connecting to [ip-0A420969:29000] m30999| Fri May 10 08:19:58.544 [conn1] SyncClusterConnection connecting to [ip-0A420969:29001] m29000| Fri May 10 08:19:58.544 [initandlisten] connection accepted from 10.66.9.105:53340 #9 (8 connections now open) m29001| Fri May 10 08:19:58.545 [initandlisten] connection accepted from 10.66.9.105:53341 #9 (8 connections now open) m30999| Fri May 10 08:19:58.544 [conn1] SyncClusterConnection connecting to [ip-0A420969:29002] m29002| Fri May 10 08:19:58.545 [initandlisten] connection accepted from 10.66.9.105:53342 #9 (8 connections now open) m29000| Fri May 10 08:19:58.545 [conn9] CMD fsync: sync:1 lock:0 m29000| Fri May 10 08:19:58.711 [conn9] command admin.$cmd command: { fsync: 1 } ntoreturn:1 keyUpdates:0 locks(micros) W:92 reslen:51 165ms m29001| Fri May 10 08:19:58.712 [conn9] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:58.722 [conn9] CMD fsync: sync:1 lock:0 Waiting for active hosts... Waiting for the balancer lock... Waiting again for active hosts after balancer is off... ---- Creating new connections... ---- m30999| Fri May 10 08:19:58.766 [mongosMain] connection accepted from 10.66.9.105:53343 #2 (2 connections now open) m30999| Fri May 10 08:19:58.767 [conn2] couldn't find database [foo] in config db m29000| Fri May 10 08:19:58.769 [conn8] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:19:58.779 [conn8] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:19:58.787 [conn8] CMD fsync: sync:1 lock:0 m30999| Fri May 10 08:19:58.831 [conn2] put [foo] on: test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101 m31100| Fri May 10 08:19:58.832 [initandlisten] connection accepted from 10.66.9.105:53344 #15 (10 connections now open) m30999| Fri May 10 08:19:58.835 [mongosMain] connection accepted from 10.66.9.105:53345 #3 (3 connections now open) m31100| Fri May 10 08:19:58.836 [initandlisten] connection accepted from 10.66.9.105:53346 #16 (11 connections now open) m30999| Fri May 10 08:19:58.838 [mongosMain] connection accepted from 10.66.9.105:53347 #4 (4 connections now open) m31100| Fri May 10 08:19:58.839 [initandlisten] connection accepted from 10.66.9.105:53348 #17 (12 connections now open) m30999| Fri May 10 08:19:58.841 [mongosMain] connection accepted from 10.66.9.105:53349 #5 (5 connections now open) m31100| Fri May 10 08:19:58.842 [initandlisten] connection accepted from 10.66.9.105:53350 #18 (13 connections now open) m30999| Fri May 10 08:19:58.843 [mongosMain] connection accepted from 10.66.9.105:53351 #6 (6 connections now open) m31100| Fri May 10 08:19:58.844 [initandlisten] connection accepted from 10.66.9.105:53352 #19 (14 connections now open) m30999| Fri May 10 08:19:58.845 [mongosMain] connection accepted from 10.66.9.105:53353 #7 (7 connections now open) m31100| Fri May 10 08:19:58.846 [initandlisten] connection accepted from 10.66.9.105:53354 #20 (15 connections now open) m30999| Fri May 10 08:19:58.848 [mongosMain] connection accepted from 10.66.9.105:53355 #8 (8 connections now open) m31100| Fri May 10 08:19:58.849 [initandlisten] connection accepted from 10.66.9.105:53356 #21 (16 connections now open) m30999| Fri May 10 08:19:58.851 [mongosMain] connection accepted from 10.66.9.105:53357 #9 (9 connections now open) m31100| Fri May 10 08:19:58.852 [initandlisten] connection accepted from 10.66.9.105:53358 #22 (17 connections now open) m30999| Fri May 10 08:19:58.854 [mongosMain] connection accepted from 10.66.9.105:53359 #10 (10 connections now open) m31100| Fri May 10 08:19:58.856 [initandlisten] connection accepted from 10.66.9.105:53360 #23 (18 connections now open) m30999| Fri May 10 08:19:58.857 [mongosMain] connection accepted from 10.66.9.105:53361 #11 (11 connections now open) m31100| Fri May 10 08:19:58.858 [initandlisten] connection accepted from 10.66.9.105:53362 #24 (19 connections now open) m30999| Fri May 10 08:19:58.859 [mongosMain] connection accepted from 10.66.9.105:53363 #12 (12 connections now open) m31100| Fri May 10 08:19:58.861 [initandlisten] connection accepted from 10.66.9.105:53364 #25 (20 connections now open) m30999| Fri May 10 08:19:58.862 [mongosMain] connection accepted from 10.66.9.105:53365 #13 (13 connections now open) m31100| Fri May 10 08:19:58.864 [initandlisten] connection accepted from 10.66.9.105:53366 #26 (21 connections now open) m30999| Fri May 10 08:19:58.865 [mongosMain] connection accepted from 10.66.9.105:53367 #14 (14 connections now open) m31100| Fri May 10 08:19:58.867 [initandlisten] connection accepted from 10.66.9.105:53368 #27 (22 connections now open) m30999| Fri May 10 08:19:58.868 [mongosMain] connection accepted from 10.66.9.105:53369 #15 (15 connections now open) m31100| Fri May 10 08:19:58.869 [initandlisten] connection accepted from 10.66.9.105:53370 #28 (23 connections now open) m30999| Fri May 10 08:19:58.871 [mongosMain] connection accepted from 10.66.9.105:53371 #16 (16 connections now open) m31100| Fri May 10 08:19:58.872 [initandlisten] connection accepted from 10.66.9.105:53372 #29 (24 connections now open) m30999| Fri May 10 08:19:58.874 [mongosMain] connection accepted from 10.66.9.105:53373 #17 (17 connections now open) m31100| Fri May 10 08:19:58.875 [initandlisten] connection accepted from 10.66.9.105:53374 #30 (25 connections now open) m30999| Fri May 10 08:19:58.877 [mongosMain] connection accepted from 10.66.9.105:53375 #18 (18 connections now open) m31100| Fri May 10 08:19:58.879 [initandlisten] connection accepted from 10.66.9.105:53376 #31 (26 connections now open) m30999| Fri May 10 08:19:58.880 [mongosMain] connection accepted from 10.66.9.105:53377 #19 (19 connections now open) m31100| Fri May 10 08:19:58.881 [initandlisten] connection accepted from 10.66.9.105:53378 #32 (27 connections now open) m30999| Fri May 10 08:19:58.882 [mongosMain] connection accepted from 10.66.9.105:53379 #20 (20 connections now open) m31100| Fri May 10 08:19:58.883 [initandlisten] connection accepted from 10.66.9.105:53380 #33 (28 connections now open) m30999| Fri May 10 08:19:58.885 [mongosMain] connection accepted from 10.66.9.105:53381 #21 (21 connections now open) m31100| Fri May 10 08:19:58.886 [initandlisten] connection accepted from 10.66.9.105:53382 #34 (29 connections now open) m30999| Fri May 10 08:19:58.887 [mongosMain] connection accepted from 10.66.9.105:53383 #22 (22 connections now open) m31100| Fri May 10 08:19:58.888 [initandlisten] connection accepted from 10.66.9.105:53384 #35 (30 connections now open) m30999| Fri May 10 08:19:58.889 [mongosMain] connection accepted from 10.66.9.105:53385 #23 (23 connections now open) m31100| Fri May 10 08:19:58.890 [initandlisten] connection accepted from 10.66.9.105:53386 #36 (31 connections now open) m30999| Fri May 10 08:19:58.892 [mongosMain] connection accepted from 10.66.9.105:53387 #24 (24 connections now open) m31100| Fri May 10 08:19:58.893 [initandlisten] connection accepted from 10.66.9.105:53388 #37 (32 connections now open) m30999| Fri May 10 08:19:58.894 [mongosMain] connection accepted from 10.66.9.105:53389 #25 (25 connections now open) m31100| Fri May 10 08:19:58.895 [initandlisten] connection accepted from 10.66.9.105:53390 #38 (33 connections now open) m30999| Fri May 10 08:19:58.896 [mongosMain] connection accepted from 10.66.9.105:53391 #26 (26 connections now open) m31100| Fri May 10 08:19:58.897 [initandlisten] connection accepted from 10.66.9.105:53392 #39 (34 connections now open) m30999| Fri May 10 08:19:58.898 [mongosMain] connection accepted from 10.66.9.105:53393 #27 (27 connections now open) m31100| Fri May 10 08:19:58.900 [initandlisten] connection accepted from 10.66.9.105:53394 #40 (35 connections now open) m30999| Fri May 10 08:19:58.901 [mongosMain] connection accepted from 10.66.9.105:53395 #28 (28 connections now open) m31100| Fri May 10 08:19:58.902 [initandlisten] connection accepted from 10.66.9.105:53396 #41 (36 connections now open) m30999| Fri May 10 08:19:58.903 [mongosMain] connection accepted from 10.66.9.105:53397 #29 (29 connections now open) m31100| Fri May 10 08:19:58.904 [initandlisten] connection accepted from 10.66.9.105:53398 #42 (37 connections now open) m30999| Fri May 10 08:19:58.906 [mongosMain] connection accepted from 10.66.9.105:53399 #30 (30 connections now open) m31100| Fri May 10 08:19:58.907 [initandlisten] connection accepted from 10.66.9.105:53400 #43 (38 connections now open) m30999| Fri May 10 08:19:58.909 [mongosMain] connection accepted from 10.66.9.105:53401 #31 (31 connections now open) m31100| Fri May 10 08:19:58.910 [initandlisten] connection accepted from 10.66.9.105:53402 #44 (39 connections now open) m30999| Fri May 10 08:19:58.912 [mongosMain] connection accepted from 10.66.9.105:53403 #32 (32 connections now open) m31100| Fri May 10 08:19:58.913 [initandlisten] connection accepted from 10.66.9.105:53404 #45 (40 connections now open) m30999| Fri May 10 08:19:58.915 [mongosMain] connection accepted from 10.66.9.105:53405 #33 (33 connections now open) m31100| Fri May 10 08:19:58.916 [initandlisten] connection accepted from 10.66.9.105:53406 #46 (41 connections now open) m30999| Fri May 10 08:19:58.918 [mongosMain] connection accepted from 10.66.9.105:53407 #34 (34 connections now open) m31100| Fri May 10 08:19:58.919 [initandlisten] connection accepted from 10.66.9.105:53408 #47 (42 connections now open) m30999| Fri May 10 08:19:58.921 [mongosMain] connection accepted from 10.66.9.105:53409 #35 (35 connections now open) m31100| Fri May 10 08:19:58.922 [initandlisten] connection accepted from 10.66.9.105:53410 #48 (43 connections now open) m30999| Fri May 10 08:19:58.924 [mongosMain] connection accepted from 10.66.9.105:53411 #36 (36 connections now open) m31100| Fri May 10 08:19:58.925 [initandlisten] connection accepted from 10.66.9.105:53412 #49 (44 connections now open) m30999| Fri May 10 08:19:58.927 [mongosMain] connection accepted from 10.66.9.105:53413 #37 (37 connections now open) m31100| Fri May 10 08:19:58.928 [initandlisten] connection accepted from 10.66.9.105:53414 #50 (45 connections now open) m30999| Fri May 10 08:19:58.930 [mongosMain] connection accepted from 10.66.9.105:53415 #38 (38 connections now open) m31100| Fri May 10 08:19:58.931 [initandlisten] connection accepted from 10.66.9.105:53416 #51 (46 connections now open) m30999| Fri May 10 08:19:58.933 [mongosMain] connection accepted from 10.66.9.105:53417 #39 (39 connections now open) m31100| Fri May 10 08:19:58.934 [initandlisten] connection accepted from 10.66.9.105:53418 #52 (47 connections now open) m30999| Fri May 10 08:19:58.936 [mongosMain] connection accepted from 10.66.9.105:53419 #40 (40 connections now open) m31100| Fri May 10 08:19:58.937 [initandlisten] connection accepted from 10.66.9.105:53420 #53 (48 connections now open) m30999| Fri May 10 08:19:58.939 [mongosMain] connection accepted from 10.66.9.105:53421 #41 (41 connections now open) m31100| Fri May 10 08:19:58.940 [initandlisten] connection accepted from 10.66.9.105:53422 #54 (49 connections now open) m30999| Fri May 10 08:19:58.942 [mongosMain] connection accepted from 10.66.9.105:53423 #42 (42 connections now open) m31100| Fri May 10 08:19:58.943 [initandlisten] connection accepted from 10.66.9.105:53424 #55 (50 connections now open) m30999| Fri May 10 08:19:58.945 [mongosMain] connection accepted from 10.66.9.105:53425 #43 (43 connections now open) m31100| Fri May 10 08:19:58.946 [initandlisten] connection accepted from 10.66.9.105:53426 #56 (51 connections now open) m30999| Fri May 10 08:19:58.948 [mongosMain] connection accepted from 10.66.9.105:53427 #44 (44 connections now open) m31100| Fri May 10 08:19:58.949 [initandlisten] connection accepted from 10.66.9.105:53428 #57 (52 connections now open) m30999| Fri May 10 08:19:58.951 [mongosMain] connection accepted from 10.66.9.105:53429 #45 (45 connections now open) m31100| Fri May 10 08:19:58.952 [initandlisten] connection accepted from 10.66.9.105:53430 #58 (53 connections now open) m30999| Fri May 10 08:19:58.954 [mongosMain] connection accepted from 10.66.9.105:53431 #46 (46 connections now open) m31100| Fri May 10 08:19:58.955 [initandlisten] connection accepted from 10.66.9.105:53432 #59 (54 connections now open) m30999| Fri May 10 08:19:58.956 [mongosMain] connection accepted from 10.66.9.105:53433 #47 (47 connections now open) m31100| Fri May 10 08:19:58.958 [initandlisten] connection accepted from 10.66.9.105:53434 #60 (55 connections now open) m30999| Fri May 10 08:19:58.959 [mongosMain] connection accepted from 10.66.9.105:53435 #48 (48 connections now open) m31100| Fri May 10 08:19:58.961 [initandlisten] connection accepted from 10.66.9.105:53436 #61 (56 connections now open) m30999| Fri May 10 08:19:58.962 [mongosMain] connection accepted from 10.66.9.105:53437 #49 (49 connections now open) m31100| Fri May 10 08:19:58.964 [initandlisten] connection accepted from 10.66.9.105:53438 #62 (57 connections now open) m30999| Fri May 10 08:19:58.965 [mongosMain] connection accepted from 10.66.9.105:53439 #50 (50 connections now open) m31100| Fri May 10 08:19:58.967 [initandlisten] connection accepted from 10.66.9.105:53440 #63 (58 connections now open) m30999| Fri May 10 08:19:58.968 [mongosMain] connection accepted from 10.66.9.105:53441 #51 (51 connections now open) m31100| Fri May 10 08:19:58.970 [initandlisten] connection accepted from 10.66.9.105:53442 #64 (59 connections now open) ---- Returning the connections back to the pool. ---- m30999| Fri May 10 08:19:58.976 [conn35] end connection 10.66.9.105:53409 (50 connections now open) m30999| Fri May 10 08:19:58.977 [conn36] end connection 10.66.9.105:53411 (49 connections now open) m30999| Fri May 10 08:19:58.977 [conn37] end connection 10.66.9.105:53413 (48 connections now open) m30999| Fri May 10 08:19:58.977 [conn38] end connection 10.66.9.105:53415 (47 connections now open) m30999| Fri May 10 08:19:58.977 [conn39] end connection 10.66.9.105:53417 (46 connections now open) m30999| Fri May 10 08:19:58.977 [conn40] end connection 10.66.9.105:53419 (45 connections now open) m30999| Fri May 10 08:19:58.977 [conn41] end connection 10.66.9.105:53421 (44 connections now open) m30999| Fri May 10 08:19:58.977 [conn42] end connection 10.66.9.105:53423 (43 connections now open) m30999| Fri May 10 08:19:58.977 [conn43] end connection 10.66.9.105:53425 (42 connections now open) m30999| Fri May 10 08:19:58.978 [conn44] end connection 10.66.9.105:53427 (41 connections now open) m30999| Fri May 10 08:19:58.978 [conn45] end connection 10.66.9.105:53429 (40 connections now open) m30999| Fri May 10 08:19:58.978 [conn46] end connection 10.66.9.105:53431 (39 connections now open) m30999| Fri May 10 08:19:58.978 [conn47] end connection 10.66.9.105:53433 (38 connections now open) m30999| Fri May 10 08:19:58.978 [conn48] end connection 10.66.9.105:53435 (37 connections now open) m30999| Fri May 10 08:19:58.979 [conn49] end connection 10.66.9.105:53437 (36 connections now open) m30999| Fri May 10 08:19:58.979 [conn50] end connection 10.66.9.105:53439 (35 connections now open) m30999| Fri May 10 08:19:58.979 [conn51] end connection 10.66.9.105:53441 (34 connections now open) m30999| Fri May 10 08:19:58.979 [conn2] end connection 10.66.9.105:53343 (33 connections now open) m30999| Fri May 10 08:19:58.979 [conn3] end connection 10.66.9.105:53345 (32 connections now open) m30999| Fri May 10 08:19:58.979 [conn4] end connection 10.66.9.105:53347 (31 connections now open) m30999| Fri May 10 08:19:58.979 [conn5] end connection 10.66.9.105:53349 (30 connections now open) m30999| Fri May 10 08:19:58.979 [conn6] end connection 10.66.9.105:53351 (29 connections now open) m30999| Fri May 10 08:19:58.980 [conn7] end connection 10.66.9.105:53353 (28 connections now open) m30999| Fri May 10 08:19:58.980 [conn8] end connection 10.66.9.105:53355 (27 connections now open) m30999| Fri May 10 08:19:58.980 [conn9] end connection 10.66.9.105:53357 (26 connections now open) m30999| Fri May 10 08:19:58.980 [conn10] end connection 10.66.9.105:53359 (25 connections now open) m30999| Fri May 10 08:19:58.980 [conn11] end connection 10.66.9.105:53361 (24 connections now open) m30999| Fri May 10 08:19:58.980 [conn12] end connection 10.66.9.105:53363 (23 connections now open) m30999| Fri May 10 08:19:58.980 [conn13] end connection 10.66.9.105:53365 (22 connections now open) m30999| Fri May 10 08:19:58.981 [conn14] end connection 10.66.9.105:53367 (21 connections now open) m30999| Fri May 10 08:19:58.981 [conn15] end connection 10.66.9.105:53369 (20 connections now open) m30999| Fri May 10 08:19:58.981 [conn16] end connection 10.66.9.105:53371 (19 connections now open) m30999| Fri May 10 08:19:58.981 [conn17] end connection 10.66.9.105:53373 (18 connections now open) m30999| Fri May 10 08:19:58.981 [conn18] end connection 10.66.9.105:53375 (17 connections now open) m30999| Fri May 10 08:19:58.981 [conn19] end connection 10.66.9.105:53377 (16 connections now open) m30999| Fri May 10 08:19:58.981 [conn20] end connection 10.66.9.105:53379 (15 connections now open) m30999| Fri May 10 08:19:58.982 [conn21] end connection 10.66.9.105:53381 (14 connections now open) m30999| Fri May 10 08:19:58.982 [conn22] end connection 10.66.9.105:53383 (13 connections now open) m30999| Fri May 10 08:19:58.982 [conn23] end connection 10.66.9.105:53385 (12 connections now open) m30999| Fri May 10 08:19:58.982 [conn24] end connection 10.66.9.105:53387 (11 connections now open) m30999| Fri May 10 08:19:58.982 [conn25] end connection 10.66.9.105:53389 (10 connections now open) m30999| Fri May 10 08:19:58.982 [conn26] end connection 10.66.9.105:53391 (9 connections now open) m30999| Fri May 10 08:19:58.983 [conn27] end connection 10.66.9.105:53393 (8 connections now open) m30999| Fri May 10 08:19:58.983 [conn28] end connection 10.66.9.105:53395 (7 connections now open) m30999| Fri May 10 08:19:58.983 [conn29] end connection 10.66.9.105:53397 (6 connections now open) m30999| Fri May 10 08:19:58.983 [conn30] end connection 10.66.9.105:53399 (5 connections now open) m30999| Fri May 10 08:19:58.983 [conn31] end connection 10.66.9.105:53401 (4 connections now open) m30999| Fri May 10 08:19:58.983 [conn32] end connection 10.66.9.105:53403 (3 connections now open) m30999| Fri May 10 08:19:58.984 [conn33] end connection 10.66.9.105:53405 (2 connections now open) m30999| Fri May 10 08:19:58.984 [conn34] end connection 10.66.9.105:53407 (1 connection now open) { "hosts" : { "ip-0A420969:29000,ip-0A420969:29001,ip-0A420969:29002::0" : { "available" : 0, "created" : 1 }, "test-rs0/ip-0A420969:31100,ip-0A420969:31101::0" : { "available" : 50, "created" : 51 } }, "replicaSets" : { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } }, "createdByType" : { "set" : 51, "sync" : 1 }, "totalAvailable" : 50, "totalCreated" : 52, "threads" : [ { "hosts" : [ { "host" : "ip-0A420969:29000,ip-0A420969:29001,ip-0A420969:29002", "created" : NumberLong(1), "avail" : true }, { "host" : "test-rs0/ip-0A420969:31100,ip-0A420969:31101", "created" : NumberLong(1), "avail" : true } ], "seenNS" : [ "config.locks", "config.mongos", "config.settings" ] } ], "ok" : 1 } ---- Stepdown primary and then step back up... ---- m31100| Fri May 10 08:19:59.680 [conn1] replSet info stepping down as primary secs=60 m31100| Fri May 10 08:19:59.680 [conn1] replSet relinquishing primary state m31100| Fri May 10 08:19:59.680 [conn1] replSet SECONDARY m31100| Fri May 10 08:19:59.680 [conn1] replSet closing client sockets after relinquishing primary m31100| Fri May 10 08:19:59.680 [conn1] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:53260] Fri May 10 08:19:59.681 DBClientCursor::init call() failed Error: error doing query: failed ---- Waiting for mongos to acknowledge stepdown... ---- Awaiting ip-0A420969:31101 to be { "ismaster" : true } for connection to ip-0A420969:30999 (rs: test-rs0) m31100| Fri May 10 08:19:59.681 [conn6] end connection 10.66.9.105:53281 (57 connections now open) m31100| Fri May 10 08:19:59.681 [conn12] end connection 10.66.9.105:53336 (56 connections now open) m31100| Fri May 10 08:19:59.681 [conn16] end connection 10.66.9.105:53346 (55 connections now open) m31100| Fri May 10 08:19:59.681 [conn18] end connection 10.66.9.105:53350 (54 connections now open) m31100| Fri May 10 08:19:59.682 [conn20] end connection 10.66.9.105:53354 (53 connections now open) m31100| Fri May 10 08:19:59.682 [conn22] end connection 10.66.9.105:53358 (52 connections now open) m31100| Fri May 10 08:19:59.682 [conn24] end connection 10.66.9.105:53362 (51 connections now open) m31100| Fri May 10 08:19:59.682 [conn26] end connection 10.66.9.105:53366 (50 connections now open) m31100| Fri May 10 08:19:59.682 [conn28] end connection 10.66.9.105:53370 (49 connections now open) m31100| Fri May 10 08:19:59.682 [conn30] end connection 10.66.9.105:53374 (48 connections now open) m31100| Fri May 10 08:19:59.682 [conn32] end connection 10.66.9.105:53378 (47 connections now open) m31100| Fri May 10 08:19:59.682 [conn34] end connection 10.66.9.105:53382 (46 connections now open) m31100| Fri May 10 08:19:59.682 [conn36] end connection 10.66.9.105:53386 (45 connections now open) m31100| Fri May 10 08:19:59.682 [conn38] end connection 10.66.9.105:53390 (44 connections now open) m31100| Fri May 10 08:19:59.683 [conn40] end connection 10.66.9.105:53394 (43 connections now open) m31100| Fri May 10 08:19:59.683 [conn42] end connection 10.66.9.105:53398 (42 connections now open) m31100| Fri May 10 08:19:59.683 [conn44] end connection 10.66.9.105:53402 (41 connections now open) m31100| Fri May 10 08:19:59.683 [conn46] end connection 10.66.9.105:53406 (40 connections now open) m31100| Fri May 10 08:19:59.683 [conn48] end connection 10.66.9.105:53410 (39 connections now open) m31100| Fri May 10 08:19:59.683 [conn50] end connection 10.66.9.105:53414 (38 connections now open) m31100| Fri May 10 08:19:59.683 [conn52] end connection 10.66.9.105:53418 (37 connections now open) m31100| Fri May 10 08:19:59.683 [conn54] end connection 10.66.9.105:53422 (36 connections now open) m31100| Fri May 10 08:19:59.683 [conn56] end connection 10.66.9.105:53426 (35 connections now open) m31100| Fri May 10 08:19:59.684 [conn11] end connection 10.66.9.105:53334 (35 connections now open) m31100| Fri May 10 08:19:59.684 [conn8] end connection 10.66.9.105:53283 (34 connections now open) m31100| Fri May 10 08:19:59.684 [conn58] end connection 10.66.9.105:53430 (32 connections now open) m31100| Fri May 10 08:19:59.684 [conn15] end connection 10.66.9.105:53344 (32 connections now open) m31100| Fri May 10 08:19:59.684 [conn60] end connection 10.66.9.105:53434 (30 connections now open) m31100| Fri May 10 08:19:59.684 [conn13] end connection 10.66.9.105:53337 (29 connections now open) m31100| Fri May 10 08:19:59.684 [conn62] end connection 10.66.9.105:53438 (28 connections now open) m31100| Fri May 10 08:19:59.684 [conn17] end connection 10.66.9.105:53348 (27 connections now open) m31100| Fri May 10 08:19:59.684 [conn64] end connection 10.66.9.105:53442 (27 connections now open) m31100| Fri May 10 08:19:59.684 [conn19] end connection 10.66.9.105:53352 (25 connections now open) m31100| Fri May 10 08:19:59.684 [conn21] end connection 10.66.9.105:53356 (24 connections now open) m31100| Fri May 10 08:19:59.685 [conn23] end connection 10.66.9.105:53360 (23 connections now open) m31100| Fri May 10 08:19:59.685 [conn25] end connection 10.66.9.105:53364 (22 connections now open) m31100| Fri May 10 08:19:59.685 [conn27] end connection 10.66.9.105:53368 (21 connections now open) m31100| Fri May 10 08:19:59.685 [conn29] end connection 10.66.9.105:53372 (20 connections now open) m31100| Fri May 10 08:19:59.685 [conn31] end connection 10.66.9.105:53376 (19 connections now open) m31100| Fri May 10 08:19:59.685 [conn33] end connection 10.66.9.105:53380 (18 connections now open) m31100| Fri May 10 08:19:59.685 [conn35] end connection 10.66.9.105:53384 (17 connections now open) m31100| Fri May 10 08:19:59.685 [conn37] end connection 10.66.9.105:53388 (16 connections now open) m31100| Fri May 10 08:19:59.686 [conn39] end connection 10.66.9.105:53392 (15 connections now open) m31100| Fri May 10 08:19:59.686 [conn41] end connection 10.66.9.105:53396 (14 connections now open) m31100| Fri May 10 08:19:59.686 [conn43] end connection 10.66.9.105:53400 (13 connections now open) m31100| Fri May 10 08:19:59.686 [conn45] end connection 10.66.9.105:53404 (12 connections now open) m31100| Fri May 10 08:19:59.686 [conn47] end connection 10.66.9.105:53408 (11 connections now open) m31100| Fri May 10 08:19:59.686 [conn49] end connection 10.66.9.105:53412 (10 connections now open) m31100| Fri May 10 08:19:59.686 [conn51] end connection 10.66.9.105:53416 (9 connections now open) m31100| Fri May 10 08:19:59.686 [conn53] end connection 10.66.9.105:53420 (8 connections now open) m31100| Fri May 10 08:19:59.686 [conn55] end connection 10.66.9.105:53424 (7 connections now open) m31100| Fri May 10 08:19:59.687 [conn57] end connection 10.66.9.105:53428 (6 connections now open) m31100| Fri May 10 08:19:59.687 [conn59] end connection 10.66.9.105:53432 (5 connections now open) m31100| Fri May 10 08:19:59.687 [conn61] end connection 10.66.9.105:53436 (4 connections now open) m31100| Fri May 10 08:19:59.687 [conn63] end connection 10.66.9.105:53440 (3 connections now open) m31101| Fri May 10 08:19:59.684 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: ip-0A420969:31100 m30999| Fri May 10 08:19:59.685 [WriteBackListener-ip-0A420969:31100] DBClientCursor::init call() failed m30999| Fri May 10 08:19:59.685 [WriteBackListener-ip-0A420969:31100] Detected bad connection created at 1368188398544119 microSec, clearing pool for ip-0A420969:31100 of 0 connections m30999| Fri May 10 08:19:59.685 [WriteBackListener-ip-0A420969:31100] WriteBackListener exception : DBClientBase::findN: transport error: ip-0A420969:31100 ns: admin.$cmd query: { writebacklisten: ObjectId('518ce5eca1fe09096bd5fd9b') } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } m31101| Fri May 10 08:20:00.501 [rsHealthPoll] replSet member ip-0A420969:31100 is now in state SECONDARY m31101| Fri May 10 08:20:00.593 [rsMgr] replSet not trying to elect self as responded yea to someone else recently m31100| Fri May 10 08:20:00.686 [initandlisten] connection accepted from 10.66.9.105:53447 #65 (4 connections now open) m31100| Fri May 10 08:20:00.690 [conn5] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.66.9.105:53277] { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } Fri May 10 08:20:05.765 [ReplicaSetMonitorWatcher] Socket recv() errno:10053 An established connection was aborted by the software in your host machine. 10.66.9.105:31100 Fri May 10 08:20:05.765 [ReplicaSetMonitorWatcher] SocketException: remote: 10.66.9.105:31100 error: 9001 socket exception [RECV_ERROR] server [10.66.9.105:31100] Fri May 10 08:20:05.765 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Fri May 10 08:20:06.765 [ReplicaSetMonitorWatcher] trying reconnect to ip-0A420969:31100 Fri May 10 08:20:06.766 [ReplicaSetMonitorWatcher] reconnect ip-0A420969:31100 ok m31100| Fri May 10 08:20:06.766 [initandlisten] connection accepted from 10.66.9.105:53457 #66 (4 connections now open) Fri May 10 08:20:06.766 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31100 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31100", "ip-0A420969:31101" ], me: "ip-0A420969:31100", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188406766), ok: 1.0 } Fri May 10 08:20:06.767 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31101 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31101", "ip-0A420969:31100" ], me: "ip-0A420969:31101", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188406767), ok: 1.0 } m31101| Fri May 10 08:20:07.018 [rsMgr] replSet info electSelf 1 m31100| Fri May 10 08:20:07.019 [conn9] replSet info voting yea for ip-0A420969:31101 (1) m30999| Fri May 10 08:20:07.308 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : false, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } Fri May 10 08:20:07.767 [ReplicaSetMonitorWatcher] warning: No primary detected for set test-rs0 m30999| Fri May 10 08:20:08.308 [ReplicaSetMonitorWatcher] trying reconnect to ip-0A420969:31100 m30999| Fri May 10 08:20:08.309 [ReplicaSetMonitorWatcher] reconnect ip-0A420969:31100 ok m31100| Fri May 10 08:20:08.309 [initandlisten] connection accepted from 10.66.9.105:53460 #67 (5 connections now open) m30999| Fri May 10 08:20:08.309 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31100 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31100", "ip-0A420969:31101" ], me: "ip-0A420969:31100", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188408309), ok: 1.0 } m30999| Fri May 10 08:20:08.310 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31101 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31101", "ip-0A420969:31100" ], me: "ip-0A420969:31101", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188408310), ok: 1.0 } m30999| Fri May 10 08:20:09.310 [ReplicaSetMonitorWatcher] warning: No primary detected for set test-rs0 m31101| Fri May 10 08:20:09.684 [rsMgr] replSet PRIMARY { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : -1, "nextSlave" : 0 } } m31100| Fri May 10 08:20:10.269 [rsHealthPoll] replSet member ip-0A420969:31101 is now in state PRIMARY m31100| Fri May 10 08:20:10.271 [rsBackgroundSync] replSet syncing to: ip-0A420969:31101 m31101| Fri May 10 08:20:10.272 [initandlisten] connection accepted from 10.66.9.105:53463 #8 (6 connections now open) m31100| Fri May 10 08:20:11.269 [rsSyncNotifier] replset setting oplog notifier to ip-0A420969:31101 m31101| Fri May 10 08:20:11.270 [initandlisten] connection accepted from 10.66.9.105:53464 #9 (7 connections now open) m31100| Fri May 10 08:20:11.270 [rsSyncNotifier] build index local.me { _id: 1 } m31100| Fri May 10 08:20:11.271 [rsSyncNotifier] build index done. scanned 0 total records. 0 secs { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : -1, "nextSlave" : 0 } } m31101| Fri May 10 08:20:12.272 [slaveTracking] build index local.slaves { _id: 1 } m31101| Fri May 10 08:20:12.273 [slaveTracking] build index done. scanned 0 total records. 0.001 secs { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : -1, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : -1, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : -1, "nextSlave" : 0 } } Fri May 10 08:20:17.767 [ReplicaSetMonitorWatcher] Primary for replica set test-rs0 changed to ip-0A420969:31101 m30999| Fri May 10 08:20:19.310 [ReplicaSetMonitorWatcher] Primary for replica set test-rs0 changed to ip-0A420969:31101 ---- Stepping back up... ---- m31101| Fri May 10 08:20:19.506 [conn1] replSet info stepping down as primary secs=10000 m31101| Fri May 10 08:20:19.506 [conn1] replSet relinquishing primary state m31101| Fri May 10 08:20:19.506 [conn1] replSet SECONDARY m31101| Fri May 10 08:20:19.506 [conn1] replSet closing client sockets after relinquishing primary Fri May 10 08:20:19.506 DBClientCursor::init call() failed Error: error doing query: failed ---- Waiting for mongos to acknowledge step up... ---- Awaiting ip-0A420969:31100 to be { "ismaster" : true } for connection to ip-0A420969:30999 (rs: test-rs0) m31101| Fri May 10 08:20:19.506 [conn1] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [127.0.0.1:53261] m31101| Fri May 10 08:20:19.507 [conn4] end connection 10.66.9.105:53284 (5 connections now open) m31101| Fri May 10 08:20:19.507 [conn6] end connection 10.66.9.105:53335 (4 connections now open) m31101| Fri May 10 08:20:19.507 [conn9] end connection 10.66.9.105:53464 (3 connections now open) m31100| Fri May 10 08:20:19.507 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: ip-0A420969:31101 m30999| Fri May 10 08:20:19.507 [WriteBackListener-ip-0A420969:31101] DBClientCursor::init call() failed m30999| Fri May 10 08:20:19.507 [WriteBackListener-ip-0A420969:31101] Detected bad connection created at 1368188398543601 microSec, clearing pool for ip-0A420969:31101 of 0 connections m30999| Fri May 10 08:20:19.508 [WriteBackListener-ip-0A420969:31101] WriteBackListener exception : DBClientBase::findN: transport error: ip-0A420969:31101 ns: admin.$cmd query: { writebacklisten: ObjectId('518ce5eca1fe09096bd5fd9b') } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 } ], "master" : 1, "nextSlave" : 0 } } m31100| Fri May 10 08:20:20.269 [rsHealthPoll] replSet member ip-0A420969:31101 is now in state SECONDARY m31101| Fri May 10 08:20:20.295 [conn8] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.66.9.105:53463] m31101| Fri May 10 08:20:20.509 [initandlisten] connection accepted from 10.66.9.105:53476 #10 (3 connections now open) { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 } ], "master" : 1, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 } ], "master" : 1, "nextSlave" : 0 } } m31101| Fri May 10 08:20:24.269 [conn5] end connection 10.66.9.105:53319 (2 connections now open) m31101| Fri May 10 08:20:24.270 [initandlisten] connection accepted from 10.66.9.105:53481 #11 (3 connections now open) m29000| Fri May 10 08:20:25.075 [conn8] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:20:25.085 [conn8] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:20:25.094 [conn8] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:20:25.213 [conn8] command admin.$cmd command: { getlasterror: 1, fsync: 1 } ntoreturn:1 keyUpdates:0 reslen:113 102ms m29000| Fri May 10 08:20:25.246 [conn8] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:20:25.256 [conn8] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:20:25.263 [conn8] CMD fsync: sync:1 lock:0 { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 } ], "master" : 1, "nextSlave" : 0 } } m31100| Fri May 10 08:20:26.501 [conn9] end connection 10.66.9.105:53330 (4 connections now open) m31100| Fri May 10 08:20:26.502 [initandlisten] connection accepted from 10.66.9.105:53484 #68 (5 connections now open) { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 } ], "master" : 1, "nextSlave" : 0 } } Fri May 10 08:20:27.767 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Fri May 10 08:20:28.767 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31100 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31100", "ip-0A420969:31101" ], me: "ip-0A420969:31100", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188428767), ok: 1.0 } Fri May 10 08:20:28.767 [ReplicaSetMonitorWatcher] trying reconnect to ip-0A420969:31101 m31101| Fri May 10 08:20:28.768 [initandlisten] connection accepted from 10.66.9.105:53490 #12 (4 connections now open) Fri May 10 08:20:28.769 [ReplicaSetMonitorWatcher] reconnect ip-0A420969:31101 ok Fri May 10 08:20:28.769 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31101 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31101", "ip-0A420969:31100" ], me: "ip-0A420969:31101", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188428769), ok: 1.0 } m30999| Fri May 10 08:20:29.310 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : false, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 } ], "master" : 1, "nextSlave" : 0 } } Fri May 10 08:20:29.769 [ReplicaSetMonitorWatcher] warning: No primary detected for set test-rs0 m30999| Fri May 10 08:20:30.310 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31100 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31100", "ip-0A420969:31101" ], me: "ip-0A420969:31100", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188430310), ok: 1.0 } m30999| Fri May 10 08:20:30.310 [ReplicaSetMonitorWatcher] trying reconnect to ip-0A420969:31101 m30999| Fri May 10 08:20:30.311 [ReplicaSetMonitorWatcher] reconnect ip-0A420969:31101 ok m31101| Fri May 10 08:20:30.311 [initandlisten] connection accepted from 10.66.9.105:53493 #13 (5 connections now open) m30999| Fri May 10 08:20:30.312 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31101 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31101", "ip-0A420969:31100" ], me: "ip-0A420969:31101", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188430311), ok: 1.0 } m30999| Fri May 10 08:20:31.312 [ReplicaSetMonitorWatcher] warning: No primary detected for set test-rs0 { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } Fri May 10 08:20:40.769 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31100 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31100", "ip-0A420969:31101" ], me: "ip-0A420969:31100", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188440769), ok: 1.0 } Fri May 10 08:20:40.769 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31101 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31101", "ip-0A420969:31100" ], me: "ip-0A420969:31101", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188440769), ok: 1.0 } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } Fri May 10 08:20:41.769 [ReplicaSetMonitorWatcher] warning: No primary detected for set test-rs0 m30999| Fri May 10 08:20:42.312 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31100 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31100", "ip-0A420969:31101" ], me: "ip-0A420969:31100", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188442312), ok: 1.0 } m30999| Fri May 10 08:20:42.312 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31101 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31101", "ip-0A420969:31100" ], me: "ip-0A420969:31101", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188442312), ok: 1.0 } m30999| Fri May 10 08:20:43.312 [ReplicaSetMonitorWatcher] warning: No primary detected for set test-rs0 { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } Fri May 10 08:20:52.769 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31100 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31100", "ip-0A420969:31101" ], me: "ip-0A420969:31100", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188452769), ok: 1.0 } Fri May 10 08:20:52.769 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31101 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31101", "ip-0A420969:31100" ], me: "ip-0A420969:31101", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188452769), ok: 1.0 } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } Fri May 10 08:20:53.769 [ReplicaSetMonitorWatcher] warning: No primary detected for set test-rs0 m31101| Fri May 10 08:20:54.270 [conn11] end connection 10.66.9.105:53481 (4 connections now open) m31101| Fri May 10 08:20:54.271 [initandlisten] connection accepted from 10.66.9.105:53523 #14 (5 connections now open) m30999| Fri May 10 08:20:54.312 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31100 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31100", "ip-0A420969:31101" ], me: "ip-0A420969:31100", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188454312), ok: 1.0 } m30999| Fri May 10 08:20:54.312 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31101 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31101", "ip-0A420969:31100" ], me: "ip-0A420969:31101", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188454312), ok: 1.0 } m30999| Fri May 10 08:20:55.312 [ReplicaSetMonitorWatcher] warning: No primary detected for set test-rs0 m29000| Fri May 10 08:20:55.314 [conn8] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:20:55.320 [conn8] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:20:55.326 [conn8] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:20:55.502 [conn8] command admin.$cmd command: { getlasterror: 1, fsync: 1 } ntoreturn:1 keyUpdates:0 reslen:113 102ms m29000| Fri May 10 08:20:55.502 [conn8] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:20:55.514 [conn8] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:20:55.522 [conn8] CMD fsync: sync:1 lock:0 { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } m31100| Fri May 10 08:20:56.502 [conn68] end connection 10.66.9.105:53484 (4 connections now open) m31100| Fri May 10 08:20:56.503 [initandlisten] connection accepted from 10.66.9.105:53526 #69 (5 connections now open) { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } m31100| Fri May 10 08:21:02.272 [rsMgr] not electing self, ip-0A420969:31101 would veto with 'I don't think ip-0A420969:31100 is electable' { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } Fri May 10 08:21:04.769 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31100 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31100", "ip-0A420969:31101" ], me: "ip-0A420969:31100", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188464769), ok: 1.0 } Fri May 10 08:21:04.769 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31101 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31101", "ip-0A420969:31100" ], me: "ip-0A420969:31101", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188464769), ok: 1.0 } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } Fri May 10 08:21:05.769 [ReplicaSetMonitorWatcher] warning: No primary detected for set test-rs0 m30999| Fri May 10 08:21:06.312 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31100 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31100", "ip-0A420969:31101" ], me: "ip-0A420969:31100", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188466312), ok: 1.0 } m30999| Fri May 10 08:21:06.312 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: ip-0A420969:31101 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "ip-0A420969:31101", "ip-0A420969:31100" ], me: "ip-0A420969:31101", maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, localTime: new Date(1368188466312), ok: 1.0 } m30999| Fri May 10 08:21:07.312 [ReplicaSetMonitorWatcher] warning: No primary detected for set test-rs0 { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } m31100| Fri May 10 08:21:08.271 [rsMgr] replSet info electSelf 0 m31101| Fri May 10 08:21:08.272 [conn14] replSet info voting yea for ip-0A420969:31100 (0) m31100| Fri May 10 08:21:08.507 [rsMgr] replSet PRIMARY { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } m31101| Fri May 10 08:21:10.503 [rsHealthPoll] replSet member ip-0A420969:31100 is now in state PRIMARY m31101| Fri May 10 08:21:10.684 [rsBackgroundSync] replSet syncing to: ip-0A420969:31100 m31100| Fri May 10 08:21:10.684 [initandlisten] connection accepted from 10.66.9.105:53547 #70 (6 connections now open) { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "ip-0A420969:31100", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 }, { "addr" : "ip-0A420969:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 1 } ], "master" : -1, "nextSlave" : 0 } } Fri May 10 08:21:15.769 [ReplicaSetMonitorWatcher] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:17.312 [ReplicaSetMonitorWatcher] Primary for replica set test-rs0 changed to ip-0A420969:31100 ---- Waiting for socket timeout time... ---- m31101| Fri May 10 08:21:24.271 [conn14] end connection 10.66.9.105:53523 (4 connections now open) m31101| Fri May 10 08:21:24.272 [initandlisten] connection accepted from 10.66.9.105:53557 #15 (5 connections now open) m29000| Fri May 10 08:21:25.570 [conn7] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:21:25.577 [conn7] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:21:25.584 [conn7] CMD fsync: sync:1 lock:0 m29000| Fri May 10 08:21:25.660 [conn7] CMD fsync: sync:1 lock:0 m29001| Fri May 10 08:21:25.670 [conn7] CMD fsync: sync:1 lock:0 m29002| Fri May 10 08:21:25.678 [conn7] CMD fsync: sync:1 lock:0 m31100| Fri May 10 08:21:26.503 [conn69] end connection 10.66.9.105:53526 (5 connections now open) m31100| Fri May 10 08:21:26.504 [initandlisten] connection accepted from 10.66.9.105:53558 #71 (6 connections now open) ---- Run queries using new connections. ---- m30999| Fri May 10 08:21:27.367 [mongosMain] connection accepted from 10.66.9.105:53559 #52 (2 connections now open) m30999| Fri May 10 08:21:27.368 [conn52] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:27.368 [conn52] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:27.368 [conn52] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:27.368 [conn52] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:27.369 [mongosMain] connection accepted from 10.66.9.105:53561 #53 (3 connections now open) m30999| Fri May 10 08:21:27.370 [conn53] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:27.371 [conn53] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:27.371 [conn53] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:27.371 [conn53] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:27.371 [conn53] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:27.372 [mongosMain] connection accepted from 10.66.9.105:53562 #54 (4 connections now open) m30999| Fri May 10 08:21:27.372 [conn54] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:27.373 [conn54] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:27.373 [conn54] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:27.373 [conn54] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:27.373 [conn54] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:27.374 [mongosMain] connection accepted from 10.66.9.105:53563 #55 (5 connections now open) m30999| Fri May 10 08:21:27.375 [conn55] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:27.375 [conn55] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:27.375 [conn55] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:27.375 [conn55] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:27.375 [conn55] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.193 [mongosMain] connection accepted from 10.66.9.105:53565 #56 (6 connections now open) m30999| Fri May 10 08:21:28.194 [conn56] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.194 [conn56] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.194 [conn56] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.194 [conn56] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.194 [conn56] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.195 [mongosMain] connection accepted from 10.66.9.105:53566 #57 (7 connections now open) m30999| Fri May 10 08:21:28.196 [conn57] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.196 [conn57] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.196 [conn57] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.196 [conn57] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.196 [conn57] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.198 [mongosMain] connection accepted from 10.66.9.105:53567 #58 (8 connections now open) m30999| Fri May 10 08:21:28.198 [conn58] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.198 [conn58] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.199 [conn58] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.199 [conn58] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.199 [conn58] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.200 [mongosMain] connection accepted from 10.66.9.105:53568 #59 (9 connections now open) m30999| Fri May 10 08:21:28.201 [conn59] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.201 [conn59] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.201 [conn59] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.201 [conn59] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.201 [conn59] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.204 [mongosMain] connection accepted from 10.66.9.105:53569 #60 (10 connections now open) m30999| Fri May 10 08:21:28.205 [conn60] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.206 [conn60] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.206 [conn60] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.206 [conn60] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.206 [conn60] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.207 [mongosMain] connection accepted from 10.66.9.105:53570 #61 (11 connections now open) m30999| Fri May 10 08:21:28.208 [conn61] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.208 [conn61] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.208 [conn61] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.208 [conn61] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.208 [conn61] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.209 [mongosMain] connection accepted from 10.66.9.105:53571 #62 (12 connections now open) m30999| Fri May 10 08:21:28.210 [conn62] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.210 [conn62] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.210 [conn62] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.210 [conn62] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.211 [conn62] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.212 [mongosMain] connection accepted from 10.66.9.105:53572 #63 (13 connections now open) m30999| Fri May 10 08:21:28.212 [conn63] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.213 [conn63] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.213 [conn63] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.213 [conn63] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.213 [conn63] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.214 [mongosMain] connection accepted from 10.66.9.105:53573 #64 (14 connections now open) m30999| Fri May 10 08:21:28.215 [conn64] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.215 [conn64] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.215 [conn64] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.215 [conn64] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.215 [conn64] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.216 [mongosMain] connection accepted from 10.66.9.105:53574 #65 (15 connections now open) m30999| Fri May 10 08:21:28.217 [conn65] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.217 [conn65] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.217 [conn65] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.217 [conn65] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.217 [conn65] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.219 [mongosMain] connection accepted from 10.66.9.105:53575 #66 (16 connections now open) m30999| Fri May 10 08:21:28.219 [conn66] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.220 [conn66] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.220 [conn66] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.220 [conn66] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.220 [conn66] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.221 [mongosMain] connection accepted from 10.66.9.105:53576 #67 (17 connections now open) m30999| Fri May 10 08:21:28.221 [conn67] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.222 [conn67] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.222 [conn67] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.222 [conn67] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.222 [conn67] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.223 [mongosMain] connection accepted from 10.66.9.105:53577 #68 (18 connections now open) m30999| Fri May 10 08:21:28.224 [conn68] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.224 [conn68] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.224 [conn68] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.224 [conn68] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.224 [conn68] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.225 [mongosMain] connection accepted from 10.66.9.105:53578 #69 (19 connections now open) m30999| Fri May 10 08:21:28.226 [conn69] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.226 [conn69] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.226 [conn69] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.226 [conn69] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.226 [conn69] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.231 [mongosMain] connection accepted from 10.66.9.105:53579 #70 (20 connections now open) m30999| Fri May 10 08:21:28.232 [conn70] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.233 [conn70] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.233 [conn70] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.233 [conn70] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.233 [conn70] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.234 [mongosMain] connection accepted from 10.66.9.105:53580 #71 (21 connections now open) m30999| Fri May 10 08:21:28.235 [conn71] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.235 [conn71] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.235 [conn71] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.235 [conn71] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.235 [conn71] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.237 [mongosMain] connection accepted from 10.66.9.105:53581 #72 (22 connections now open) m30999| Fri May 10 08:21:28.237 [conn72] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.238 [conn72] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.238 [conn72] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.238 [conn72] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.238 [conn72] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.239 [mongosMain] connection accepted from 10.66.9.105:53582 #73 (23 connections now open) m30999| Fri May 10 08:21:28.239 [conn73] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.240 [conn73] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.240 [conn73] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.240 [conn73] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.240 [conn73] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.241 [mongosMain] connection accepted from 10.66.9.105:53583 #74 (24 connections now open) m30999| Fri May 10 08:21:28.242 [conn74] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.242 [conn74] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.242 [conn74] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.242 [conn74] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.242 [conn74] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.243 [mongosMain] connection accepted from 10.66.9.105:53584 #75 (25 connections now open) m30999| Fri May 10 08:21:28.244 [conn75] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.244 [conn75] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.244 [conn75] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.244 [conn75] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.244 [conn75] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.245 [mongosMain] connection accepted from 10.66.9.105:53585 #76 (26 connections now open) m30999| Fri May 10 08:21:28.246 [conn76] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.246 [conn76] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.246 [conn76] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.246 [conn76] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.246 [conn76] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.247 [mongosMain] connection accepted from 10.66.9.105:53586 #77 (27 connections now open) m30999| Fri May 10 08:21:28.248 [conn77] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.249 [conn77] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.249 [conn77] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.249 [conn77] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.249 [conn77] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.250 [mongosMain] connection accepted from 10.66.9.105:53587 #78 (28 connections now open) m30999| Fri May 10 08:21:28.250 [conn78] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.251 [conn78] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.251 [conn78] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.251 [conn78] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.251 [conn78] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.252 [mongosMain] connection accepted from 10.66.9.105:53588 #79 (29 connections now open) m30999| Fri May 10 08:21:28.253 [conn79] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.253 [conn79] Socket recv() errno:10053 An established connection was aborted by the software in your host machine. 10.66.9.105:31100 m30999| Fri May 10 08:21:28.253 [conn79] SocketException: remote: 10.66.9.105:31100 error: 9001 socket exception [RECV_ERROR] server [10.66.9.105:31100] m30999| Fri May 10 08:21:28.253 [conn79] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.253 [conn79] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.254 [conn79] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.254 [conn79] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.255 [mongosMain] connection accepted from 10.66.9.105:53590 #80 (30 connections now open) m30999| Fri May 10 08:21:28.256 [conn80] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.256 [conn80] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.256 [conn80] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.256 [conn80] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.256 [conn80] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.258 [mongosMain] connection accepted from 10.66.9.105:53591 #81 (31 connections now open) m30999| Fri May 10 08:21:28.258 [conn81] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.259 [conn81] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.259 [conn81] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.259 [conn81] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.259 [conn81] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:28.260 [mongosMain] connection accepted from 10.66.9.105:53592 #82 (32 connections now open) m30999| Fri May 10 08:21:28.261 [conn82] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:28.261 [conn82] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:28.261 [conn82] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:28.261 [conn82] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:28.261 [conn82] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.412 [mongosMain] connection accepted from 10.66.9.105:53594 #83 (33 connections now open) m30999| Fri May 10 08:21:29.414 [conn83] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.414 [conn83] Socket recv() errno:10053 An established connection was aborted by the software in your host machine. 10.66.9.105:31100 m30999| Fri May 10 08:21:29.414 [conn83] SocketException: remote: 10.66.9.105:31100 error: 9001 socket exception [RECV_ERROR] server [10.66.9.105:31100] m30999| Fri May 10 08:21:29.414 [conn83] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.414 [conn83] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.414 [conn83] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.414 [conn83] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.416 [mongosMain] connection accepted from 10.66.9.105:53595 #84 (34 connections now open) m30999| Fri May 10 08:21:29.417 [conn84] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.417 [conn84] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.417 [conn84] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.417 [conn84] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.417 [conn84] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.419 [mongosMain] connection accepted from 10.66.9.105:53596 #85 (35 connections now open) m30999| Fri May 10 08:21:29.420 [conn85] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.420 [conn85] Socket recv() errno:10053 An established connection was aborted by the software in your host machine. 10.66.9.105:31100 m30999| Fri May 10 08:21:29.420 [conn85] SocketException: remote: 10.66.9.105:31100 error: 9001 socket exception [RECV_ERROR] server [10.66.9.105:31100] m30999| Fri May 10 08:21:29.420 [conn85] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.420 [conn85] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.420 [conn85] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.421 [conn85] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.422 [mongosMain] connection accepted from 10.66.9.105:53597 #86 (36 connections now open) m30999| Fri May 10 08:21:29.423 [conn86] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.423 [conn86] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.423 [conn86] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.423 [conn86] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.423 [conn86] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.425 [mongosMain] connection accepted from 10.66.9.105:53598 #87 (37 connections now open) m30999| Fri May 10 08:21:29.426 [conn87] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.426 [conn87] Socket recv() errno:10053 An established connection was aborted by the software in your host machine. 10.66.9.105:31100 m30999| Fri May 10 08:21:29.426 [conn87] SocketException: remote: 10.66.9.105:31100 error: 9001 socket exception [RECV_ERROR] server [10.66.9.105:31100] m30999| Fri May 10 08:21:29.426 [conn87] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.426 [conn87] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.426 [conn87] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.427 [conn87] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.429 [mongosMain] connection accepted from 10.66.9.105:53599 #88 (38 connections now open) m30999| Fri May 10 08:21:29.429 [conn88] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.429 [conn88] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.429 [conn88] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.429 [conn88] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.429 [conn88] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.431 [mongosMain] connection accepted from 10.66.9.105:53600 #89 (39 connections now open) m30999| Fri May 10 08:21:29.432 [conn89] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.432 [conn89] Socket recv() errno:10053 An established connection was aborted by the software in your host machine. 10.66.9.105:31100 m30999| Fri May 10 08:21:29.432 [conn89] SocketException: remote: 10.66.9.105:31100 error: 9001 socket exception [RECV_ERROR] server [10.66.9.105:31100] m30999| Fri May 10 08:21:29.432 [conn89] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.432 [conn89] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.432 [conn89] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.432 [conn89] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.434 [mongosMain] connection accepted from 10.66.9.105:53601 #90 (40 connections now open) m30999| Fri May 10 08:21:29.435 [conn90] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.435 [conn90] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.435 [conn90] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.435 [conn90] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.435 [conn90] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.437 [mongosMain] connection accepted from 10.66.9.105:53602 #91 (41 connections now open) m30999| Fri May 10 08:21:29.438 [conn91] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.438 [conn91] Socket recv() errno:10053 An established connection was aborted by the software in your host machine. 10.66.9.105:31100 m30999| Fri May 10 08:21:29.438 [conn91] SocketException: remote: 10.66.9.105:31100 error: 9001 socket exception [RECV_ERROR] server [10.66.9.105:31100] m30999| Fri May 10 08:21:29.438 [conn91] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.438 [conn91] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.438 [conn91] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.438 [conn91] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.440 [mongosMain] connection accepted from 10.66.9.105:53603 #92 (42 connections now open) m30999| Fri May 10 08:21:29.441 [conn92] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.441 [conn92] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.441 [conn92] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.441 [conn92] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.441 [conn92] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.443 [mongosMain] connection accepted from 10.66.9.105:53604 #93 (43 connections now open) m30999| Fri May 10 08:21:29.444 [conn93] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.444 [conn93] Socket recv() errno:10053 An established connection was aborted by the software in your host machine. 10.66.9.105:31100 m30999| Fri May 10 08:21:29.444 [conn93] SocketException: remote: 10.66.9.105:31100 error: 9001 socket exception [RECV_ERROR] server [10.66.9.105:31100] m30999| Fri May 10 08:21:29.444 [conn93] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.444 [conn93] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.444 [conn93] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.444 [conn93] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.446 [mongosMain] connection accepted from 10.66.9.105:53605 #94 (44 connections now open) m30999| Fri May 10 08:21:29.447 [conn94] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.447 [conn94] Socket recv() errno:10053 An established connection was aborted by the software in your host machine. 10.66.9.105:31100 m30999| Fri May 10 08:21:29.447 [conn94] SocketException: remote: 10.66.9.105:31100 error: 9001 socket exception [RECV_ERROR] server [10.66.9.105:31100] m30999| Fri May 10 08:21:29.447 [conn94] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.447 [conn94] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.447 [conn94] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.447 [conn94] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.449 [mongosMain] connection accepted from 10.66.9.105:53606 #95 (45 connections now open) m30999| Fri May 10 08:21:29.450 [conn95] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.450 [conn95] Socket recv() errno:10053 An established connection was aborted by the software in your host machine. 10.66.9.105:31100 m30999| Fri May 10 08:21:29.450 [conn95] SocketException: remote: 10.66.9.105:31100 error: 9001 socket exception [RECV_ERROR] server [10.66.9.105:31100] m30999| Fri May 10 08:21:29.450 [conn95] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.450 [conn95] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.450 [conn95] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.450 [conn95] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.452 [mongosMain] connection accepted from 10.66.9.105:53607 #96 (46 connections now open) m30999| Fri May 10 08:21:29.452 [conn96] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.453 [conn96] Socket recv() errno:10053 An established connection was aborted by the software in your host machine. 10.66.9.105:31100 m30999| Fri May 10 08:21:29.453 [conn96] SocketException: remote: 10.66.9.105:31100 error: 9001 socket exception [RECV_ERROR] server [10.66.9.105:31100] m30999| Fri May 10 08:21:29.453 [conn96] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.453 [conn96] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.453 [conn96] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.453 [conn96] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.455 [mongosMain] connection accepted from 10.66.9.105:53608 #97 (47 connections now open) m30999| Fri May 10 08:21:29.455 [conn97] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.456 [conn97] Socket recv() errno:10053 An established connection was aborted by the software in your host machine. 10.66.9.105:31100 m30999| Fri May 10 08:21:29.456 [conn97] SocketException: remote: 10.66.9.105:31100 error: 9001 socket exception [RECV_ERROR] server [10.66.9.105:31100] m30999| Fri May 10 08:21:29.456 [conn97] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.456 [conn97] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.456 [conn97] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.456 [conn97] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.458 [mongosMain] connection accepted from 10.66.9.105:53609 #98 (48 connections now open) m30999| Fri May 10 08:21:29.459 [conn98] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.459 [conn98] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.459 [conn98] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.459 [conn98] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.459 [conn98] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.460 [mongosMain] connection accepted from 10.66.9.105:53610 #99 (49 connections now open) m30999| Fri May 10 08:21:29.461 [conn99] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.462 [conn99] Socket recv() errno:10053 An established connection was aborted by the software in your host machine. 10.66.9.105:31100 m30999| Fri May 10 08:21:29.462 [conn99] SocketException: remote: 10.66.9.105:31100 error: 9001 socket exception [RECV_ERROR] server [10.66.9.105:31100] m30999| Fri May 10 08:21:29.462 [conn99] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.462 [conn99] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.462 [conn99] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.462 [conn99] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.463 [mongosMain] connection accepted from 10.66.9.105:53611 #100 (50 connections now open) m30999| Fri May 10 08:21:29.464 [conn100] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.465 [conn100] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.465 [conn100] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.465 [conn100] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.465 [conn100] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" m30999| Fri May 10 08:21:29.466 [mongosMain] connection accepted from 10.66.9.105:53612 #101 (51 connections now open) m30999| Fri May 10 08:21:29.467 [conn101] Primary for replica set test-rs0 changed to ip-0A420969:31100 m30999| Fri May 10 08:21:29.467 [conn101] Socket recv() errno:10053 An established connection was aborted by the software in your host machine. 10.66.9.105:31100 m30999| Fri May 10 08:21:29.467 [conn101] SocketException: remote: 10.66.9.105:31100 error: 9001 socket exception [RECV_ERROR] server [10.66.9.105:31100] m30999| Fri May 10 08:21:29.467 [conn101] DBClientCursor::init lazy say() failed m30999| Fri May 10 08:21:29.467 [conn101] DBClientCursor::init message from say() was empty m30999| Fri May 10 08:21:29.467 [conn101] got not master for: ip-0A420969:31100 m30999| Fri May 10 08:21:29.468 [conn101] warning: exception when receiving data from test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101, current connection state is { state: { conn: "test-rs0/ip-0A420969:31100,ip-0A420969:31101", vinfo: "test-rs0:test-rs0/ip-0A420969:31100,ip-0A420969:31101", cursor: "(empty)", count: 0, done: false }, retryNext: false, init: true, finish: true, errored: false } :: caused by :: 15988 error querying server "error: { \"$err\" : \"error querying server\", \"code\" : 15988 }" assert: [0] != [50] are not equal : undefined Error: Printing Stack Trace at printStackTrace (src/mongo/shell/utils.js:37:15) at doassert (src/mongo/shell/assert.js:6:5) at Function.assert.eq (src/mongo/shell/assert.js:32:5) at C:\10gen\buildslaves\mongo\Windows_32bit\mongo\jstests\sharding\rs_stepdown_and_pooling.js:99:8 Fri May 10 08:21:29.473 JavaScript execution failed: [0] != [50] are not equal : undefined at src/mongo/shell/assert.js:L7 failed to load: C:\10gen\buildslaves\mongo\Windows_32bit\mongo\jstests\sharding\rs_stepdown_and_pooling.js m29000| Fri May 10 08:21:29.474 [initandlisten] connection accepted from 127.0.0.1:53613 #10 (9 connections now open) m29000| Fri May 10 08:21:29.474 [conn10] terminating, shutdown command received m29000| Fri May 10 08:21:29.474 dbexit: shutdown called m29000| Fri May 10 08:21:29.474 [conn10] shutdown: going to close listening sockets... m29000| Fri May 10 08:21:29.474 [conn10] closing listening socket: 524 m29000| Fri May 10 08:21:29.474 [conn10] closing listening socket: 532 m29000| Fri May 10 08:21:29.474 [conn10] shutdown: going to flush diaglog... m29000| Fri May 10 08:21:29.474 [conn10] shutdown: going to close sockets... m29000| Fri May 10 08:21:29.474 [conn10] shutdown: waiting for fs preallocator... m29000| Fri May 10 08:21:29.474 [conn10] shutdown: lock for final commit... m29000| Fri May 10 08:21:29.474 [conn10] shutdown: final commit... m29000| Fri May 10 08:21:29.475 [conn3] end connection 10.66.9.105:53300 (8 connections now open) m29000| Fri May 10 08:21:29.475 [conn1] end connection 127.0.0.1:53286 (8 connections now open) m29000| Fri May 10 08:21:29.475 [conn2] end connection 10.66.9.105:53296 (8 connections now open) m29000| Fri May 10 08:21:29.475 [conn7] end connection 10.66.9.105:53310 (8 connections now open) m29000| Fri May 10 08:21:29.475 [conn6] end connection 10.66.9.105:53309 (8 connections now open) m29000| Fri May 10 08:21:29.475 [conn5] end connection 10.66.9.105:53306 (8 connections now open) m29000| Fri May 10 08:21:29.475 [conn9] end connection 10.66.9.105:53340 (8 connections now open) m29000| Fri May 10 08:21:29.475 [conn8] end connection 10.66.9.105:53321 (8 connections now open) Fri May 10 08:21:29.475 DBClientCursor::init call() failed m29000| Fri May 10 08:21:29.483 [conn10] shutdown: closing all files... m29000| Fri May 10 08:21:29.485 [conn10] closeAllFiles() finished m29000| Fri May 10 08:21:29.485 [conn10] journalCleanup... m29000| Fri May 10 08:21:29.487 [conn10] removeJournalFiles m29000| Fri May 10 08:21:29.488 [conn10] shutdown: removing fs lock... m29000| Fri May 10 08:21:29.489 dbexit: really exiting now m29001| Fri May 10 08:21:30.476 [initandlisten] connection accepted from 127.0.0.1:53619 #10 (9 connections now open) m29001| Fri May 10 08:21:30.477 [conn10] terminating, shutdown command received m29001| Fri May 10 08:21:30.477 dbexit: shutdown called m29001| Fri May 10 08:21:30.477 [conn10] shutdown: going to close listening sockets... m29001| Fri May 10 08:21:30.477 [conn10] closing listening socket: 548 m29001| Fri May 10 08:21:30.477 [conn10] closing listening socket: 556 m29001| Fri May 10 08:21:30.477 [conn10] shutdown: going to flush diaglog... m29001| Fri May 10 08:21:30.477 [conn10] shutdown: going to close sockets... m29001| Fri May 10 08:21:30.477 [conn10] shutdown: waiting for fs preallocator... m29001| Fri May 10 08:21:30.477 [conn10] shutdown: lock for final commit... m29001| Fri May 10 08:21:30.477 [conn10] shutdown: final commit... m29001| Fri May 10 08:21:30.477 [conn3] end connection 10.66.9.105:53301 (8 connections now open) m29001| Fri May 10 08:21:30.477 [conn6] end connection 10.66.9.105:53311 (8 connections now open) m29001| Fri May 10 08:21:30.477 [conn1] end connection 127.0.0.1:53290 (8 connections now open) m29001| Fri May 10 08:21:30.477 [conn2] end connection 10.66.9.105:53297 (8 connections now open) m29001| Fri May 10 08:21:30.477 [conn8] end connection 10.66.9.105:53322 (8 connections now open) m29001| Fri May 10 08:21:30.477 [conn5] end connection 10.66.9.105:53307 (8 connections now open) m29001| Fri May 10 08:21:30.477 [conn7] end connection 10.66.9.105:53312 (8 connections now open) m29001| Fri May 10 08:21:30.478 [conn9] end connection 10.66.9.105:53341 (8 connections now open) Fri May 10 08:21:30.478 DBClientCursor::init call() failed m29001| Fri May 10 08:21:30.483 [conn10] shutdown: closing all files... m29001| Fri May 10 08:21:30.484 [conn10] closeAllFiles() finished m29001| Fri May 10 08:21:30.485 [conn10] journalCleanup... m29001| Fri May 10 08:21:30.487 [conn10] removeJournalFiles m29001| Fri May 10 08:21:30.488 [conn10] shutdown: removing fs lock... m29001| Fri May 10 08:21:30.488 dbexit: really exiting now m29002| Fri May 10 08:21:31.478 [initandlisten] connection accepted from 127.0.0.1:53620 #10 (9 connections now open) m29002| Fri May 10 08:21:31.479 [conn10] terminating, shutdown command received m29002| Fri May 10 08:21:31.479 dbexit: shutdown called m29002| Fri May 10 08:21:31.479 [conn10] shutdown: going to close listening sockets... m29002| Fri May 10 08:21:31.479 [conn10] closing listening socket: 512 m29002| Fri May 10 08:21:31.479 [conn10] closing listening socket: 536 m29002| Fri May 10 08:21:31.479 [conn10] shutdown: going to flush diaglog... m29002| Fri May 10 08:21:31.479 [conn10] shutdown: going to close sockets... m29002| Fri May 10 08:21:31.479 [conn10] shutdown: waiting for fs preallocator... m29002| Fri May 10 08:21:31.479 [conn10] shutdown: lock for final commit... m29002| Fri May 10 08:21:31.479 [conn10] shutdown: final commit... m29002| Fri May 10 08:21:31.479 [conn3] end connection 10.66.9.105:53302 (8 connections now open) m29002| Fri May 10 08:21:31.479 [conn1] end connection 127.0.0.1:53294 (8 connections now open) m29002| Fri May 10 08:21:31.479 [conn2] end connection 10.66.9.105:53298 (8 connections now open) m29002| Fri May 10 08:21:31.479 [conn5] end connection 10.66.9.105:53308 (8 connections now open) m29002| Fri May 10 08:21:31.479 [conn6] end connection 10.66.9.105:53313 (8 connections now open) m29002| Fri May 10 08:21:31.479 [conn7] end connection 10.66.9.105:53314 (8 connections now open) m29002| Fri May 10 08:21:31.479 [conn8] end connection 10.66.9.105:53323 (8 connections now open) m29002| Fri May 10 08:21:31.479 [conn9] end connection 10.66.9.105:53342 (8 connections now open) Fri May 10 08:21:31.480 DBClientCursor::init call() failed m29002| Fri May 10 08:21:31.485 [conn10] shutdown: closing all files... m29002| Fri May 10 08:21:31.486 [conn10] closeAllFiles() finished m29002| Fri May 10 08:21:31.486 [conn10] journalCleanup... m29002| Fri May 10 08:21:31.489 [conn10] removeJournalFiles m29002| Fri May 10 08:21:31.490 [conn10] shutdown: removing fs lock... m29002| Fri May 10 08:21:31.490 dbexit: really exiting now m30999| Fri May 10 08:21:33.320 [Balancer] DBClientCursor::init call() failed m30999| Fri May 10 08:21:33.320 [Balancer] query failed to: ip-0A420969:29000 no data m30999| Fri May 10 08:21:33.320 [Balancer] DBClientCursor::init call() failed m30999| Fri May 10 08:21:33.320 [Balancer] query failed to: ip-0A420969:29001 no data m30999| Fri May 10 08:21:33.320 [Balancer] DBClientCursor::init call() failed m30999| Fri May 10 08:21:33.320 [Balancer] query failed to: ip-0A420969:29002 no data m30999| Fri May 10 08:21:33.320 [Balancer] scoped connection to ip-0A420969:29000,ip-0A420969:29001,ip-0A420969:29002 not being returned to the pool m30999| Fri May 10 08:21:33.320 [Balancer] scoped connection to ip-0A420969:29000,ip-0A420969:29001,ip-0A420969:29002 not being returned to the pool m30999| Fri May 10 08:21:33.320 [Balancer] caught exception while doing balance: all servers down/unreachable when querying: ip-0A420969:29000,ip-0A420969:29001,ip-0A420969:29002 m30999| Fri May 10 08:21:34.375 [mongosMain] connection accepted from 127.0.0.1:53623 #102 (52 connections now open) m30999| Fri May 10 08:21:34.376 [conn102] terminating, shutdown command received m30999| Fri May 10 08:21:34.376 [conn102] dbexit: shutdown called rc:0 shutdown called Fri May 10 08:21:34.378 Socket recv() errno:10054 An existing connection was forcibly closed by the remote host. 127.0.0.1:30999 Fri May 10 08:21:34.378 SocketException: remote: 127.0.0.1:30999 error: 9001 socket exception [RECV_ERROR] server [127.0.0.1:30999] Fri May 10 08:21:34.378 DBClientCursor::init call() failed m31101| Fri May 10 08:21:34.379 [conn13] end connection 10.66.9.105:53493 (4 connections now open) m31100| Fri May 10 08:21:34.379 [conn67] end connection 10.66.9.105:53460 (5 connections now open) m31100| Fri May 10 08:21:34.380 [initandlisten] connection accepted from 127.0.0.1:53624 #72 (6 connections now open) m31100| Fri May 10 08:21:34.380 [conn72] terminating, shutdown command received m31100| Fri May 10 08:21:34.380 dbexit: shutdown called m31100| Fri May 10 08:21:34.380 [conn72] shutdown: going to close listening sockets... m31100| Fri May 10 08:21:34.380 [conn72] closing listening socket: 504 m31100| Fri May 10 08:21:34.380 [conn72] closing listening socket: 564 m31100| Fri May 10 08:21:34.380 [conn72] shutdown: going to flush diaglog... m31100| Fri May 10 08:21:34.380 [conn72] shutdown: going to close sockets... m31100| Fri May 10 08:21:34.381 [conn72] shutdown: waiting for fs preallocator... m31100| Fri May 10 08:21:34.381 [conn72] shutdown: closing all files... m31100| Fri May 10 08:21:34.382 [conn72] closeAllFiles() finished Fri May 10 08:21:34.382 DBClientCursor::init call() failed m31101| Fri May 10 08:21:34.382 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: ip-0A420969:31100 m31101| Fri May 10 08:21:34.383 [conn15] end connection 10.66.9.105:53557 (3 connections now open) m31100| Fri May 10 08:21:34.382 [conn66] end connection 10.66.9.105:53457 (5 connections now open) m31100| Fri May 10 08:21:34.383 [conn72] shutdown: removing fs lock... m31100| Fri May 10 08:21:34.383 [conn71] end connection 10.66.9.105:53558 (5 connections now open) m31100| Fri May 10 08:21:34.383 dbexit: really exiting now m31101| Fri May 10 08:21:34.504 [rsHealthPoll] DBClientCursor::init call() failed m31101| Fri May 10 08:21:34.504 [rsHealthPoll] replset info ip-0A420969:31100 heartbeat failed, retrying m31101| Fri May 10 08:21:35.382 [initandlisten] connection accepted from 127.0.0.1:53626 #16 (4 connections now open) m31101| Fri May 10 08:21:35.383 [conn16] terminating, shutdown command received m31101| Fri May 10 08:21:35.383 dbexit: shutdown called m31101| Fri May 10 08:21:35.383 [conn16] shutdown: going to close listening sockets... m31101| Fri May 10 08:21:35.383 [conn16] closing listening socket: 544 m31101| Fri May 10 08:21:35.383 [conn16] closing listening socket: 548 m31101| Fri May 10 08:21:35.383 [conn16] shutdown: going to flush diaglog... m31101| Fri May 10 08:21:35.383 [conn16] shutdown: going to close sockets... m31101| Fri May 10 08:21:35.383 [conn16] shutdown: waiting for fs preallocator... m31101| Fri May 10 08:21:35.383 [conn16] shutdown: closing all files... Fri May 10 08:21:35.383 DBClientCursor::init call() failed m31101| Fri May 10 08:21:35.384 [rsHealthPoll] replSet info ip-0A420969:31100 is down (or slow to respond): m31101| Fri May 10 08:21:35.384 [rsHealthPoll] replSet member ip-0A420969:31100 is now in state DOWN m31101| Fri May 10 08:21:35.384 [conn12] end connection 10.66.9.105:53490 (3 connections now open) m31101| Fri May 10 08:21:35.385 [conn16] closeAllFiles() finished m31101| Fri May 10 08:21:35.385 [conn16] shutdown: removing fs lock... m31101| Fri May 10 08:21:35.385 dbexit: really exiting now Fri May 10 08:21:35.769 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Fri May 10 08:21:35.769 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed