Tue Oct 16 12:43:56 [conn12] end connection 127.0.0.1:38706 (0 connections now open) MongoDB shell version: 2.1.3-pre- null Replica set test! ReplSetTest Starting Set ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 31100, 31101, 31102 ] 31100 number { "useHostName" : true, "oplogSize" : 10, "keyFile" : "jstests/libs/key1", "port" : 31100, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "test-rs0", "dbpath" : "$set-$node", "useHostname" : true, "noJournalPrealloc" : 1, "pathOpts" : { "testName" : "test", "shard" : 0, "node" : 0, "set" : "test-rs0" }, "restart" : undefined } ReplSetTest Starting.... Resetting db path '/data/db/test-rs0-0' Tue Oct 16 12:43:57 shell: started program /data/buildslaves/Linux_64bit_GCov_Nightly/mongo/mongod --oplogSize 10 --keyFile jstests/libs/key1 --port 31100 --noprealloc --smallfiles --rest --replSet test-rs0 --dbpath /data/db/test-rs0-0 --nopreallocj m31100| note: noprealloc may hurt performance in many applications m31100| Tue Oct 16 12:43:57 [initandlisten] MongoDB starting : pid=14793 port=31100 dbpath=/data/db/test-rs0-0 64-bit host=gcov1 m31100| Tue Oct 16 12:43:57 [initandlisten] m31100| Tue Oct 16 12:43:57 [initandlisten] ** NOTE: This is a development version (2.1.3-pre-) of MongoDB. m31100| Tue Oct 16 12:43:57 [initandlisten] ** Not recommended for production. m31100| Tue Oct 16 12:43:57 [initandlisten] m31100| Tue Oct 16 12:43:57 [initandlisten] db version v2.1.3-pre-, pdfile version 4.5 m31100| Tue Oct 16 12:43:57 [initandlisten] git version: 0f8117c897127c9c83fe00ff07b58d87a3ae2c89 m31100| Tue Oct 16 12:43:57 [initandlisten] build info: Linux gcov1 3.2.20-1.29.6.amzn1.x86_64 #1 SMP Tue Jun 12 01:19:28 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31100| Tue Oct 16 12:43:57 [initandlisten] options: { dbpath: "/data/db/test-rs0-0", keyFile: "jstests/libs/key1", noprealloc: true, nopreallocj: true, oplogSize: 10, port: 31100, replSet: "test-rs0", rest: true, smallfiles: true } m31100| Tue Oct 16 12:43:57 [initandlisten] journal dir=/data/db/test-rs0-0/journal m31100| Tue Oct 16 12:43:57 [initandlisten] recover : no journal files present, no recovery needed m31100| Tue Oct 16 12:43:57 [initandlisten] waiting for connections on port 31100 m31100| Tue Oct 16 12:43:57 [websvr] admin web console waiting for connections on port 32100 m31100| Tue Oct 16 12:43:57 [initandlisten] connection accepted from 10.29.160.141:45244 #1 (1 connection now open) m31100| Tue Oct 16 12:43:57 [conn1] authenticate db: local { authenticate: 1, nonce: "b3d7cef4e021a845", user: "__system", key: "fb9f796131b76fd3b1d85cae51e79367" } m31100| Tue Oct 16 12:43:57 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31100| Tue Oct 16 12:43:57 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31100| Tue Oct 16 12:43:57 [initandlisten] connection accepted from 127.0.0.1:35257 #2 (2 connections now open) m31100| Tue Oct 16 12:43:57 [conn2] note: no users configured in admin.system.users, allowing localhost access [ connection to gcov1:31100 ] ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 31100, 31101, 31102 ] 31101 number { "useHostName" : true, "oplogSize" : 10, "keyFile" : "jstests/libs/key1", "port" : 31101, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "test-rs0", "dbpath" : "$set-$node", "useHostname" : true, "noJournalPrealloc" : 1, "pathOpts" : { "testName" : "test", "shard" : 0, "node" : 1, "set" : "test-rs0" }, "restart" : undefined } ReplSetTest Starting.... Resetting db path '/data/db/test-rs0-1' Tue Oct 16 12:43:57 shell: started program /data/buildslaves/Linux_64bit_GCov_Nightly/mongo/mongod --oplogSize 10 --keyFile jstests/libs/key1 --port 31101 --noprealloc --smallfiles --rest --replSet test-rs0 --dbpath /data/db/test-rs0-1 --nopreallocj m31101| note: noprealloc may hurt performance in many applications m31101| Tue Oct 16 12:43:57 [initandlisten] MongoDB starting : pid=14842 port=31101 dbpath=/data/db/test-rs0-1 64-bit host=gcov1 m31101| Tue Oct 16 12:43:57 [initandlisten] m31101| Tue Oct 16 12:43:57 [initandlisten] ** NOTE: This is a development version (2.1.3-pre-) of MongoDB. m31101| Tue Oct 16 12:43:57 [initandlisten] ** Not recommended for production. m31101| Tue Oct 16 12:43:57 [initandlisten] m31101| Tue Oct 16 12:43:57 [initandlisten] db version v2.1.3-pre-, pdfile version 4.5 m31101| Tue Oct 16 12:43:57 [initandlisten] git version: 0f8117c897127c9c83fe00ff07b58d87a3ae2c89 m31101| Tue Oct 16 12:43:57 [initandlisten] build info: Linux gcov1 3.2.20-1.29.6.amzn1.x86_64 #1 SMP Tue Jun 12 01:19:28 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31101| Tue Oct 16 12:43:57 [initandlisten] options: { dbpath: "/data/db/test-rs0-1", keyFile: "jstests/libs/key1", noprealloc: true, nopreallocj: true, oplogSize: 10, port: 31101, replSet: "test-rs0", rest: true, smallfiles: true } m31101| Tue Oct 16 12:43:57 [initandlisten] journal dir=/data/db/test-rs0-1/journal m31101| Tue Oct 16 12:43:57 [initandlisten] recover : no journal files present, no recovery needed m31101| Tue Oct 16 12:43:57 [initandlisten] waiting for connections on port 31101 m31101| Tue Oct 16 12:43:57 [websvr] admin web console waiting for connections on port 32101 m31101| Tue Oct 16 12:43:57 [initandlisten] connection accepted from 10.29.160.141:35234 #1 (1 connection now open) m31101| Tue Oct 16 12:43:57 [conn1] authenticate db: local { authenticate: 1, nonce: "33f54f61a845f7b3", user: "__system", key: "a5c0df1ec62a04c195eb1b0ce0f9d339" } m31101| Tue Oct 16 12:43:57 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31101| Tue Oct 16 12:43:57 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31101| Tue Oct 16 12:43:57 [initandlisten] connection accepted from 127.0.0.1:63762 #2 (2 connections now open) m31101| Tue Oct 16 12:43:57 [conn2] note: no users configured in admin.system.users, allowing localhost access [ connection to gcov1:31100, connection to gcov1:31101 ] ReplSetTest n is : 2 ReplSetTest n: 2 ports: [ 31100, 31101, 31102 ] 31102 number { "useHostName" : true, "oplogSize" : 10, "keyFile" : "jstests/libs/key1", "port" : 31102, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "test-rs0", "dbpath" : "$set-$node", "useHostname" : true, "noJournalPrealloc" : 1, "pathOpts" : { "testName" : "test", "shard" : 0, "node" : 2, "set" : "test-rs0" }, "restart" : undefined } ReplSetTest Starting.... Resetting db path '/data/db/test-rs0-2' Tue Oct 16 12:43:57 shell: started program /data/buildslaves/Linux_64bit_GCov_Nightly/mongo/mongod --oplogSize 10 --keyFile jstests/libs/key1 --port 31102 --noprealloc --smallfiles --rest --replSet test-rs0 --dbpath /data/db/test-rs0-2 --nopreallocj m31102| note: noprealloc may hurt performance in many applications m31102| Tue Oct 16 12:43:57 [initandlisten] MongoDB starting : pid=14891 port=31102 dbpath=/data/db/test-rs0-2 64-bit host=gcov1 m31102| Tue Oct 16 12:43:57 [initandlisten] m31102| Tue Oct 16 12:43:57 [initandlisten] ** NOTE: This is a development version (2.1.3-pre-) of MongoDB. m31102| Tue Oct 16 12:43:57 [initandlisten] ** Not recommended for production. m31102| Tue Oct 16 12:43:57 [initandlisten] m31102| Tue Oct 16 12:43:57 [initandlisten] db version v2.1.3-pre-, pdfile version 4.5 m31102| Tue Oct 16 12:43:57 [initandlisten] git version: 0f8117c897127c9c83fe00ff07b58d87a3ae2c89 m31102| Tue Oct 16 12:43:57 [initandlisten] build info: Linux gcov1 3.2.20-1.29.6.amzn1.x86_64 #1 SMP Tue Jun 12 01:19:28 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31102| Tue Oct 16 12:43:57 [initandlisten] options: { dbpath: "/data/db/test-rs0-2", keyFile: "jstests/libs/key1", noprealloc: true, nopreallocj: true, oplogSize: 10, port: 31102, replSet: "test-rs0", rest: true, smallfiles: true } m31102| Tue Oct 16 12:43:57 [initandlisten] journal dir=/data/db/test-rs0-2/journal m31102| Tue Oct 16 12:43:57 [initandlisten] recover : no journal files present, no recovery needed m31102| Tue Oct 16 12:43:57 [initandlisten] waiting for connections on port 31102 m31102| Tue Oct 16 12:43:57 [websvr] admin web console waiting for connections on port 32102 m31102| Tue Oct 16 12:43:57 [initandlisten] connection accepted from 10.29.160.141:50616 #1 (1 connection now open) m31102| Tue Oct 16 12:43:57 [conn1] authenticate db: local { authenticate: 1, nonce: "852301153a8a1092", user: "__system", key: "7bf91824a9ea8c93491ce683e70ef576" } m31102| Tue Oct 16 12:43:57 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31102| Tue Oct 16 12:43:57 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31102| Tue Oct 16 12:43:57 [initandlisten] connection accepted from 127.0.0.1:54324 #2 (2 connections now open) m31102| Tue Oct 16 12:43:57 [conn2] note: no users configured in admin.system.users, allowing localhost access [ connection to gcov1:31100, connection to gcov1:31101, connection to gcov1:31102 ] { "replSetInitiate" : { "_id" : "test-rs0", "members" : [ { "_id" : 0, "host" : "gcov1:31100" }, { "_id" : 1, "host" : "gcov1:31101" }, { "_id" : 2, "host" : "gcov1:31102" } ] } } m31100| Tue Oct 16 12:43:57 [conn2] replSet replSetInitiate admin command received from client m31100| Tue Oct 16 12:43:57 [conn2] replSet replSetInitiate config object parses ok, 3 members specified m31101| Tue Oct 16 12:43:57 [initandlisten] connection accepted from 10.29.160.141:35239 #3 (3 connections now open) m31101| Tue Oct 16 12:43:57 [conn3] authenticate db: local { authenticate: 1, nonce: "fce76616e9615de1", user: "__system", key: "d53fcffbce143ab711327d5362286af0" } m31100| Tue Oct 16 12:43:57 [conn2] replSet replSetInitiate all members seem up m31100| Tue Oct 16 12:43:57 [conn2] ****** m31100| Tue Oct 16 12:43:57 [conn2] creating replication oplog of size: 10MB... m31102| Tue Oct 16 12:43:57 [initandlisten] connection accepted from 10.29.160.141:50619 #3 (3 connections now open) m31102| Tue Oct 16 12:43:57 [conn3] authenticate db: local { authenticate: 1, nonce: "90d4e19b0afab902", user: "__system", key: "850f20312409dafc6ab1d73b82528983" } m31100| Tue Oct 16 12:43:57 [FileAllocator] allocating new datafile /data/db/test-rs0-0/local.ns, filling with zeroes... m31100| Tue Oct 16 12:43:57 [FileAllocator] creating directory /data/db/test-rs0-0/_tmp m31100| Tue Oct 16 12:43:57 [FileAllocator] done allocating datafile /data/db/test-rs0-0/local.ns, size: 16MB, took 0.002 secs m31100| Tue Oct 16 12:43:57 [FileAllocator] allocating new datafile /data/db/test-rs0-0/local.0, filling with zeroes... m31100| Tue Oct 16 12:43:57 [FileAllocator] done allocating datafile /data/db/test-rs0-0/local.0, size: 16MB, took 0.002 secs m31100| Tue Oct 16 12:43:57 [conn2] ****** m31100| Tue Oct 16 12:43:57 [conn2] replSet info saving a newer config version to local.system.replset m31100| Tue Oct 16 12:43:57 [conn2] replSet saveConfigLocally done m31100| Tue Oct 16 12:43:57 [conn2] replSet replSetInitiate config now saved locally. Should come online in about a minute. { "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1 } m31100| Tue Oct 16 12:44:07 [rsStart] replSet I am gcov1:31100 m31100| Tue Oct 16 12:44:07 [rsStart] replSet STARTUP2 m31100| Tue Oct 16 12:44:07 [rsHealthPoll] replSet member gcov1:31101 is up m31101| Tue Oct 16 12:44:07 [rsStart] trying to contact gcov1:31100 m31100| Tue Oct 16 12:44:07 [initandlisten] connection accepted from 10.29.160.141:45254 #3 (3 connections now open) m31100| Tue Oct 16 12:44:07 [conn3] authenticate db: local { authenticate: 1, nonce: "78daa8b407ebcb92", user: "__system", key: "696987c1f26437fabbf8b0645c2fdecf" } m31101| Tue Oct 16 12:44:07 [rsStart] replSet I am gcov1:31101 m31101| Tue Oct 16 12:44:07 [rsStart] replSet got config version 1 from a remote, saving locally m31101| Tue Oct 16 12:44:07 [rsStart] replSet info saving a newer config version to local.system.replset m31101| Tue Oct 16 12:44:07 [FileAllocator] allocating new datafile /data/db/test-rs0-1/local.ns, filling with zeroes... m31101| Tue Oct 16 12:44:07 [FileAllocator] creating directory /data/db/test-rs0-1/_tmp m31101| Tue Oct 16 12:44:07 [FileAllocator] done allocating datafile /data/db/test-rs0-1/local.ns, size: 16MB, took 0.002 secs m31101| Tue Oct 16 12:44:07 [FileAllocator] allocating new datafile /data/db/test-rs0-1/local.0, filling with zeroes... m31101| Tue Oct 16 12:44:07 [FileAllocator] done allocating datafile /data/db/test-rs0-1/local.0, size: 16MB, took 0.002 secs m31101| Tue Oct 16 12:44:07 [rsStart] replSet saveConfigLocally done m31101| Tue Oct 16 12:44:07 [rsStart] replSet STARTUP2 m31101| Tue Oct 16 12:44:07 [rsSync] ****** m31101| Tue Oct 16 12:44:07 [rsSync] creating replication oplog of size: 10MB... m31101| Tue Oct 16 12:44:07 [rsSync] ****** m31101| Tue Oct 16 12:44:07 [rsSync] replSet initial sync pending m31101| Tue Oct 16 12:44:07 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31102| Tue Oct 16 12:44:07 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31100| Tue Oct 16 12:44:08 [rsSync] replSet SECONDARY m31100| Tue Oct 16 12:44:09 [rsHealthPoll] replSet member gcov1:31102 is up m31100| Tue Oct 16 12:44:09 [rsHealthPoll] replSet member gcov1:31101 is now in state STARTUP2 m31100| Tue Oct 16 12:44:09 [rsMgr] not electing self, gcov1:31101 would veto m31100| Tue Oct 16 12:44:09 [rsMgr] not electing self, gcov1:31101 would veto m31101| Tue Oct 16 12:44:09 [rsHealthPoll] replSet member gcov1:31100 is up m31101| Tue Oct 16 12:44:09 [rsHealthPoll] replSet member gcov1:31100 is now in state SECONDARY m31102| Tue Oct 16 12:44:09 [initandlisten] connection accepted from 10.29.160.141:50621 #4 (4 connections now open) m31102| Tue Oct 16 12:44:09 [conn4] authenticate db: local { authenticate: 1, nonce: "52fe68526afe18f", user: "__system", key: "d89b264b62d0f1ce4d3188dfe20a1d40" } m31101| Tue Oct 16 12:44:09 [rsHealthPoll] replSet member gcov1:31102 is up m31100| Tue Oct 16 12:44:15 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes m31102| Tue Oct 16 12:44:17 [rsStart] trying to contact gcov1:31100 m31100| Tue Oct 16 12:44:17 [initandlisten] connection accepted from 10.29.160.141:45256 #4 (4 connections now open) m31100| Tue Oct 16 12:44:17 [conn4] authenticate db: local { authenticate: 1, nonce: "2081793652a851e", user: "__system", key: "0a2d89310c201237ef53a0bf2c84d54a" } m31102| Tue Oct 16 12:44:17 [rsStart] trying to contact gcov1:31101 m31101| Tue Oct 16 12:44:17 [initandlisten] connection accepted from 10.29.160.141:35244 #4 (4 connections now open) m31101| Tue Oct 16 12:44:17 [conn4] authenticate db: local { authenticate: 1, nonce: "df8dc14902fbcede", user: "__system", key: "32374e3edf88bc397889a4bccd2dc944" } m31102| Tue Oct 16 12:44:17 [rsStart] replSet I am gcov1:31102 m31102| Tue Oct 16 12:44:17 [rsStart] replSet got config version 1 from a remote, saving locally m31102| Tue Oct 16 12:44:17 [rsStart] replSet info saving a newer config version to local.system.replset m31102| Tue Oct 16 12:44:17 [FileAllocator] allocating new datafile /data/db/test-rs0-2/local.ns, filling with zeroes... m31102| Tue Oct 16 12:44:17 [FileAllocator] creating directory /data/db/test-rs0-2/_tmp m31102| Tue Oct 16 12:44:17 [FileAllocator] done allocating datafile /data/db/test-rs0-2/local.ns, size: 16MB, took 0.002 secs m31102| Tue Oct 16 12:44:17 [FileAllocator] allocating new datafile /data/db/test-rs0-2/local.0, filling with zeroes... m31102| Tue Oct 16 12:44:17 [FileAllocator] done allocating datafile /data/db/test-rs0-2/local.0, size: 16MB, took 0.011 secs m31102| Tue Oct 16 12:44:17 [rsStart] replSet saveConfigLocally done m31102| Tue Oct 16 12:44:17 [rsStart] replSet STARTUP2 m31102| Tue Oct 16 12:44:17 [rsSync] ****** m31102| Tue Oct 16 12:44:17 [rsSync] creating replication oplog of size: 10MB... m31102| Tue Oct 16 12:44:17 [rsSync] ****** m31102| Tue Oct 16 12:44:17 [rsSync] replSet initial sync pending m31102| Tue Oct 16 12:44:17 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31100| Tue Oct 16 12:44:19 [rsHealthPoll] replSet member gcov1:31102 is now in state STARTUP2 m31100| Tue Oct 16 12:44:19 [rsMgr] not electing self, gcov1:31102 would veto m31101| Tue Oct 16 12:44:19 [rsHealthPoll] replSet member gcov1:31102 is now in state STARTUP2 m31102| Tue Oct 16 12:44:19 [rsHealthPoll] replSet member gcov1:31100 is up m31102| Tue Oct 16 12:44:19 [rsHealthPoll] replSet member gcov1:31100 is now in state SECONDARY m31102| Tue Oct 16 12:44:19 [rsHealthPoll] replSet member gcov1:31101 is up m31102| Tue Oct 16 12:44:19 [rsHealthPoll] replSet member gcov1:31101 is now in state STARTUP2 m31101| Tue Oct 16 12:44:21 [conn3] end connection 10.29.160.141:35239 (3 connections now open) m31101| Tue Oct 16 12:44:21 [initandlisten] connection accepted from 10.29.160.141:35247 #5 (4 connections now open) m31101| Tue Oct 16 12:44:21 [conn5] authenticate db: local { authenticate: 1, nonce: "749dfbbd2c710c6e", user: "__system", key: "c076e7aa33fb065d778f929c939c1b62" } m31100| Tue Oct 16 12:44:23 [conn3] end connection 10.29.160.141:45254 (3 connections now open) m31100| Tue Oct 16 12:44:23 [initandlisten] connection accepted from 10.29.160.141:45261 #5 (4 connections now open) m31100| Tue Oct 16 12:44:23 [conn5] authenticate db: local { authenticate: 1, nonce: "12a702d354217f36", user: "__system", key: "ca57bfd8ce12e39fc179b9f2a6213215" } m31101| Tue Oct 16 12:44:23 [rsSync] replSet initial sync pending m31101| Tue Oct 16 12:44:23 [rsSync] replSet syncing to: gcov1:31100 m31100| Tue Oct 16 12:44:23 [initandlisten] connection accepted from 10.29.160.141:45262 #6 (5 connections now open) m31100| Tue Oct 16 12:44:23 [conn6] authenticate db: local { authenticate: 1, nonce: "bfaee3a3e93271bf", user: "__system", key: "57ba099e3a8df4d021f88edee33ae828" } m31101| Tue Oct 16 12:44:23 [rsSync] build index local.me { _id: 1 } m31101| Tue Oct 16 12:44:23 [rsSync] build index done. scanned 0 total records. 0.001 secs m31101| Tue Oct 16 12:44:23 [rsSync] replSet initial sync drop all databases m31101| Tue Oct 16 12:44:23 [rsSync] dropAllDatabasesExceptLocal 1 m31101| Tue Oct 16 12:44:23 [rsSync] replSet initial sync clone all databases m31101| Tue Oct 16 12:44:23 [rsSync] replSet initial sync cloning db: admin m31100| Tue Oct 16 12:44:23 [initandlisten] connection accepted from 10.29.160.141:45263 #7 (6 connections now open) m31100| Tue Oct 16 12:44:23 [conn7] authenticate db: local { authenticate: 1, nonce: "778705fd4a3ba8c0", user: "__system", key: "6c679ae6edad7604478bef8d45d0b9b8" } m31101| Tue Oct 16 12:44:23 [rsSync] replSet initial sync data copy, starting syncup m31100| Tue Oct 16 12:44:23 [conn7] end connection 10.29.160.141:45263 (5 connections now open) m31101| Tue Oct 16 12:44:23 [rsSync] replSet initial sync building indexes m31101| Tue Oct 16 12:44:23 [rsSync] replSet initial sync cloning indexes for : admin m31100| Tue Oct 16 12:44:23 [initandlisten] connection accepted from 10.29.160.141:45264 #8 (6 connections now open) m31100| Tue Oct 16 12:44:23 [conn8] authenticate db: local { authenticate: 1, nonce: "d0cd03a552da88da", user: "__system", key: "b94c893818b519860199ae9567a23e38" } m31101| Tue Oct 16 12:44:23 [rsSync] replSet initial sync query minValid m31101| Tue Oct 16 12:44:23 [rsSync] replSet initial sync finishing up m31100| Tue Oct 16 12:44:23 [conn8] end connection 10.29.160.141:45264 (5 connections now open) m31101| Tue Oct 16 12:44:23 [rsSync] replSet set minValid=507d568d:b m31101| Tue Oct 16 12:44:23 [rsSync] build index local.replset.minvalid { _id: 1 } m31101| Tue Oct 16 12:44:23 [rsSync] build index done. scanned 0 total records. 0 secs m31101| Tue Oct 16 12:44:23 [rsSync] replSet RECOVERING m31101| Tue Oct 16 12:44:23 [rsSync] replSet initial sync done m31100| Tue Oct 16 12:44:23 [conn6] end connection 10.29.160.141:45262 (4 connections now open) m31102| Tue Oct 16 12:44:23 [rsHealthPoll] replSet member gcov1:31101 is now in state RECOVERING m31100| Tue Oct 16 12:44:25 [rsMgr] replSet info electSelf 0 m31101| Tue Oct 16 12:44:25 [conn5] replSet info voting yea for gcov1:31100 (0) m31102| Tue Oct 16 12:44:25 [conn3] replSet RECOVERING m31102| Tue Oct 16 12:44:25 [conn3] replSet info voting yea for gcov1:31100 (0) m31100| Tue Oct 16 12:44:25 [rsMgr] replSet PRIMARY m31100| Tue Oct 16 12:44:25 [rsHealthPoll] replSet member gcov1:31101 is now in state RECOVERING m31101| Tue Oct 16 12:44:25 [rsHealthPoll] replSet member gcov1:31102 is now in state RECOVERING m31101| Tue Oct 16 12:44:25 [rsHealthPoll] replSet member gcov1:31100 is now in state PRIMARY m31101| Tue Oct 16 12:44:25 [rsBackgroundSync] replSet syncing to: gcov1:31100 m31100| Tue Oct 16 12:44:25 [initandlisten] connection accepted from 10.29.160.141:45267 #9 (5 connections now open) m31100| Tue Oct 16 12:44:25 [conn9] authenticate db: local { authenticate: 1, nonce: "faf6d8977043c784", user: "__system", key: "e8891488d038396523a3e4075260ffca" } m31101| Tue Oct 16 12:44:25 [rsSyncNotifier] replset setting oplog notifier to gcov1:31100 m31100| Tue Oct 16 12:44:25 [initandlisten] connection accepted from 10.29.160.141:45268 #10 (6 connections now open) m31100| Tue Oct 16 12:44:25 [conn10] authenticate db: local { authenticate: 1, nonce: "5a6b2c0737b1f408", user: "__system", key: "7d0f484f2563b58296c43c57d0e58e6b" } m31101| Tue Oct 16 12:44:25 [rsSync] replSet SECONDARY m31102| Tue Oct 16 12:44:25 [rsHealthPoll] replSet member gcov1:31100 is now in state PRIMARY m31102| Tue Oct 16 12:44:25 [rsHealthPoll] replSet member gcov1:31101 is now in state SECONDARY m31100| Tue Oct 16 12:44:25 [FileAllocator] allocating new datafile /data/db/test-rs0-0/admin.ns, filling with zeroes... m31100| Tue Oct 16 12:44:25 [FileAllocator] done allocating datafile /data/db/test-rs0-0/admin.ns, size: 16MB, took 0.004 secs m31100| Tue Oct 16 12:44:25 [FileAllocator] allocating new datafile /data/db/test-rs0-0/admin.0, filling with zeroes... m31100| Tue Oct 16 12:44:25 [FileAllocator] done allocating datafile /data/db/test-rs0-0/admin.0, size: 16MB, took 0.004 secs m31100| Tue Oct 16 12:44:25 [conn2] build index admin.foo { _id: 1 } m31100| Tue Oct 16 12:44:25 [conn2] build index done. scanned 0 total records. 0 secs m31101| Tue Oct 16 12:44:25 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.ns, filling with zeroes... ReplSetTest Timestamp(1350391465000, 1) m31101| Tue Oct 16 12:44:25 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.ns, size: 16MB, took 0.006 secs m31101| Tue Oct 16 12:44:25 [FileAllocator] allocating new datafile /data/db/test-rs0-1/admin.0, filling with zeroes... m31101| Tue Oct 16 12:44:25 [FileAllocator] done allocating datafile /data/db/test-rs0-1/admin.0, size: 16MB, took 0.002 secs m31101| Tue Oct 16 12:44:25 [repl writer worker] build index admin.foo { _id: 1 } m31101| Tue Oct 16 12:44:25 [repl writer worker] build index done. scanned 0 total records. 0 secs { "ts" : Timestamp(1350391465000, 1), "h" : NumberLong("6364361673985228817"), "op" : "i", "ns" : "admin.foo", "o" : { "_id" : ObjectId("507d56a9fa5b8ffccc4d0f78"), "x" : 1 } } ReplSetTest await TS for connection to gcov1:31101 is 1350391465000:1 and latest is 1350391465000:1 ReplSetTest await oplog size for connection to gcov1:31101 is 2 ReplSetTest waiting for connection to gcov1:31102 to have an oplog built. m31100| Tue Oct 16 12:44:27 [rsHealthPoll] replSet member gcov1:31102 is now in state RECOVERING m31100| Tue Oct 16 12:44:27 [rsHealthPoll] replSet member gcov1:31101 is now in state SECONDARY { "ts" : Timestamp(1350391465000, 1), "h" : NumberLong("6364361673985228817"), "op" : "i", "ns" : "admin.foo", "o" : { "_id" : ObjectId("507d56a9fa5b8ffccc4d0f78"), "x" : 1 } } ReplSetTest await TS for connection to gcov1:31101 is 1350391465000:1 and latest is 1350391465000:1 ReplSetTest await oplog size for connection to gcov1:31101 is 2 ReplSetTest waiting for connection to gcov1:31102 to have an oplog built. { "ts" : Timestamp(1350391465000, 1), "h" : NumberLong("6364361673985228817"), "op" : "i", "ns" : "admin.foo", "o" : { "_id" : ObjectId("507d56a9fa5b8ffccc4d0f78"), "x" : 1 } } ReplSetTest await TS for connection to gcov1:31101 is 1350391465000:1 and latest is 1350391465000:1 ReplSetTest await oplog size for connection to gcov1:31101 is 2 ReplSetTest waiting for connection to gcov1:31102 to have an oplog built. { "ts" : Timestamp(1350391465000, 1), "h" : NumberLong("6364361673985228817"), "op" : "i", "ns" : "admin.foo", "o" : { "_id" : ObjectId("507d56a9fa5b8ffccc4d0f78"), "x" : 1 } } ReplSetTest await TS for connection to gcov1:31101 is 1350391465000:1 and latest is 1350391465000:1 ReplSetTest await oplog size for connection to gcov1:31101 is 2 ReplSetTest waiting for connection to gcov1:31102 to have an oplog built. m31100| Tue Oct 16 12:44:33 [conn4] end connection 10.29.160.141:45256 (5 connections now open) m31100| Tue Oct 16 12:44:33 [initandlisten] connection accepted from 10.29.160.141:45277 #11 (6 connections now open) m31100| Tue Oct 16 12:44:33 [conn11] authenticate db: local { authenticate: 1, nonce: "2b287688d5c2b8f9", user: "__system", key: "e011362167f0b8199618d6aed6a3ca19" } m31102| Tue Oct 16 12:44:33 [rsSync] replSet initial sync pending m31102| Tue Oct 16 12:44:33 [rsSync] replSet syncing to: gcov1:31100 m31100| Tue Oct 16 12:44:33 [initandlisten] connection accepted from 10.29.160.141:45278 #12 (7 connections now open) m31100| Tue Oct 16 12:44:33 [conn12] authenticate db: local { authenticate: 1, nonce: "aee4612f9b5719e6", user: "__system", key: "7d23865a9fb5bc1775d6f543b8df22e8" } m31102| Tue Oct 16 12:44:33 [rsSync] build index local.me { _id: 1 } m31102| Tue Oct 16 12:44:33 [rsSync] build index done. scanned 0 total records. 0.001 secs m31102| Tue Oct 16 12:44:33 [rsSync] replSet initial sync drop all databases m31102| Tue Oct 16 12:44:33 [rsSync] dropAllDatabasesExceptLocal 1 m31102| Tue Oct 16 12:44:33 [rsSync] replSet initial sync clone all databases m31102| Tue Oct 16 12:44:33 [rsSync] replSet initial sync cloning db: admin m31100| Tue Oct 16 12:44:33 [initandlisten] connection accepted from 10.29.160.141:45279 #13 (8 connections now open) m31100| Tue Oct 16 12:44:33 [conn13] authenticate db: local { authenticate: 1, nonce: "4afe4e69ebfce503", user: "__system", key: "fddf5a5ace4276ede8e2a1914afd6e89" } m31102| Tue Oct 16 12:44:33 [FileAllocator] allocating new datafile /data/db/test-rs0-2/admin.ns, filling with zeroes... m31102| Tue Oct 16 12:44:33 [FileAllocator] done allocating datafile /data/db/test-rs0-2/admin.ns, size: 16MB, took 0.005 secs m31102| Tue Oct 16 12:44:33 [FileAllocator] allocating new datafile /data/db/test-rs0-2/admin.0, filling with zeroes... m31102| Tue Oct 16 12:44:33 [FileAllocator] done allocating datafile /data/db/test-rs0-2/admin.0, size: 16MB, took 0.002 secs m31102| Tue Oct 16 12:44:33 [rsSync] build index admin.foo { _id: 1 } m31102| Tue Oct 16 12:44:33 [rsSync] fastBuildIndex dupsToDrop:0 m31102| Tue Oct 16 12:44:33 [rsSync] build index done. scanned 1 total records. 0 secs m31102| Tue Oct 16 12:44:33 [rsSync] replSet initial sync data copy, starting syncup m31100| Tue Oct 16 12:44:33 [conn13] end connection 10.29.160.141:45279 (7 connections now open) m31102| Tue Oct 16 12:44:33 [rsSync] replSet initial sync building indexes m31102| Tue Oct 16 12:44:33 [rsSync] replSet initial sync cloning indexes for : admin m31100| Tue Oct 16 12:44:33 [initandlisten] connection accepted from 10.29.160.141:45280 #14 (8 connections now open) m31100| Tue Oct 16 12:44:33 [conn14] authenticate db: local { authenticate: 1, nonce: "894f1d359768acdb", user: "__system", key: "c0c6453627fcb784dba2211b9690bb58" } m31102| Tue Oct 16 12:44:33 [rsSync] replSet initial sync query minValid m31100| Tue Oct 16 12:44:33 [conn14] end connection 10.29.160.141:45280 (7 connections now open) m31102| Tue Oct 16 12:44:33 [rsSync] replSet initial sync finishing up m31102| Tue Oct 16 12:44:33 [rsSync] replSet set minValid=507d56a9:1 m31102| Tue Oct 16 12:44:33 [rsSync] build index local.replset.minvalid { _id: 1 } m31102| Tue Oct 16 12:44:33 [rsSync] build index done. scanned 0 total records. 0 secs m31102| Tue Oct 16 12:44:33 [rsSync] replSet initial sync done m31100| Tue Oct 16 12:44:33 [conn12] end connection 10.29.160.141:45278 (6 connections now open) { "ts" : Timestamp(1350391465000, 1), "h" : NumberLong("6364361673985228817"), "op" : "i", "ns" : "admin.foo", "o" : { "_id" : ObjectId("507d56a9fa5b8ffccc4d0f78"), "x" : 1 } } ReplSetTest await TS for connection to gcov1:31101 is 1350391465000:1 and latest is 1350391465000:1 ReplSetTest await oplog size for connection to gcov1:31101 is 2 { "ts" : Timestamp(1350391465000, 1), "h" : NumberLong("6364361673985228817"), "op" : "i", "ns" : "admin.foo", "o" : { "_id" : ObjectId("507d56a9fa5b8ffccc4d0f78"), "x" : 1 } } ReplSetTest await TS for connection to gcov1:31102 is 1350391465000:1 and latest is 1350391465000:1 ReplSetTest await oplog size for connection to gcov1:31102 is 1 ReplSetTest await synced=true Tue Oct 16 12:44:33 starting new replica set monitor for replica set test-rs0 with seed of gcov1:31100,gcov1:31101,gcov1:31102 m31100| Tue Oct 16 12:44:33 [initandlisten] connection accepted from 10.29.160.141:45282 #15 (7 connections now open) Tue Oct 16 12:44:33 successfully connected to seed gcov1:31100 for replica set test-rs0 Tue Oct 16 12:44:33 changing hosts to { 0: "gcov1:31100", 1: "gcov1:31102", 2: "gcov1:31101" } from test-rs0/ Tue Oct 16 12:44:33 trying to add new host gcov1:31100 to replica set test-rs0 m31100| Tue Oct 16 12:44:33 [initandlisten] connection accepted from 10.29.160.141:45283 #16 (8 connections now open) Tue Oct 16 12:44:33 successfully connected to new host gcov1:31100 in replica set test-rs0 Tue Oct 16 12:44:33 trying to add new host gcov1:31101 to replica set test-rs0 m31101| Tue Oct 16 12:44:33 [initandlisten] connection accepted from 10.29.160.141:35271 #6 (5 connections now open) Tue Oct 16 12:44:33 successfully connected to new host gcov1:31101 in replica set test-rs0 Tue Oct 16 12:44:33 trying to add new host gcov1:31102 to replica set test-rs0 m31102| Tue Oct 16 12:44:33 [initandlisten] connection accepted from 10.29.160.141:50651 #5 (5 connections now open) Tue Oct 16 12:44:33 successfully connected to new host gcov1:31102 in replica set test-rs0 m31100| Tue Oct 16 12:44:33 [initandlisten] connection accepted from 10.29.160.141:45286 #17 (9 connections now open) m31100| Tue Oct 16 12:44:33 [conn15] end connection 10.29.160.141:45282 (8 connections now open) Tue Oct 16 12:44:33 Primary for replica set test-rs0 changed to gcov1:31100 m31101| Tue Oct 16 12:44:33 [initandlisten] connection accepted from 10.29.160.141:35274 #7 (6 connections now open) m31102| Tue Oct 16 12:44:33 [initandlisten] connection accepted from 10.29.160.141:50654 #6 (6 connections now open) Tue Oct 16 12:44:33 replica set monitor for replica set test-rs0 started, address is test-rs0/gcov1:31100,gcov1:31101,gcov1:31102 Tue Oct 16 12:44:33 [ReplicaSetMonitorWatcher] starting m31100| Tue Oct 16 12:44:33 [initandlisten] connection accepted from 10.29.160.141:45289 #18 (9 connections now open) Resetting db path '/data/db/test-config0' Tue Oct 16 12:44:33 shell: started program /data/buildslaves/Linux_64bit_GCov_Nightly/mongo/mongod --port 29000 --dbpath /data/db/test-config0 --keyFile jstests/libs/key1 --configsvr --nopreallocj m29000| Tue Oct 16 12:44:34 [initandlisten] MongoDB starting : pid=15025 port=29000 dbpath=/data/db/test-config0 64-bit host=gcov1 m29000| Tue Oct 16 12:44:34 [initandlisten] m29000| Tue Oct 16 12:44:34 [initandlisten] ** NOTE: This is a development version (2.1.3-pre-) of MongoDB. m29000| Tue Oct 16 12:44:34 [initandlisten] ** Not recommended for production. m29000| Tue Oct 16 12:44:34 [initandlisten] m29000| Tue Oct 16 12:44:34 [initandlisten] db version v2.1.3-pre-, pdfile version 4.5 m29000| Tue Oct 16 12:44:34 [initandlisten] git version: 0f8117c897127c9c83fe00ff07b58d87a3ae2c89 m29000| Tue Oct 16 12:44:34 [initandlisten] build info: Linux gcov1 3.2.20-1.29.6.amzn1.x86_64 #1 SMP Tue Jun 12 01:19:28 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m29000| Tue Oct 16 12:44:34 [initandlisten] options: { configsvr: true, dbpath: "/data/db/test-config0", keyFile: "jstests/libs/key1", nopreallocj: true, port: 29000 } m29000| Tue Oct 16 12:44:34 [initandlisten] journal dir=/data/db/test-config0/journal m29000| Tue Oct 16 12:44:34 [initandlisten] recover : no journal files present, no recovery needed m29000| Tue Oct 16 12:44:34 [initandlisten] waiting for connections on port 29000 m29000| Tue Oct 16 12:44:34 [websvr] admin web console waiting for connections on port 30000 m29000| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 127.0.0.1:64013 #1 (1 connection now open) m29000| Tue Oct 16 12:44:34 [conn1] note: no users configured in admin.system.users, allowing localhost access "gcov1:29000" ShardingTest test : { "config" : "gcov1:29000", "shards" : [ connection to test-rs0/gcov1:31100,gcov1:31101,gcov1:31102 ] } m29000| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:44869 #2 (2 connections now open) Tue Oct 16 12:44:34 shell: started program /data/buildslaves/Linux_64bit_GCov_Nightly/mongo/mongos --port 30999 --configdb gcov1:29000 --keyFile jstests/libs/key1 m30999| Tue Oct 16 12:44:34 warning: running with 1 config server should be done only for testing purposes and is not recommended for production m30999| Tue Oct 16 12:44:34 [mongosMain] MongoS version 2.1.3-pre- starting: pid=15041 port=30999 64-bit host=gcov1 (--help for usage) m30999| Tue Oct 16 12:44:34 [mongosMain] git version: 0f8117c897127c9c83fe00ff07b58d87a3ae2c89 m30999| Tue Oct 16 12:44:34 [mongosMain] build info: Linux gcov1 3.2.20-1.29.6.amzn1.x86_64 #1 SMP Tue Jun 12 01:19:28 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m30999| Tue Oct 16 12:44:34 [mongosMain] options: { configdb: "gcov1:29000", keyFile: "jstests/libs/key1", port: 30999 } m29000| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:44871 #3 (3 connections now open) m29000| Tue Oct 16 12:44:34 [conn3] authenticate db: local { authenticate: 1, nonce: "bc8f4d0d6efbe6f7", user: "__system", key: "b8feedda3d753571b7bfbab14ba538ae" } m29000| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:44872 #4 (4 connections now open) m29000| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:44873 #5 (5 connections now open) m29000| Tue Oct 16 12:44:34 [conn4] authenticate db: local { authenticate: 1, nonce: "9a29fe766243340b", user: "__system", key: "d77782f95177ee9be0f8486797690edf" } m29000| Tue Oct 16 12:44:34 [conn5] authenticate db: local { authenticate: 1, nonce: "20eed231ae833060", user: "__system", key: "54d877dde83910cfad968a28fa733170" } m29000| Tue Oct 16 12:44:34 [FileAllocator] allocating new datafile /data/db/test-config0/config.ns, filling with zeroes... m29000| Tue Oct 16 12:44:34 [FileAllocator] creating directory /data/db/test-config0/_tmp m29000| Tue Oct 16 12:44:34 [FileAllocator] done allocating datafile /data/db/test-config0/config.ns, size: 16MB, took 0.002 secs m29000| Tue Oct 16 12:44:34 [FileAllocator] allocating new datafile /data/db/test-config0/config.0, filling with zeroes... m29000| Tue Oct 16 12:44:34 [FileAllocator] done allocating datafile /data/db/test-config0/config.0, size: 16MB, took 0.002 secs m29000| Tue Oct 16 12:44:34 [FileAllocator] allocating new datafile /data/db/test-config0/config.1, filling with zeroes... m29000| Tue Oct 16 12:44:34 [conn5] build index config.version { _id: 1 } m29000| Tue Oct 16 12:44:34 [conn5] build index done. scanned 0 total records. 0.001 secs m29000| Tue Oct 16 12:44:34 [FileAllocator] done allocating datafile /data/db/test-config0/config.1, size: 32MB, took 0.002 secs m29000| Tue Oct 16 12:44:34 [conn4] build index config.settings { _id: 1 } m30999| Tue Oct 16 12:44:34 [websvr] admin web console waiting for connections on port 31999 m30999| Tue Oct 16 12:44:34 [mongosMain] waiting for connections on port 30999 m30999| Tue Oct 16 12:44:34 [Balancer] about to contact config servers and shards m29000| Tue Oct 16 12:44:34 [conn4] build index done. scanned 0 total records. 0.002 secs m30999| Tue Oct 16 12:44:34 [Balancer] config servers and shards contacted successfully m30999| Tue Oct 16 12:44:34 [Balancer] balancer id: gcov1:30999 started at Oct 16 12:44:34 m30999| Tue Oct 16 12:44:34 [Balancer] created new distributed lock for balancer on gcov1:29000 ( lock timeout : 900000, ping interval : 30000, process : 0 ) m29000| Tue Oct 16 12:44:34 [conn4] build index config.chunks { _id: 1 } m29000| Tue Oct 16 12:44:34 [conn4] build index done. scanned 0 total records. 0 secs m29000| Tue Oct 16 12:44:34 [conn4] info: creating collection config.chunks on add index m29000| Tue Oct 16 12:44:34 [conn4] build index config.chunks { ns: 1, min: 1 } m29000| Tue Oct 16 12:44:34 [conn4] build index done. scanned 0 total records. 0 secs m29000| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:44874 #6 (6 connections now open) m29000| Tue Oct 16 12:44:34 [conn5] build index config.mongos { _id: 1 } m29000| Tue Oct 16 12:44:34 [conn5] build index done. scanned 0 total records. 0 secs m29000| Tue Oct 16 12:44:34 [conn4] build index config.chunks { ns: 1, shard: 1, min: 1 } m29000| Tue Oct 16 12:44:34 [conn6] authenticate db: local { authenticate: 1, nonce: "e660613702395072", user: "__system", key: "46841e40e0e0b79941e17b534115135b" } m29000| Tue Oct 16 12:44:34 [conn4] build index done. scanned 0 total records. 0 secs m30999| Tue Oct 16 12:44:34 [Balancer] distributed lock 'balancer/gcov1:30999:1350391474:1804289383' acquired, ts : 507d56b204bb5fa99d70aeef m29000| Tue Oct 16 12:44:34 [conn6] build index config.locks { _id: 1 } m29000| Tue Oct 16 12:44:34 [conn6] build index done. scanned 0 total records. 0 secs m30999| Tue Oct 16 12:44:34 [Balancer] distributed lock 'balancer/gcov1:30999:1350391474:1804289383' unlocked. m30999| Tue Oct 16 12:44:34 [LockPinger] creating distributed lock ping thread for gcov1:29000 and process gcov1:30999:1350391474:1804289383 (sleeping for 30000ms) m29000| Tue Oct 16 12:44:34 [conn4] build index config.chunks { ns: 1, lastmod: 1 } m29000| Tue Oct 16 12:44:34 [conn4] build index done. scanned 0 total records. 0 secs m29000| Tue Oct 16 12:44:34 [conn4] build index config.shards { _id: 1 } m29000| Tue Oct 16 12:44:34 [conn4] build index done. scanned 0 total records. 0 secs m29000| Tue Oct 16 12:44:34 [conn4] info: creating collection config.shards on add index m29000| Tue Oct 16 12:44:34 [conn4] build index config.shards { host: 1 } m29000| Tue Oct 16 12:44:34 [conn4] build index done. scanned 0 total records. 0 secs m29000| Tue Oct 16 12:44:34 [conn4] build index config.lockpings { _id: 1 } m29000| Tue Oct 16 12:44:34 [conn4] build index done. scanned 0 total records. 0 secs m29000| Tue Oct 16 12:44:34 [conn4] build index config.lockpings { ping: 1 } m29000| Tue Oct 16 12:44:34 [conn4] build index done. scanned 1 total records. 0 secs m30999| Tue Oct 16 12:44:34 [mongosMain] connection accepted from 127.0.0.1:54756 #1 (1 connection now open) ShardingTest undefined going to add shard : test-rs0/gcov1:31100,gcov1:31101,gcov1:31102 m30999| Tue Oct 16 12:44:34 [conn1] couldn't find database [admin] in config db m30999| Tue Oct 16 12:44:34 [conn1] put [admin] on: config:gcov1:29000 m29000| Tue Oct 16 12:44:34 [conn4] build index config.databases { _id: 1 } m29000| Tue Oct 16 12:44:34 [conn4] build index done. scanned 0 total records. 0 secs m30999| Tue Oct 16 12:44:34 [conn1] creating WriteBackListener for: gcov1:29000 serverID: 507d56b204bb5fa99d70aeee m29000| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:44876 #7 (7 connections now open) m29000| Tue Oct 16 12:44:34 [conn7] authenticate db: local { authenticate: 1, nonce: "f6019c1b599725ab", user: "__system", key: "64ac55fdadc407e2406f2fb79278ff30" } m30999| Tue Oct 16 12:44:34 [conn1] note: no users configured in admin.system.users, allowing localhost access m30999| Tue Oct 16 12:44:34 [conn1] starting new replica set monitor for replica set test-rs0 with seed of gcov1:31100,gcov1:31101,gcov1:31102 m31100| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:45301 #19 (10 connections now open) m30999| Tue Oct 16 12:44:34 [conn1] successfully connected to seed gcov1:31100 for replica set test-rs0 m30999| Tue Oct 16 12:44:34 [conn1] changing hosts to { 0: "gcov1:31100", 1: "gcov1:31102", 2: "gcov1:31101" } from test-rs0/ m30999| Tue Oct 16 12:44:34 [conn1] trying to add new host gcov1:31100 to replica set test-rs0 m31100| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:45302 #20 (11 connections now open) m30999| Tue Oct 16 12:44:34 [conn1] successfully connected to new host gcov1:31100 in replica set test-rs0 m30999| Tue Oct 16 12:44:34 [conn1] trying to add new host gcov1:31101 to replica set test-rs0 m31101| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:35290 #8 (7 connections now open) m30999| Tue Oct 16 12:44:34 [conn1] successfully connected to new host gcov1:31101 in replica set test-rs0 m30999| Tue Oct 16 12:44:34 [conn1] trying to add new host gcov1:31102 to replica set test-rs0 m31102| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:50670 #7 (7 connections now open) m30999| Tue Oct 16 12:44:34 [conn1] successfully connected to new host gcov1:31102 in replica set test-rs0 m31100| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:45305 #21 (12 connections now open) m31100| Tue Oct 16 12:44:34 [conn21] authenticate db: local { authenticate: 1, nonce: "4e8f4c97671bb0ba", user: "__system", key: "a83774d68f893f497715a2e70e67f49b" } m31100| Tue Oct 16 12:44:34 [conn19] end connection 10.29.160.141:45301 (11 connections now open) m30999| Tue Oct 16 12:44:34 [conn1] Primary for replica set test-rs0 changed to gcov1:31100 m31101| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:35293 #9 (8 connections now open) m31101| Tue Oct 16 12:44:34 [conn9] authenticate db: local { authenticate: 1, nonce: "38e2b81e3c369a30", user: "__system", key: "c8d3894ad75d7ef2be8b3797a57ed58c" } m31102| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:50673 #8 (8 connections now open) m31102| Tue Oct 16 12:44:34 [conn8] authenticate db: local { authenticate: 1, nonce: "dab0d29f9fc9fbce", user: "__system", key: "f3cbb177f4a4ace4f8506c7eebc59ef6" } m30999| Tue Oct 16 12:44:34 [conn1] replica set monitor for replica set test-rs0 started, address is test-rs0/gcov1:31100,gcov1:31101,gcov1:31102 m30999| Tue Oct 16 12:44:34 [ReplicaSetMonitorWatcher] starting m31100| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:45308 #22 (12 connections now open) m31100| Tue Oct 16 12:44:34 [conn22] authenticate db: local { authenticate: 1, nonce: "34884500a06e65af", user: "__system", key: "63fa88f6adde98bf1be7c8caf8047443" } m30999| Tue Oct 16 12:44:34 [conn1] going to add shard: { _id: "test-rs0", host: "test-rs0/gcov1:31100,gcov1:31101,gcov1:31102" } { "shardAdded" : "test-rs0", "ok" : 1 } { "user" : "user", "readOnly" : false, "pwd" : "3bcfc22a1cd6be41bc7814c13d3ce94c", "_id" : ObjectId("507d56b2fa5b8ffccc4d0f7b") } m29000| Tue Oct 16 12:44:34 [FileAllocator] allocating new datafile /data/db/test-config0/admin.ns, filling with zeroes... m29000| Tue Oct 16 12:44:34 [FileAllocator] done allocating datafile /data/db/test-config0/admin.ns, size: 16MB, took 0.002 secs m29000| Tue Oct 16 12:44:34 [FileAllocator] allocating new datafile /data/db/test-config0/admin.0, filling with zeroes... m29000| Tue Oct 16 12:44:34 [FileAllocator] done allocating datafile /data/db/test-config0/admin.0, size: 16MB, took 0.002 secs m29000| Tue Oct 16 12:44:34 [conn7] build index admin.system.users { _id: 1 } m29000| Tue Oct 16 12:44:34 [conn7] build index done. scanned 0 total records. 0 secs m29000| Tue Oct 16 12:44:34 [FileAllocator] allocating new datafile /data/db/test-config0/admin.1, filling with zeroes... addUser succeeded, but cannot wait for replication since we no longer have auth m30999| Tue Oct 16 12:44:34 [conn1] authenticate db: admin { authenticate: 1, nonce: "4708a05a8619d7e1", user: "user", key: "b67835dcaf2fb0e7df59f92db8d73168" } m29000| Tue Oct 16 12:44:34 [FileAllocator] done allocating datafile /data/db/test-config0/admin.1, size: 32MB, took 0.003 secs { "user" : "user", "readOnly" : false, "pwd" : "3bcfc22a1cd6be41bc7814c13d3ce94c", "_id" : ObjectId("507d56b2fa5b8ffccc4d0f7c") } m31100| Tue Oct 16 12:44:34 [conn2] build index admin.system.users { _id: 1 } m31100| Tue Oct 16 12:44:34 [conn2] build index done. scanned 0 total records. 0.001 secs addUser succeeded, but cannot wait for replication since we no longer have auth m30999| Tue Oct 16 12:44:34 [conn1] couldn't find database [AAAAA] in config db m30999| Tue Oct 16 12:44:34 [conn1] put [AAAAA] on: test-rs0:test-rs0/gcov1:31100,gcov1:31101,gcov1:31102 m31101| Tue Oct 16 12:44:34 [repl writer worker] build index admin.system.users { _id: 1 } m31101| Tue Oct 16 12:44:34 [repl writer worker] build index done. scanned 0 total records. 0.001 secs m30999| Tue Oct 16 12:44:34 [conn1] DROP: AAAAA.user m31100| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:45311 #23 (13 connections now open) m31100| Tue Oct 16 12:44:34 [conn23] authenticate db: local { authenticate: 1, nonce: "3775fb31beb71322", user: "__system", key: "0dd5645fbdf616676c48893b1e72399b" } m30999| Tue Oct 16 12:44:34 [conn1] creating WriteBackListener for: gcov1:31100 serverID: 507d56b204bb5fa99d70aeee m30999| Tue Oct 16 12:44:34 [conn1] creating WriteBackListener for: gcov1:31101 serverID: 507d56b204bb5fa99d70aeee m30999| Tue Oct 16 12:44:34 [conn1] creating WriteBackListener for: gcov1:31102 serverID: 507d56b204bb5fa99d70aeee m31100| Tue Oct 16 12:44:34 [conn23] CMD: drop AAAAA.user Awaiting gcov1:31101 to be { "ok" : true, "secondary" : true } for connection to gcov1:30999 (rs: undefined) { "test-rs0" : { "hosts" : [ { "addr" : "gcov1:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31102", "ok" : false, "ismaster" : false, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } Awaiting gcov1:31102 to be { "ok" : true, "secondary" : true } for connection to gcov1:30999 (rs: undefined) { "test-rs0" : { "hosts" : [ { "addr" : "gcov1:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31102", "ok" : false, "ismaster" : false, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } m31102| Tue Oct 16 12:44:34 [rsBackgroundSync] replSet syncing to: gcov1:31100 m31100| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:45312 #24 (14 connections now open) m31100| Tue Oct 16 12:44:34 [conn24] authenticate db: local { authenticate: 1, nonce: "ebb7cdfa214e301a", user: "__system", key: "7e3a02cc5016cbf9938c6276e3703d1d" } m31102| Tue Oct 16 12:44:34 [rsSyncNotifier] replset setting oplog notifier to gcov1:31100 m31100| Tue Oct 16 12:44:34 [initandlisten] connection accepted from 10.29.160.141:45313 #25 (15 connections now open) m31100| Tue Oct 16 12:44:34 [conn25] authenticate db: local { authenticate: 1, nonce: "91b464aa44baf3f", user: "__system", key: "51bf9ef4990e3044bcc6f36f3a9f955f" } m31102| Tue Oct 16 12:44:34 [repl writer worker] build index admin.system.users { _id: 1 } m31102| Tue Oct 16 12:44:34 [repl writer worker] build index done. scanned 0 total records. 0 secs m31102| Tue Oct 16 12:44:35 [rsSync] replSet SECONDARY { "test-rs0" : { "hosts" : [ { "addr" : "gcov1:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31102", "ok" : false, "ismaster" : false, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } m31102| Tue Oct 16 12:44:37 [conn3] end connection 10.29.160.141:50619 (7 connections now open) m31102| Tue Oct 16 12:44:37 [initandlisten] connection accepted from 10.29.160.141:50682 #9 (8 connections now open) m31102| Tue Oct 16 12:44:37 [conn9] authenticate db: local { authenticate: 1, nonce: "592bb90b10819c19", user: "__system", key: "e093e0f8901ea68e2e433a6233a65e79" } m31100| Tue Oct 16 12:44:37 [rsHealthPoll] replSet member gcov1:31102 is now in state SECONDARY m31102| Tue Oct 16 12:44:37 [conn4] end connection 10.29.160.141:50621 (7 connections now open) m31102| Tue Oct 16 12:44:37 [initandlisten] connection accepted from 10.29.160.141:50683 #10 (8 connections now open) m31102| Tue Oct 16 12:44:37 [conn10] authenticate db: local { authenticate: 1, nonce: "9f7cbaaa838b4ffd", user: "__system", key: "bf9d2079a5373870ef3e8276ef9ee9e8" } m31101| Tue Oct 16 12:44:37 [rsHealthPoll] replSet member gcov1:31102 is now in state SECONDARY { "test-rs0" : { "hosts" : [ { "addr" : "gcov1:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31102", "ok" : false, "ismaster" : false, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } m30999| Tue Oct 16 12:44:40 [Balancer] distributed lock 'balancer/gcov1:30999:1350391474:1804289383' acquired, ts : 507d56b804bb5fa99d70aef0 m30999| Tue Oct 16 12:44:40 [Balancer] distributed lock 'balancer/gcov1:30999:1350391474:1804289383' unlocked. { "test-rs0" : { "hosts" : [ { "addr" : "gcov1:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31102", "ok" : false, "ismaster" : false, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "gcov1:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31102", "ok" : false, "ismaster" : false, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } m31100| Tue Oct 16 12:44:44 [initandlisten] connection accepted from 10.29.160.141:45324 #26 (16 connections now open) m31100| Tue Oct 16 12:44:44 [conn26] authenticate db: local { authenticate: 1, nonce: "e0c7ba83f9109de3", user: "__system", key: "379ffa4e05fcbc7b244789bc93c16bba" } m31101| Tue Oct 16 12:44:44 [initandlisten] connection accepted from 10.29.160.141:35312 #10 (9 connections now open) m31101| Tue Oct 16 12:44:44 [conn10] authenticate db: local { authenticate: 1, nonce: "b8e863ffb640069c", user: "__system", key: "15da8665a0a6f2b921e89eb42a2ed7cc" } m31102| Tue Oct 16 12:44:44 [initandlisten] connection accepted from 10.29.160.141:50692 #11 (9 connections now open) m31102| Tue Oct 16 12:44:44 [conn11] authenticate db: local { authenticate: 1, nonce: "13cfbdc503dd3ccb", user: "__system", key: "5f4a00a24ff5c6fa3fc7caaaf2b7385e" } { "test-rs0" : { "hosts" : [ { "addr" : "gcov1:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31102", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } m31100| Tue Oct 16 12:44:44 [FileAllocator] allocating new datafile /data/db/test-rs0-0/AAAAA.ns, filling with zeroes... m31100| Tue Oct 16 12:44:44 [FileAllocator] done allocating datafile /data/db/test-rs0-0/AAAAA.ns, size: 16MB, took 0.006 secs m31100| Tue Oct 16 12:44:44 [FileAllocator] allocating new datafile /data/db/test-rs0-0/AAAAA.0, filling with zeroes... m31100| Tue Oct 16 12:44:44 [FileAllocator] done allocating datafile /data/db/test-rs0-0/AAAAA.0, size: 16MB, took 0.003 secs m31100| Tue Oct 16 12:44:44 [conn23] build index AAAAA.user { _id: 1 } m31100| Tue Oct 16 12:44:44 [conn23] build index done. scanned 0 total records. 0 secs m31102| Tue Oct 16 12:44:44 [FileAllocator] allocating new datafile /data/db/test-rs0-2/AAAAA.ns, filling with zeroes... m31101| Tue Oct 16 12:44:44 [FileAllocator] allocating new datafile /data/db/test-rs0-1/AAAAA.ns, filling with zeroes... m31101| Tue Oct 16 12:44:44 [FileAllocator] done allocating datafile /data/db/test-rs0-1/AAAAA.ns, size: 16MB, took 0.006 secs m31102| Tue Oct 16 12:44:44 [FileAllocator] done allocating datafile /data/db/test-rs0-2/AAAAA.ns, size: 16MB, took 0.007 secs m31101| Tue Oct 16 12:44:44 [FileAllocator] allocating new datafile /data/db/test-rs0-1/AAAAA.0, filling with zeroes... m31101| Tue Oct 16 12:44:44 [FileAllocator] done allocating datafile /data/db/test-rs0-1/AAAAA.0, size: 16MB, took 0.002 secs m31101| Tue Oct 16 12:44:44 [repl writer worker] build index AAAAA.user { _id: 1 } m31101| Tue Oct 16 12:44:44 [repl writer worker] build index done. scanned 0 total records. 0 secs m31102| Tue Oct 16 12:44:44 [FileAllocator] allocating new datafile /data/db/test-rs0-2/AAAAA.0, filling with zeroes... m31102| Tue Oct 16 12:44:44 [FileAllocator] done allocating datafile /data/db/test-rs0-2/AAAAA.0, size: 16MB, took 0.005 secs m31102| Tue Oct 16 12:44:44 [repl writer worker] build index AAAAA.user { _id: 1 } m31102| Tue Oct 16 12:44:44 [repl writer worker] build index done. scanned 0 total records. 0 secs ---- First query to SEC ---- m31101| Tue Oct 16 12:44:44 [initandlisten] connection accepted from 10.29.160.141:35316 #11 (10 connections now open) m31101| Tue Oct 16 12:44:44 [conn11] authenticate db: local { authenticate: 1, nonce: "86a2648e68e5d034", user: "__system", key: "98f89907451afd51fe3e4d22589d7bfc" } m31101| Tue Oct 16 12:44:44 [initandlisten] connection accepted from 10.29.160.141:35317 #12 (11 connections now open) ReplSetTest n: 0 ports: [ 31100, 31101, 31102 ] 31100 number ReplSetTest stop *** Shutting down mongod in port 31100 *** m31100| Tue Oct 16 12:44:44 got signal 15 (Terminated), will terminate after current cmd ends m31100| Tue Oct 16 12:44:44 [interruptThread] now exiting m31100| Tue Oct 16 12:44:44 dbexit: m31100| Tue Oct 16 12:44:44 [interruptThread] shutdown: going to close listening sockets... m31100| Tue Oct 16 12:44:44 [interruptThread] closing listening socket: 13 m31100| Tue Oct 16 12:44:44 [interruptThread] closing listening socket: 14 m31100| Tue Oct 16 12:44:44 [interruptThread] closing listening socket: 15 m31100| Tue Oct 16 12:44:44 [interruptThread] removing socket file: /tmp/mongodb-31100.sock m31100| Tue Oct 16 12:44:44 [interruptThread] shutdown: going to flush diaglog... m31100| Tue Oct 16 12:44:44 [interruptThread] shutdown: going to close sockets... m31100| Tue Oct 16 12:44:44 [interruptThread] shutdown: waiting for fs preallocator... m31100| Tue Oct 16 12:44:44 [interruptThread] shutdown: lock for final commit... m31100| Tue Oct 16 12:44:44 [interruptThread] shutdown: final commit... m31102| Tue Oct 16 12:44:44 [conn9] end connection 10.29.160.141:50682 (8 connections now open) m31102| Tue Oct 16 12:44:44 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: gcov1:31100 m31102| Tue Oct 16 12:44:44 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: gcov1:31100 m31100| Tue Oct 16 12:44:44 [conn1] end connection 10.29.160.141:45244 (15 connections now open) m31101| Tue Oct 16 12:44:44 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: gcov1:31100 m31101| Tue Oct 16 12:44:44 [conn5] end connection 10.29.160.141:35247 (10 connections now open) m31101| Tue Oct 16 12:44:44 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: gcov1:31100 m30999| Tue Oct 16 12:44:44 [WriteBackListener-gcov1:31100] DBClientCursor::init call() failed m30999| Tue Oct 16 12:44:44 [WriteBackListener-gcov1:31100] WriteBackListener exception : DBClientBase::findN: transport error: gcov1:31100 ns: admin.$cmd query: { writebacklisten: ObjectId('507d56b204bb5fa99d70aeee'), $auth: { local: { __system: 2 } } } m31100| Tue Oct 16 12:44:44 [interruptThread] shutdown: closing all files... m31100| Tue Oct 16 12:44:44 [interruptThread] closeAllFiles() finished m31100| Tue Oct 16 12:44:44 [interruptThread] journalCleanup... m31100| Tue Oct 16 12:44:44 [interruptThread] removeJournalFiles m31100| Tue Oct 16 12:44:44 [interruptThread] shutdown: removing fs lock... m31100| Tue Oct 16 12:44:44 dbexit: really exiting now m31101| Tue Oct 16 12:44:45 [rsHealthPoll] DBClientCursor::init call() failed m31101| Tue Oct 16 12:44:45 [rsHealthPoll] replSet info gcov1:31100 is down (or slow to respond): DBClientBase::findN: transport error: gcov1:31100 ns: admin.$cmd query: { replSetHeartbeat: "test-rs0", v: 1, pv: 1, checkEmpty: false, from: "gcov1:31101", $auth: { local: { __system: 2 } } } m31101| Tue Oct 16 12:44:45 [rsHealthPoll] replSet member gcov1:31100 is now in state DOWN m31101| Tue Oct 16 12:44:45 [rsMgr] not electing self, gcov1:31102 would veto Tue Oct 16 12:44:45 shell: stopped mongo program on port 31100 ReplSetTest n: 1 ports: [ 31100, 31101, 31102 ] 31101 number ReplSetTest stop *** Shutting down mongod in port 31101 *** m31101| Tue Oct 16 12:44:45 got signal 15 (Terminated), will terminate after current cmd ends m31101| Tue Oct 16 12:44:45 [interruptThread] now exiting m31101| Tue Oct 16 12:44:45 dbexit: m31101| Tue Oct 16 12:44:45 [interruptThread] shutdown: going to close listening sockets... m31101| Tue Oct 16 12:44:45 [interruptThread] closing listening socket: 16 m31101| Tue Oct 16 12:44:45 [interruptThread] closing listening socket: 17 m31101| Tue Oct 16 12:44:45 [interruptThread] closing listening socket: 18 m31101| Tue Oct 16 12:44:45 [interruptThread] removing socket file: /tmp/mongodb-31101.sock m31101| Tue Oct 16 12:44:45 [interruptThread] shutdown: going to flush diaglog... m31101| Tue Oct 16 12:44:45 [interruptThread] shutdown: going to close sockets... m31101| Tue Oct 16 12:44:45 [interruptThread] shutdown: waiting for fs preallocator... m31101| Tue Oct 16 12:44:45 [interruptThread] shutdown: lock for final commit... m31101| Tue Oct 16 12:44:45 [interruptThread] shutdown: final commit... m30999| Tue Oct 16 12:44:45 [WriteBackListener-gcov1:31100] DBClientCursor::init call() failed m30999| Tue Oct 16 12:44:45 [WriteBackListener-gcov1:31100] WriteBackListener exception : DBClientBase::findN: transport error: gcov1:31100 ns: admin.$cmd query: { writebacklisten: ObjectId('507d56b204bb5fa99d70aeee'), $auth: { local: { __system: 2 } } } m30999| Tue Oct 16 12:44:45 [WriteBackListener-gcov1:31101] DBClientCursor::init call() failed m30999| Tue Oct 16 12:44:45 [WriteBackListener-gcov1:31101] WriteBackListener exception : DBClientBase::findN: transport error: gcov1:31101 ns: admin.$cmd query: { writebacklisten: ObjectId('507d56b204bb5fa99d70aeee'), $auth: { local: { __system: 2 } } } m31101| Tue Oct 16 12:44:45 [conn1] end connection 10.29.160.141:35234 (9 connections now open) m31102| Tue Oct 16 12:44:45 [conn10] end connection 10.29.160.141:50683 (7 connections now open) m31101| Tue Oct 16 12:44:45 [conn4] end connection 10.29.160.141:35244 (8 connections now open) m31102| Tue Oct 16 12:44:45 [rsHealthPoll] DBClientCursor::init call() failed m31102| Tue Oct 16 12:44:45 [rsHealthPoll] replSet info gcov1:31101 is down (or slow to respond): DBClientBase::findN: transport error: gcov1:31101 ns: admin.$cmd query: { replSetHeartbeat: "test-rs0", v: 1, pv: 1, checkEmpty: false, from: "gcov1:31102", $auth: { local: { __system: 2 } } } m31102| Tue Oct 16 12:44:45 [rsHealthPoll] replSet member gcov1:31101 is now in state DOWN m31102| Tue Oct 16 12:44:45 [rsHealthPoll] DBClientCursor::init call() failed m31102| Tue Oct 16 12:44:45 [rsHealthPoll] replSet info gcov1:31100 is down (or slow to respond): DBClientBase::findN: transport error: gcov1:31100 ns: admin.$cmd query: { replSetHeartbeat: "test-rs0", v: 1, pv: 1, checkEmpty: false, from: "gcov1:31102", $auth: { local: { __system: 2 } } } m31102| Tue Oct 16 12:44:45 [rsHealthPoll] replSet member gcov1:31100 is now in state DOWN m31102| Tue Oct 16 12:44:45 [rsMgr] replSet can't see a majority, will not try to elect self m31101| Tue Oct 16 12:44:45 [interruptThread] shutdown: closing all files... m31101| Tue Oct 16 12:44:45 [interruptThread] closeAllFiles() finished m31101| Tue Oct 16 12:44:45 [interruptThread] journalCleanup... m31101| Tue Oct 16 12:44:45 [interruptThread] removeJournalFiles m31101| Tue Oct 16 12:44:45 [interruptThread] shutdown: removing fs lock... m31101| Tue Oct 16 12:44:45 dbexit: really exiting now m30999| Tue Oct 16 12:44:46 [Balancer] DBClientCursor::init call() failed m30999| Tue Oct 16 12:44:46 [Balancer] scoped connection to gcov1:29000 not being returned to the pool m30999| Tue Oct 16 12:44:46 [Balancer] caught exception while doing balance: DBClientBase::findN: transport error: gcov1:31100 ns: admin.$cmd query: { features: 1, $auth: { local: { __system: 2 } } } m29000| Tue Oct 16 12:44:46 [conn5] end connection 10.29.160.141:44873 (6 connections now open) Tue Oct 16 12:44:46 shell: stopped mongo program on port 31101 ReplSetTest n: 2 ports: [ 31100, 31101, 31102 ] 31102 number ReplSetTest stop *** Shutting down mongod in port 31102 *** m31102| Tue Oct 16 12:44:46 got signal 15 (Terminated), will terminate after current cmd ends m31102| Tue Oct 16 12:44:46 [interruptThread] now exiting m31102| Tue Oct 16 12:44:46 dbexit: m31102| Tue Oct 16 12:44:46 [interruptThread] shutdown: going to close listening sockets... m31102| Tue Oct 16 12:44:46 [interruptThread] closing listening socket: 19 m31102| Tue Oct 16 12:44:46 [interruptThread] closing listening socket: 20 m31102| Tue Oct 16 12:44:46 [interruptThread] closing listening socket: 21 m31102| Tue Oct 16 12:44:46 [interruptThread] removing socket file: /tmp/mongodb-31102.sock m31102| Tue Oct 16 12:44:46 [interruptThread] shutdown: going to flush diaglog... m31102| Tue Oct 16 12:44:46 [interruptThread] shutdown: going to close sockets... m31102| Tue Oct 16 12:44:46 [interruptThread] shutdown: waiting for fs preallocator... m31102| Tue Oct 16 12:44:46 [interruptThread] shutdown: lock for final commit... m31102| Tue Oct 16 12:44:46 [interruptThread] shutdown: final commit... m30999| Tue Oct 16 12:44:46 [WriteBackListener-gcov1:31102] DBClientCursor::init call() failed m30999| Tue Oct 16 12:44:46 [WriteBackListener-gcov1:31102] WriteBackListener exception : DBClientBase::findN: transport error: gcov1:31102 ns: admin.$cmd query: { writebacklisten: ObjectId('507d56b204bb5fa99d70aeee'), $auth: { local: { __system: 2 } } } m31102| Tue Oct 16 12:44:46 [conn1] end connection 10.29.160.141:50616 (6 connections now open) m30999| Tue Oct 16 12:44:46 [WriteBackListener-gcov1:31101] DBClientCursor::init call() failed m30999| Tue Oct 16 12:44:46 [WriteBackListener-gcov1:31101] WriteBackListener exception : DBClientBase::findN: transport error: gcov1:31101 ns: admin.$cmd query: { writebacklisten: ObjectId('507d56b204bb5fa99d70aeee'), $auth: { local: { __system: 2 } } } m31102| Tue Oct 16 12:44:46 [interruptThread] shutdown: closing all files... m31102| Tue Oct 16 12:44:46 [interruptThread] closeAllFiles() finished m31102| Tue Oct 16 12:44:46 [interruptThread] journalCleanup... m31102| Tue Oct 16 12:44:46 [interruptThread] removeJournalFiles m31102| Tue Oct 16 12:44:46 [interruptThread] shutdown: removing fs lock... m31102| Tue Oct 16 12:44:46 dbexit: really exiting now Tue Oct 16 12:44:47 shell: stopped mongo program on port 31102 ReplSetTest n: 0 ports: [ 31100, 31101, 31102 ] 31100 number ReplSetTest stop *** Shutting down mongod in port 31100 *** Tue Oct 16 12:44:47 No db started on port: 31100 Tue Oct 16 12:44:47 shell: stopped mongo program on port 31100 ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 31100, 31101, 31102 ] 31100 number { "useHostName" : true, "oplogSize" : 10, "keyFile" : "jstests/libs/key1", "port" : 31100, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "test-rs0", "dbpath" : "$set-$node", "restart" : true, "pathOpts" : { "node" : 0, "set" : "test-rs0" } } ReplSetTest (Re)Starting.... m30999| Tue Oct 16 12:44:47 [WriteBackListener-gcov1:31102] DBClientCursor::init call() failed m30999| Tue Oct 16 12:44:47 [WriteBackListener-gcov1:31102] WriteBackListener exception : DBClientBase::findN: transport error: gcov1:31102 ns: admin.$cmd query: { writebacklisten: ObjectId('507d56b204bb5fa99d70aeee'), $auth: { local: { __system: 2 } } } m30999| Tue Oct 16 12:44:47 [WriteBackListener-gcov1:31100] WriteBackListener exception : socket exception Tue Oct 16 12:44:47 shell: started program /data/buildslaves/Linux_64bit_GCov_Nightly/mongo/mongod --oplogSize 10 --keyFile jstests/libs/key1 --port 31100 --noprealloc --smallfiles --rest --replSet test-rs0 --dbpath /data/db/test-rs0-0 m31100| note: noprealloc may hurt performance in many applications m31100| Tue Oct 16 12:44:47 [initandlisten] MongoDB starting : pid=15108 port=31100 dbpath=/data/db/test-rs0-0 64-bit host=gcov1 m31100| Tue Oct 16 12:44:47 [initandlisten] m31100| Tue Oct 16 12:44:47 [initandlisten] ** NOTE: This is a development version (2.1.3-pre-) of MongoDB. m31100| Tue Oct 16 12:44:47 [initandlisten] ** Not recommended for production. m31100| Tue Oct 16 12:44:47 [initandlisten] m31100| Tue Oct 16 12:44:47 [initandlisten] db version v2.1.3-pre-, pdfile version 4.5 m31100| Tue Oct 16 12:44:47 [initandlisten] git version: 0f8117c897127c9c83fe00ff07b58d87a3ae2c89 m31100| Tue Oct 16 12:44:47 [initandlisten] build info: Linux gcov1 3.2.20-1.29.6.amzn1.x86_64 #1 SMP Tue Jun 12 01:19:28 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31100| Tue Oct 16 12:44:47 [initandlisten] options: { dbpath: "/data/db/test-rs0-0", keyFile: "jstests/libs/key1", noprealloc: true, oplogSize: 10, port: 31100, replSet: "test-rs0", rest: true, smallfiles: true } m31100| Tue Oct 16 12:44:47 [initandlisten] journal dir=/data/db/test-rs0-0/journal m31100| Tue Oct 16 12:44:47 [initandlisten] recover : no journal files present, no recovery needed m31100| Tue Oct 16 12:44:48 [initandlisten] preallocateIsFaster=true 2.58 m31100| Tue Oct 16 12:44:48 [initandlisten] preallocateIsFaster=true 2.12 m30999| Tue Oct 16 12:44:48 [WriteBackListener-gcov1:31101] WriteBackListener exception : socket exception m30999| Tue Oct 16 12:44:49 [WriteBackListener-gcov1:31102] WriteBackListener exception : socket exception m31100| Tue Oct 16 12:44:49 [initandlisten] preallocateIsFaster=true 4.14 m31100| Tue Oct 16 12:44:49 [initandlisten] preallocating a journal file /data/db/test-rs0-0/journal/prealloc.0 m30999| Tue Oct 16 12:44:50 [WriteBackListener-gcov1:31100] WriteBackListener exception : socket exception m30999| Tue Oct 16 12:44:51 [WriteBackListener-gcov1:31101] WriteBackListener exception : socket exception m29000| Tue Oct 16 12:44:52 [initandlisten] connection accepted from 10.29.160.141:44937 #8 (7 connections now open) m29000| Tue Oct 16 12:44:52 [conn8] authenticate db: local { authenticate: 1, nonce: "65b621e2b6390cba", user: "__system", key: "62432acd8f3e0e5933e1b6e8d6e7ce05" } m30999| Tue Oct 16 12:44:52 [Balancer] scoped connection to gcov1:29000 not being returned to the pool m30999| Tue Oct 16 12:44:52 [Balancer] caught exception while doing balance: socket exception m29000| Tue Oct 16 12:44:52 [conn6] end connection 10.29.160.141:44874 (6 connections now open) m31100| Tue Oct 16 12:44:52 [initandlisten] preallocating a journal file /data/db/test-rs0-0/journal/prealloc.1 m30999| Tue Oct 16 12:44:52 [WriteBackListener-gcov1:31102] WriteBackListener exception : socket exception Tue Oct 16 12:44:53 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Tue Oct 16 12:44:53 [ReplicaSetMonitorWatcher] trying reconnect to gcov1:31100 Tue Oct 16 12:44:53 [ReplicaSetMonitorWatcher] reconnect gcov1:31100 failed couldn't connect to server gcov1:31100 Tue Oct 16 12:44:53 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Tue Oct 16 12:44:53 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m30999| Tue Oct 16 12:44:54 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m30999| Tue Oct 16 12:44:54 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m30999| Tue Oct 16 12:44:54 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m30999| Tue Oct 16 12:44:54 [WriteBackListener-gcov1:31100] WriteBackListener exception : socket exception Tue Oct 16 12:44:54 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception gcov1:31100 socket exception Tue Oct 16 12:44:54 [ReplicaSetMonitorWatcher] trying reconnect to gcov1:31101 Tue Oct 16 12:44:54 [ReplicaSetMonitorWatcher] reconnect gcov1:31101 failed couldn't connect to server gcov1:31101 Tue Oct 16 12:44:54 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception gcov1:31101 socket exception Tue Oct 16 12:44:54 [ReplicaSetMonitorWatcher] trying reconnect to gcov1:31102 Tue Oct 16 12:44:54 [ReplicaSetMonitorWatcher] reconnect gcov1:31102 failed couldn't connect to server gcov1:31102 Tue Oct 16 12:44:54 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception gcov1:31102 socket exception m30999| Tue Oct 16 12:44:55 [ReplicaSetMonitorWatcher] trying reconnect to gcov1:31100 m30999| Tue Oct 16 12:44:55 [ReplicaSetMonitorWatcher] reconnect gcov1:31100 failed couldn't connect to server gcov1:31100 m30999| Tue Oct 16 12:44:55 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception gcov1:31100 socket exception m30999| Tue Oct 16 12:44:55 [ReplicaSetMonitorWatcher] trying reconnect to gcov1:31101 m30999| Tue Oct 16 12:44:55 [ReplicaSetMonitorWatcher] reconnect gcov1:31101 failed couldn't connect to server gcov1:31101 m30999| Tue Oct 16 12:44:55 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception gcov1:31101 socket exception m30999| Tue Oct 16 12:44:55 [ReplicaSetMonitorWatcher] trying reconnect to gcov1:31102 m30999| Tue Oct 16 12:44:55 [ReplicaSetMonitorWatcher] reconnect gcov1:31102 failed couldn't connect to server gcov1:31102 m30999| Tue Oct 16 12:44:55 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception gcov1:31102 socket exception m31100| Tue Oct 16 12:44:55 [initandlisten] preallocating a journal file /data/db/test-rs0-0/journal/prealloc.2 m30999| Tue Oct 16 12:44:55 [WriteBackListener-gcov1:31101] WriteBackListener exception : socket exception Tue Oct 16 12:44:55 [ReplicaSetMonitorWatcher] warning: No primary detected for set test-rs0 Tue Oct 16 12:44:55 [ReplicaSetMonitorWatcher] All nodes for set test-rs0 are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks m30999| Tue Oct 16 12:44:56 [ReplicaSetMonitorWatcher] warning: No primary detected for set test-rs0 m30999| Tue Oct 16 12:44:56 [ReplicaSetMonitorWatcher] All nodes for set test-rs0 are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks m30999| Tue Oct 16 12:44:56 [WriteBackListener-gcov1:31102] WriteBackListener exception : socket exception m31100| Tue Oct 16 12:44:58 [initandlisten] waiting for connections on port 31100 m31100| Tue Oct 16 12:44:58 [websvr] admin web console waiting for connections on port 32100 m31100| Tue Oct 16 12:44:58 [initandlisten] connection accepted from 10.29.160.141:45407 #1 (1 connection now open) m31100| Tue Oct 16 12:44:58 [conn1] authenticate db: local { authenticate: 1, nonce: "9849116ae3908536", user: "__system", key: "463174fa80bf640f12a5529cc8717cab" } m31100| Tue Oct 16 12:44:58 [rsStart] replSet I am gcov1:31100 m31100| Tue Oct 16 12:44:58 [rsStart] replSet STARTUP2 m31100| Tue Oct 16 12:44:58 [rsHealthPoll] couldn't connect to gcov1:31101: couldn't connect to server gcov1:31101 m31100| Tue Oct 16 12:44:58 [rsHealthPoll] replSet info gcov1:31101 is down (or slow to respond): socket exception m31100| Tue Oct 16 12:44:58 [rsHealthPoll] replSet member gcov1:31101 is now in state DOWN m29000| Tue Oct 16 12:44:58 [initandlisten] connection accepted from 10.29.160.141:44986 #9 (7 connections now open) m29000| Tue Oct 16 12:44:58 [conn9] authenticate db: local { authenticate: 1, nonce: "71ed419e125fc7ad", user: "__system", key: "632b6d2e5d1ca6c2f9c9c79670bf45ca" } m30999| Tue Oct 16 12:44:58 [Balancer] trying reconnect to gcov1:31100 m31100| Tue Oct 16 12:44:58 [initandlisten] connection accepted from 10.29.160.141:45411 #2 (2 connections now open) m30999| Tue Oct 16 12:44:58 [Balancer] reconnect gcov1:31100 ok m31100| Tue Oct 16 12:44:58 [initandlisten] connection accepted from 10.29.160.141:45412 #3 (3 connections now open) m31100| Tue Oct 16 12:44:58 [conn3] authenticate db: local { authenticate: 1, nonce: "c4142cd29f8e1ea1", user: "__system", key: "1e7529e7c55a9f60033c2d75fe306854" } m30999| Tue Oct 16 12:44:58 [Balancer] trying reconnect to gcov1:31101 m30999| Tue Oct 16 12:44:58 [Balancer] reconnect gcov1:31101 failed couldn't connect to server gcov1:31101 m30999| Tue Oct 16 12:44:58 [Balancer] trying reconnect to gcov1:31102 m30999| Tue Oct 16 12:44:58 [Balancer] reconnect gcov1:31102 failed couldn't connect to server gcov1:31102 m31100| Tue Oct 16 12:44:58 [initandlisten] connection accepted from 127.0.0.1:35427 #4 (4 connections now open) [ connection to gcov1:31100, connection to gcov1:31101, connection to gcov1:31102 ] ReplSetTest n: 1 ports: [ 31100, 31101, 31102 ] 31101 number ReplSetTest stop *** Shutting down mongod in port 31101 *** Tue Oct 16 12:44:58 No db started on port: 31101 Tue Oct 16 12:44:58 shell: stopped mongo program on port 31101 ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 31100, 31101, 31102 ] 31101 number { "useHostName" : true, "oplogSize" : 10, "keyFile" : "jstests/libs/key1", "port" : 31101, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "test-rs0", "dbpath" : "$set-$node", "restart" : true, "pathOpts" : { "node" : 1, "set" : "test-rs0" } } ReplSetTest (Re)Starting.... Tue Oct 16 12:44:58 shell: started program /data/buildslaves/Linux_64bit_GCov_Nightly/mongo/mongod --oplogSize 10 --keyFile jstests/libs/key1 --port 31101 --noprealloc --smallfiles --rest --replSet test-rs0 --dbpath /data/db/test-rs0-1 m31101| note: noprealloc may hurt performance in many applications m31101| Tue Oct 16 12:44:58 [initandlisten] MongoDB starting : pid=15243 port=31101 dbpath=/data/db/test-rs0-1 64-bit host=gcov1 m31101| Tue Oct 16 12:44:58 [initandlisten] m31101| Tue Oct 16 12:44:58 [initandlisten] ** NOTE: This is a development version (2.1.3-pre-) of MongoDB. m31101| Tue Oct 16 12:44:58 [initandlisten] ** Not recommended for production. m31101| Tue Oct 16 12:44:58 [initandlisten] m31101| Tue Oct 16 12:44:58 [initandlisten] db version v2.1.3-pre-, pdfile version 4.5 m31101| Tue Oct 16 12:44:58 [initandlisten] git version: 0f8117c897127c9c83fe00ff07b58d87a3ae2c89 m31101| Tue Oct 16 12:44:58 [initandlisten] build info: Linux gcov1 3.2.20-1.29.6.amzn1.x86_64 #1 SMP Tue Jun 12 01:19:28 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31101| Tue Oct 16 12:44:58 [initandlisten] options: { dbpath: "/data/db/test-rs0-1", keyFile: "jstests/libs/key1", noprealloc: true, oplogSize: 10, port: 31101, replSet: "test-rs0", rest: true, smallfiles: true } m31101| Tue Oct 16 12:44:58 [initandlisten] journal dir=/data/db/test-rs0-1/journal m31101| Tue Oct 16 12:44:58 [initandlisten] recover : no journal files present, no recovery needed m31101| Tue Oct 16 12:44:58 [initandlisten] preallocateIsFaster=true 2.58 m31101| Tue Oct 16 12:44:59 [initandlisten] preallocateIsFaster=true 3.4 m31100| Tue Oct 16 12:44:59 [rsSync] replSet SECONDARY m30999| Tue Oct 16 12:44:59 [Balancer] ReplicaSetMonitor::_checkConnection: gcov1:31100 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "gcov1:31100", "gcov1:31102", "gcov1:31101" ], me: "gcov1:31100", maxBsonObjectSize: 16777216, localTime: new Date(1350391499311), ok: 1.0 } m30999| Tue Oct 16 12:44:59 [Balancer] ReplicaSetMonitor::_checkConnection: caught exception gcov1:31101 socket exception m30999| Tue Oct 16 12:44:59 [Balancer] ReplicaSetMonitor::_checkConnection: caught exception gcov1:31102 socket exception m31100| Tue Oct 16 12:45:00 [rsHealthPoll] couldn't connect to gcov1:31102: couldn't connect to server gcov1:31102 m31100| Tue Oct 16 12:45:00 [rsHealthPoll] replSet info gcov1:31102 is down (or slow to respond): socket exception m31100| Tue Oct 16 12:45:00 [rsHealthPoll] replSet member gcov1:31102 is now in state DOWN m31100| Tue Oct 16 12:45:00 [rsMgr] replSet can't see a majority, will not try to elect self m31100| Tue Oct 16 12:45:00 [rsHealthPoll] couldn't connect to gcov1:31101: couldn't connect to server gcov1:31101 m30999| Tue Oct 16 12:45:00 [Balancer] warning: No primary detected for set test-rs0 m30999| Tue Oct 16 12:45:00 [Balancer] scoped connection to gcov1:29000 not being returned to the pool m30999| Tue Oct 16 12:45:00 [Balancer] caught exception while doing balance: socket exception m29000| Tue Oct 16 12:45:00 [conn8] end connection 10.29.160.141:44937 (6 connections now open) m31101| Tue Oct 16 12:45:00 [initandlisten] preallocateIsFaster=true 2.08 m31101| Tue Oct 16 12:45:00 [initandlisten] preallocating a journal file /data/db/test-rs0-1/journal/prealloc.0 m30999| Tue Oct 16 12:45:00 [WriteBackListener-gcov1:31101] WriteBackListener exception : socket exception m30999| Tue Oct 16 12:45:01 [WriteBackListener-gcov1:31102] WriteBackListener exception : socket exception m31100| Tue Oct 16 12:45:02 [rsHealthPoll] couldn't connect to gcov1:31102: couldn't connect to server gcov1:31102 m31100| Tue Oct 16 12:45:02 [rsHealthPoll] couldn't connect to gcov1:31101: couldn't connect to server gcov1:31101 m31101| Tue Oct 16 12:45:03 [initandlisten] preallocating a journal file /data/db/test-rs0-1/journal/prealloc.1 m31100| Tue Oct 16 12:45:04 [rsHealthPoll] couldn't connect to gcov1:31102: couldn't connect to server gcov1:31102 m31100| Tue Oct 16 12:45:04 [rsHealthPoll] couldn't connect to gcov1:31101: couldn't connect to server gcov1:31101 m31101| Tue Oct 16 12:45:05 [initandlisten] preallocating a journal file /data/db/test-rs0-1/journal/prealloc.2 Tue Oct 16 12:45:05 [ReplicaSetMonitorWatcher] trying reconnect to gcov1:31100 m31100| Tue Oct 16 12:45:05 [initandlisten] connection accepted from 10.29.160.141:45470 #5 (5 connections now open) Tue Oct 16 12:45:05 [ReplicaSetMonitorWatcher] reconnect gcov1:31100 ok Tue Oct 16 12:45:05 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Tue Oct 16 12:45:05 [ReplicaSetMonitorWatcher] trying reconnect to gcov1:31101 Tue Oct 16 12:45:05 [ReplicaSetMonitorWatcher] reconnect gcov1:31101 failed couldn't connect to server gcov1:31101 Tue Oct 16 12:45:05 [ReplicaSetMonitorWatcher] trying reconnect to gcov1:31102 Tue Oct 16 12:45:05 [ReplicaSetMonitorWatcher] reconnect gcov1:31102 failed couldn't connect to server gcov1:31102 m31100| Tue Oct 16 12:45:06 [rsHealthPoll] couldn't connect to gcov1:31102: couldn't connect to server gcov1:31102 m31100| Tue Oct 16 12:45:06 [rsHealthPoll] couldn't connect to gcov1:31101: couldn't connect to server gcov1:31101 m29000| Tue Oct 16 12:45:06 [initandlisten] connection accepted from 10.29.160.141:45057 #10 (7 connections now open) m29000| Tue Oct 16 12:45:06 [conn10] authenticate db: local { authenticate: 1, nonce: "cb5039ecf95199f3", user: "__system", key: "ac65ed780490b75770270ef26af95b66" } m31100| Tue Oct 16 12:45:06 [initandlisten] connection accepted from 10.29.160.141:45482 #6 (6 connections now open) m31100| Tue Oct 16 12:45:06 [conn6] authenticate db: local { authenticate: 1, nonce: "7bea0b3c7adbb88b", user: "__system", key: "ddf193076a77c56e9f2b58709abf58c4" } m30999| Tue Oct 16 12:45:06 [Balancer] trying reconnect to gcov1:31101 m30999| Tue Oct 16 12:45:06 [Balancer] reconnect gcov1:31101 failed couldn't connect to server gcov1:31101 m30999| Tue Oct 16 12:45:06 [Balancer] trying reconnect to gcov1:31102 m30999| Tue Oct 16 12:45:06 [Balancer] reconnect gcov1:31102 failed couldn't connect to server gcov1:31102 m30999| Tue Oct 16 12:45:06 [WriteBackListener-gcov1:31101] WriteBackListener exception : socket exception Tue Oct 16 12:45:06 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: gcov1:31100 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "gcov1:31100", "gcov1:31102", "gcov1:31101" ], me: "gcov1:31100", maxBsonObjectSize: 16777216, localTime: new Date(1350391506962), ok: 1.0 } m31100| Tue Oct 16 12:45:06 [initandlisten] connection accepted from 10.29.160.141:45489 #7 (7 connections now open) Tue Oct 16 12:45:06 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception gcov1:31101 socket exception Tue Oct 16 12:45:06 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception gcov1:31102 socket exception m30999| Tue Oct 16 12:45:07 [Balancer] ReplicaSetMonitor::_checkConnection: gcov1:31100 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "gcov1:31100", "gcov1:31102", "gcov1:31101" ], me: "gcov1:31100", maxBsonObjectSize: 16777216, localTime: new Date(1350391507315), ok: 1.0 } m30999| Tue Oct 16 12:45:07 [Balancer] ReplicaSetMonitor::_checkConnection: caught exception gcov1:31101 socket exception m30999| Tue Oct 16 12:45:07 [Balancer] ReplicaSetMonitor::_checkConnection: caught exception gcov1:31102 socket exception m30999| Tue Oct 16 12:45:07 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: gcov1:31100 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "gcov1:31100", "gcov1:31102", "gcov1:31101" ], me: "gcov1:31100", maxBsonObjectSize: 16777216, localTime: new Date(1350391507407), ok: 1.0 } m30999| Tue Oct 16 12:45:07 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception gcov1:31101 socket exception m30999| Tue Oct 16 12:45:07 [ReplicaSetMonitorWatcher] ReplicaSetMonitor::_checkConnection: caught exception gcov1:31102 socket exception m30999| Tue Oct 16 12:45:07 [WriteBackListener-gcov1:31102] WriteBackListener exception : socket exception Tue Oct 16 12:45:07 [ReplicaSetMonitorWatcher] warning: No primary detected for set test-rs0 Tue Oct 16 12:45:07 [ReplicaSetMonitorWatcher] All nodes for set test-rs0 are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks m31100| Tue Oct 16 12:45:08 [rsHealthPoll] couldn't connect to gcov1:31102: couldn't connect to server gcov1:31102 m31100| Tue Oct 16 12:45:08 [rsHealthPoll] couldn't connect to gcov1:31101: couldn't connect to server gcov1:31101 m30999| Tue Oct 16 12:45:08 [Balancer] warning: No primary detected for set test-rs0 m30999| Tue Oct 16 12:45:08 [Balancer] scoped connection to gcov1:29000 not being returned to the pool m30999| Tue Oct 16 12:45:08 [Balancer] caught exception while doing balance: socket exception m29000| Tue Oct 16 12:45:08 [conn9] end connection 10.29.160.141:44986 (6 connections now open) m30999| Tue Oct 16 12:45:08 [ReplicaSetMonitorWatcher] warning: No primary detected for set test-rs0 m31101| Tue Oct 16 12:45:08 [initandlisten] waiting for connections on port 31101 m31101| Tue Oct 16 12:45:08 [websvr] admin web console waiting for connections on port 32101 m31101| Tue Oct 16 12:45:08 [initandlisten] connection accepted from 10.29.160.141:35493 #1 (1 connection now open) m31101| Tue Oct 16 12:45:08 [conn1] authenticate db: local { authenticate: 1, nonce: "a8a598cd5968373", user: "__system", key: "e8381899d5988b330788799b5de76962" } m31101| Tue Oct 16 12:45:08 [rsStart] replSet I am gcov1:31101 m31101| Tue Oct 16 12:45:08 [rsStart] replSet STARTUP2 m31101| Tue Oct 16 12:45:08 [rsHealthPoll] couldn't connect to gcov1:31102: couldn't connect to server gcov1:31102 m31100| Tue Oct 16 12:45:08 [initandlisten] connection accepted from 10.29.160.141:45508 #8 (8 connections now open) m31101| Tue Oct 16 12:45:08 [rsHealthPoll] replSet info gcov1:31102 is down (or slow to respond): socket exception m31101| Tue Oct 16 12:45:08 [rsHealthPoll] replSet member gcov1:31102 is now in state DOWN m31100| Tue Oct 16 12:45:08 [conn8] authenticate db: local { authenticate: 1, nonce: "b80324266cb652ee", user: "__system", key: "741a5b36599bfb31825f9688b8047a80" } m31101| Tue Oct 16 12:45:08 [rsHealthPoll] replSet member gcov1:31100 is up m31101| Tue Oct 16 12:45:08 [rsHealthPoll] replSet member gcov1:31100 is now in state SECONDARY m31101| Tue Oct 16 12:45:08 [initandlisten] connection accepted from 127.0.0.1:64024 #2 (2 connections now open) [ connection to gcov1:31100, connection to gcov1:31101, connection to gcov1:31102 ] ReplSetTest n: 2 ports: [ 31100, 31101, 31102 ] 31102 number ReplSetTest stop *** Shutting down mongod in port 31102 *** Tue Oct 16 12:45:08 No db started on port: 31102 Tue Oct 16 12:45:08 shell: stopped mongo program on port 31102 ReplSetTest n is : 2 ReplSetTest n: 2 ports: [ 31100, 31101, 31102 ] 31102 number { "useHostName" : true, "oplogSize" : 10, "keyFile" : "jstests/libs/key1", "port" : 31102, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "test-rs0", "dbpath" : "$set-$node", "restart" : true, "pathOpts" : { "node" : 2, "set" : "test-rs0" } } ReplSetTest (Re)Starting.... Tue Oct 16 12:45:08 shell: started program /data/buildslaves/Linux_64bit_GCov_Nightly/mongo/mongod --oplogSize 10 --keyFile jstests/libs/key1 --port 31102 --noprealloc --smallfiles --rest --replSet test-rs0 --dbpath /data/db/test-rs0-2 m31102| note: noprealloc may hurt performance in many applications m31102| Tue Oct 16 12:45:08 [initandlisten] MongoDB starting : pid=15390 port=31102 dbpath=/data/db/test-rs0-2 64-bit host=gcov1 m31102| Tue Oct 16 12:45:08 [initandlisten] m31102| Tue Oct 16 12:45:08 [initandlisten] ** NOTE: This is a development version (2.1.3-pre-) of MongoDB. m31102| Tue Oct 16 12:45:08 [initandlisten] ** Not recommended for production. m31102| Tue Oct 16 12:45:08 [initandlisten] m31102| Tue Oct 16 12:45:08 [initandlisten] db version v2.1.3-pre-, pdfile version 4.5 m31102| Tue Oct 16 12:45:08 [initandlisten] git version: 0f8117c897127c9c83fe00ff07b58d87a3ae2c89 m31102| Tue Oct 16 12:45:08 [initandlisten] build info: Linux gcov1 3.2.20-1.29.6.amzn1.x86_64 #1 SMP Tue Jun 12 01:19:28 UTC 2012 x86_64 BOOST_LIB_VERSION=1_49 m31102| Tue Oct 16 12:45:08 [initandlisten] options: { dbpath: "/data/db/test-rs0-2", keyFile: "jstests/libs/key1", noprealloc: true, oplogSize: 10, port: 31102, replSet: "test-rs0", rest: true, smallfiles: true } m31102| Tue Oct 16 12:45:08 [initandlisten] journal dir=/data/db/test-rs0-2/journal m31102| Tue Oct 16 12:45:08 [initandlisten] recover : no journal files present, no recovery needed m31102| Tue Oct 16 12:45:09 [initandlisten] preallocateIsFaster=true 2.02 m31102| Tue Oct 16 12:45:09 [initandlisten] preallocateIsFaster=true 3.06 m31101| Tue Oct 16 12:45:09 [rsSync] replSet SECONDARY m31101| Tue Oct 16 12:45:10 [initandlisten] connection accepted from 10.29.160.141:35507 #3 (3 connections now open) m31100| Tue Oct 16 12:45:10 [rsHealthPoll] couldn't connect to gcov1:31102: couldn't connect to server gcov1:31102 m31101| Tue Oct 16 12:45:10 [conn3] end connection 10.29.160.141:35507 (2 connections now open) m31101| Tue Oct 16 12:45:10 [initandlisten] connection accepted from 10.29.160.141:35509 #4 (3 connections now open) m31101| Tue Oct 16 12:45:10 [conn4] authenticate db: local { authenticate: 1, nonce: "456725a7fdebb61e", user: "__system", key: "55c1280538eb809e627867671514119e" } m31101| Tue Oct 16 12:45:10 [conn4] command denied: { getnonce: 1 } m31100| Tue Oct 16 12:45:10 [rsHealthPoll] could not authenticate against gcov1:31101, getnonce fails - connection problem? m31100| Tue Oct 16 12:45:10 [rsHealthPoll] replSet member gcov1:31101 is up m31100| Tue Oct 16 12:45:10 [rsHealthPoll] replSet member gcov1:31101 is now in state SECONDARY m31101| Tue Oct 16 12:45:10 [conn4] command denied: { replSetFresh: 1, set: "test-rs0", opTime: new Date(5799887260576907284), who: "gcov1:31100", cfgver: 1, id: 0 } m31100| Tue Oct 16 12:45:10 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes m31101| Tue Oct 16 12:45:10 [rsHealthPoll] couldn't connect to gcov1:31102: couldn't connect to server gcov1:31102 m31102| Tue Oct 16 12:45:10 [initandlisten] preallocateIsFaster=true 3.38 m31102| Tue Oct 16 12:45:10 [initandlisten] preallocating a journal file /data/db/test-rs0-2/journal/prealloc.0 m31100| Tue Oct 16 12:45:12 [rsHealthPoll] couldn't connect to gcov1:31102: couldn't connect to server gcov1:31102 m31101| Tue Oct 16 12:45:12 [conn4] end connection 10.29.160.141:35509 (2 connections now open) m31101| Tue Oct 16 12:45:12 [initandlisten] connection accepted from 10.29.160.141:35524 #5 (3 connections now open) m31101| Tue Oct 16 12:45:12 [conn5] authenticate db: local { authenticate: 1, nonce: "b4ba3b5b7fd1d561", user: "__system", key: "9c0eae3b72ad6e220646cbf50d4fb0be" } m31101| Tue Oct 16 12:45:12 [rsHealthPoll] couldn't connect to gcov1:31102: couldn't connect to server gcov1:31102 m31100| Tue Oct 16 12:45:13 [rsBackgroundSync] waiting for 4 pings from other members before syncing m31102| Tue Oct 16 12:45:13 [initandlisten] preallocating a journal file /data/db/test-rs0-2/journal/prealloc.1 m31101| Tue Oct 16 12:45:13 [initandlisten] connection accepted from 10.29.160.141:35534 #6 (4 connections now open) m31101| Tue Oct 16 12:45:13 [conn6] authenticate db: local { authenticate: 1, nonce: "ba0b54447c792a2d", user: "__system", key: "53f3bb912fa1839993c657f694700023" } m31100| Tue Oct 16 12:45:14 [rsHealthPoll] couldn't connect to gcov1:31102: couldn't connect to server gcov1:31102 m29000| Tue Oct 16 12:45:14 [initandlisten] connection accepted from 10.29.160.141:45131 #11 (7 connections now open) m29000| Tue Oct 16 12:45:14 [conn11] authenticate db: local { authenticate: 1, nonce: "beb9b03132eb6289", user: "__system", key: "63f5bcd6d612245f2920aa94a72838a2" } m30999| Tue Oct 16 12:45:14 [Balancer] trying reconnect to gcov1:31101 m31101| Tue Oct 16 12:45:14 [initandlisten] connection accepted from 10.29.160.141:35543 #7 (5 connections now open) m30999| Tue Oct 16 12:45:14 [Balancer] reconnect gcov1:31101 ok m31101| Tue Oct 16 12:45:14 [initandlisten] connection accepted from 10.29.160.141:35544 #8 (6 connections now open) m31101| Tue Oct 16 12:45:14 [conn8] authenticate db: local { authenticate: 1, nonce: "dfdde8a201ddb07d", user: "__system", key: "fac829bf1829ecf0b91f1bfe89eef667" } m30999| Tue Oct 16 12:45:14 [Balancer] trying reconnect to gcov1:31102 m30999| Tue Oct 16 12:45:14 [Balancer] reconnect gcov1:31102 failed couldn't connect to server gcov1:31102 m30999| Tue Oct 16 12:45:14 [WriteBackListener-gcov1:31102] WriteBackListener exception : socket exception m31101| Tue Oct 16 12:45:14 [rsMgr] not electing self, gcov1:31100 would veto m31101| Tue Oct 16 12:45:14 [rsHealthPoll] couldn't connect to gcov1:31102: couldn't connect to server gcov1:31102 m30999| Tue Oct 16 12:45:15 [Balancer] ReplicaSetMonitor::_checkConnection: gcov1:31100 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "gcov1:31100", "gcov1:31102", "gcov1:31101" ], me: "gcov1:31100", maxBsonObjectSize: 16777216, localTime: new Date(1350391515320), ok: 1.0 } m30999| Tue Oct 16 12:45:15 [Balancer] ReplicaSetMonitor::_checkConnection: gcov1:31101 { setName: "test-rs0", ismaster: false, secondary: true, hosts: [ "gcov1:31101", "gcov1:31102", "gcov1:31100" ], me: "gcov1:31101", maxBsonObjectSize: 16777216, localTime: new Date(1350391515320), ok: 1.0 } m30999| Tue Oct 16 12:45:15 [Balancer] ReplicaSetMonitor::_checkConnection: caught exception gcov1:31102 socket exception m31100| Tue Oct 16 12:45:16 [rsHealthPoll] couldn't connect to gcov1:31102: couldn't connect to server gcov1:31102 m31100| Tue Oct 16 12:45:16 [rsMgr] replSet info electSelf 0 m31101| Tue Oct 16 12:45:16 [conn5] replSet info voting yea for gcov1:31100 (0) m31100| Tue Oct 16 12:45:16 [rsMgr] replSet PRIMARY m30999| Tue Oct 16 12:45:16 [Balancer] warning: No primary detected for set test-rs0 m30999| Tue Oct 16 12:45:16 [Balancer] scoped connection to gcov1:29000 not being returned to the pool m30999| Tue Oct 16 12:45:16 [Balancer] caught exception while doing balance: socket exception m29000| Tue Oct 16 12:45:16 [conn10] end connection 10.29.160.141:45057 (6 connections now open) m31102| Tue Oct 16 12:45:16 [initandlisten] preallocating a journal file /data/db/test-rs0-2/journal/prealloc.2 m31101| Tue Oct 16 12:45:16 [rsHealthPoll] replSet member gcov1:31100 is now in state PRIMARY m31101| Tue Oct 16 12:45:16 [rsHealthPoll] couldn't connect to gcov1:31102: couldn't connect to server gcov1:31102 Tue Oct 16 12:45:17 [ReplicaSetMonitorWatcher] Primary for replica set test-rs0 changed to gcov1:31100 Tue Oct 16 12:45:17 [ReplicaSetMonitorWatcher] trying reconnect to gcov1:31101 Tue Oct 16 12:45:17 [ReplicaSetMonitorWatcher] reconnect gcov1:31101 ok Tue Oct 16 12:45:17 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed Tue Oct 16 12:45:17 [ReplicaSetMonitorWatcher] trying reconnect to gcov1:31102 m31101| Tue Oct 16 12:45:17 [initandlisten] connection accepted from 10.29.160.141:35571 #9 (7 connections now open) Tue Oct 16 12:45:17 [ReplicaSetMonitorWatcher] reconnect gcov1:31102 failed couldn't connect to server gcov1:31102 m31100| Tue Oct 16 12:45:18 [rsHealthPoll] couldn't connect to gcov1:31102: couldn't connect to server gcov1:31102 m30999| Tue Oct 16 12:45:18 [ReplicaSetMonitorWatcher] Primary for replica set test-rs0 changed to gcov1:31100 m30999| Tue Oct 16 12:45:18 [ReplicaSetMonitorWatcher] trying reconnect to gcov1:31102 m30999| Tue Oct 16 12:45:18 [ReplicaSetMonitorWatcher] reconnect gcov1:31102 failed couldn't connect to server gcov1:31102 m31101| Tue Oct 16 12:45:18 [rsHealthPoll] couldn't connect to gcov1:31102: couldn't connect to server gcov1:31102 m31102| Tue Oct 16 12:45:19 [initandlisten] waiting for connections on port 31102 m31102| Tue Oct 16 12:45:19 [websvr] admin web console waiting for connections on port 32102 m31102| Tue Oct 16 12:45:19 [initandlisten] connection accepted from 10.29.160.141:50965 #1 (1 connection now open) m31102| Tue Oct 16 12:45:19 [conn1] authenticate db: local { authenticate: 1, nonce: "fe628e06069ffd99", user: "__system", key: "d17f4913858246aa81a7b89b0f4f89ad" } m31102| Tue Oct 16 12:45:19 [rsStart] replSet I am gcov1:31102 m31102| Tue Oct 16 12:45:19 [rsStart] replSet STARTUP2 m31100| Tue Oct 16 12:45:19 [initandlisten] connection accepted from 10.29.160.141:45600 #9 (9 connections now open) m31100| Tue Oct 16 12:45:19 [conn9] authenticate db: local { authenticate: 1, nonce: "8e71ebc9440e1e8d", user: "__system", key: "7f6512de31dee1a1c2d69abeaa4820a2" } m31102| Tue Oct 16 12:45:19 [rsHealthPoll] replSet member gcov1:31100 is up m31102| Tue Oct 16 12:45:19 [rsHealthPoll] replSet member gcov1:31100 is now in state PRIMARY m31102| Tue Oct 16 12:45:19 [initandlisten] connection accepted from 127.0.0.1:54674 #2 (2 connections now open) [ connection to gcov1:31100, connection to gcov1:31101, connection to gcov1:31102 ] m31101| Tue Oct 16 12:45:19 [rsBackgroundSync] replSet syncing to: gcov1:31100 m31100| Tue Oct 16 12:45:19 [initandlisten] connection accepted from 10.29.160.141:45602 #10 (10 connections now open) m31100| Tue Oct 16 12:45:19 [conn10] authenticate db: local { authenticate: 1, nonce: "2346dfdb9d9205fb", user: "__system", key: "30e0610eaf36ef500fab5834ec0f4f05" } m31102| Tue Oct 16 12:45:20 [rsSync] replSet SECONDARY m31102| Tue Oct 16 12:45:20 [initandlisten] connection accepted from 10.29.160.141:50969 #3 (3 connections now open) m31102| Tue Oct 16 12:45:20 [conn3] end connection 10.29.160.141:50969 (2 connections now open) m31102| Tue Oct 16 12:45:20 [initandlisten] connection accepted from 10.29.160.141:50970 #4 (4 connections now open) m31102| Tue Oct 16 12:45:20 [conn4] authenticate db: local { authenticate: 1, nonce: "914b95c2e8dc67ff", user: "__system", key: "af9778cb35f49db9ddd1d1bbf09e4320" } m31102| Tue Oct 16 12:45:20 [conn4] command denied: { getnonce: 1 } m31100| Tue Oct 16 12:45:20 [rsHealthPoll] could not authenticate against gcov1:31102, getnonce fails - connection problem? m31100| Tue Oct 16 12:45:20 [rsHealthPoll] replSet member gcov1:31102 is up m31100| Tue Oct 16 12:45:20 [rsHealthPoll] replSet member gcov1:31102 is now in state SECONDARY m31101| Tue Oct 16 12:45:20 [rsSyncNotifier] replset setting oplog notifier to gcov1:31100 m31100| Tue Oct 16 12:45:20 [initandlisten] connection accepted from 10.29.160.141:45605 #11 (11 connections now open) m31100| Tue Oct 16 12:45:20 [conn11] authenticate db: local { authenticate: 1, nonce: "39eb1af56da9ccce", user: "__system", key: "143ceb8d35dabbea53c2f2fefe8d531e" } m31102| Tue Oct 16 12:45:20 [initandlisten] connection accepted from 10.29.160.141:50972 #5 (4 connections now open) m31102| Tue Oct 16 12:45:20 [conn5] end connection 10.29.160.141:50972 (3 connections now open) m31102| Tue Oct 16 12:45:20 [initandlisten] connection accepted from 10.29.160.141:50973 #6 (4 connections now open) m31102| Tue Oct 16 12:45:20 [conn6] authenticate db: local { authenticate: 1, nonce: "117bb207d65095c2", user: "__system", key: "67a686eb56e95e915c48145e90334136" } m31102| Tue Oct 16 12:45:20 [conn6] command denied: { getnonce: 1 } m31101| Tue Oct 16 12:45:20 [rsHealthPoll] could not authenticate against gcov1:31102, getnonce fails - connection problem? m31101| Tue Oct 16 12:45:20 [rsHealthPoll] replSet member gcov1:31102 is up m31101| Tue Oct 16 12:45:20 [rsHealthPoll] replSet member gcov1:31102 is now in state SECONDARY m31101| Tue Oct 16 12:45:21 [initandlisten] connection accepted from 10.29.160.141:35595 #10 (8 connections now open) m31101| Tue Oct 16 12:45:21 [conn10] authenticate db: local { authenticate: 1, nonce: "17734f591d23550c", user: "__system", key: "8c374cf517e5fee41a44830bc56d77a5" } m31102| Tue Oct 16 12:45:21 [rsHealthPoll] replSet member gcov1:31101 is up m31102| Tue Oct 16 12:45:21 [rsHealthPoll] replSet member gcov1:31101 is now in state SECONDARY m30999| Tue Oct 16 12:45:21 [conn1] DBClientCursor::init lazy say() failed m30999| Tue Oct 16 12:45:21 [conn1] DBClientCursor::init message from say() was empty m30999| Tue Oct 16 12:45:21 [conn1] slave no longer has secondary status: gcov1:31101 m30999| Tue Oct 16 12:45:21 [conn1] warning: Weird shift of primary detected m31100| Tue Oct 16 12:45:21 [initandlisten] connection accepted from 10.29.160.141:45609 #12 (12 connections now open) m31100| Tue Oct 16 12:45:21 [conn12] authenticate db: local { authenticate: 1, nonce: "7b1662c18ceb867", user: "__system", key: "12a63e99c491bf7f827be4b3b9074c8b" } m31100| Tue Oct 16 12:45:21 [initandlisten] connection accepted from 10.29.160.141:45610 #13 (13 connections now open) assert failed Error("Printing Stack Trace")@:0 ()@src/mongo/shell/utils.js:37 ("assert failed")@src/mongo/shell/utils.js:58 (false)@src/mongo/shell/utils.js:66 @/data/buildslaves/Linux_64bit_GCov_Nightly/mongo/jstests/sharding/auth_slaveok_routing.js:78 Exception occured (expected): assert failed Awaiting gcov1:31101 to be { "ok" : true, "secondary" : true } for connection to gcov1:30999 (rs: undefined) { "test-rs0" : { "hosts" : [ { "addr" : "gcov1:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31101", "ok" : false, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31102", "ok" : false, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } m29000| Tue Oct 16 12:45:22 [initandlisten] connection accepted from 10.29.160.141:45189 #12 (7 connections now open) m29000| Tue Oct 16 12:45:22 [conn12] authenticate db: local { authenticate: 1, nonce: "af6dcf78b62a92ee", user: "__system", key: "fd586e458a618aa4630a551440e76f7a" } m31100| Tue Oct 16 12:45:22 [initandlisten] connection accepted from 10.29.160.141:45614 #14 (14 connections now open) m31100| Tue Oct 16 12:45:22 [conn14] authenticate db: local { authenticate: 1, nonce: "6137f754fe5948f9", user: "__system", key: "c429c3fd55d86e18c7c0f8c4faf973c1" } m30999| Tue Oct 16 12:45:22 [Balancer] distributed lock 'balancer/gcov1:30999:1350391474:1804289383' acquired, ts : 507d56e204bb5fa99d70aef1 m30999| Tue Oct 16 12:45:22 [Balancer] distributed lock 'balancer/gcov1:30999:1350391474:1804289383' unlocked. m31102| Tue Oct 16 12:45:22 [initandlisten] connection accepted from 10.29.160.141:50981 #7 (5 connections now open) m31102| Tue Oct 16 12:45:22 [conn7] authenticate db: local { authenticate: 1, nonce: "ab5c8a1c8c70c8c3", user: "__system", key: "e34c0c893aad6aa8e15157cc09f798fa" } m31100| Tue Oct 16 12:45:22 [conn8] end connection 10.29.160.141:45508 (13 connections now open) m31100| Tue Oct 16 12:45:22 [initandlisten] connection accepted from 10.29.160.141:45616 #15 (14 connections now open) m31100| Tue Oct 16 12:45:22 [conn15] authenticate db: local { authenticate: 1, nonce: "7feca234e77f934f", user: "__system", key: "e5e0781820cb38ff0719f7a3a724a726" } { "test-rs0" : { "hosts" : [ { "addr" : "gcov1:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31101", "ok" : false, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31102", "ok" : false, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } { "test-rs0" : { "hosts" : [ { "addr" : "gcov1:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31101", "ok" : false, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31102", "ok" : false, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } m31102| Tue Oct 16 12:45:26 [rsBackgroundSync] replSet syncing to: gcov1:31100 m31100| Tue Oct 16 12:45:26 [initandlisten] connection accepted from 10.29.160.141:45621 #16 (15 connections now open) m31100| Tue Oct 16 12:45:26 [conn16] authenticate db: local { authenticate: 1, nonce: "e24c87f68d7f25c1", user: "__system", key: "8ff880592c79b64de8a99f3e60038d7b" } m31102| Tue Oct 16 12:45:27 [rsSyncNotifier] replset setting oplog notifier to gcov1:31100 m31100| Tue Oct 16 12:45:27 [initandlisten] connection accepted from 10.29.160.141:45622 #17 (16 connections now open) m31100| Tue Oct 16 12:45:27 [conn17] authenticate db: local { authenticate: 1, nonce: "6edd8ccd4aca68d6", user: "__system", key: "0c13901ca8b6cf9016d8fb6c4ca94484" } { "test-rs0" : { "hosts" : [ { "addr" : "gcov1:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31101", "ok" : false, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31102", "ok" : false, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } m31101| Tue Oct 16 12:45:27 [initandlisten] connection accepted from 10.29.160.141:35612 #11 (9 connections now open) Tue Oct 16 12:45:27 [ReplicaSetMonitorWatcher] trying reconnect to gcov1:31102 m31102| Tue Oct 16 12:45:27 [initandlisten] connection accepted from 10.29.160.141:50992 #8 (6 connections now open) Tue Oct 16 12:45:27 [ReplicaSetMonitorWatcher] reconnect gcov1:31102 ok Tue Oct 16 12:45:27 [ReplicaSetMonitorWatcher] DBClientCursor::init call() failed m31102| Tue Oct 16 12:45:28 [conn4] end connection 10.29.160.141:50970 (5 connections now open) m31102| Tue Oct 16 12:45:28 [initandlisten] connection accepted from 10.29.160.141:50993 #9 (6 connections now open) m31102| Tue Oct 16 12:45:28 [conn9] authenticate db: local { authenticate: 1, nonce: "21ed944984c15738", user: "__system", key: "4314b60878e65bf0374fae6d82ccdcd6" } m30999| Tue Oct 16 12:45:28 [Balancer] distributed lock 'balancer/gcov1:30999:1350391474:1804289383' acquired, ts : 507d56e804bb5fa99d70aef2 m30999| Tue Oct 16 12:45:28 [Balancer] distributed lock 'balancer/gcov1:30999:1350391474:1804289383' unlocked. m30999| Tue Oct 16 12:45:28 [ReplicaSetMonitorWatcher] trying reconnect to gcov1:31102 m31102| Tue Oct 16 12:45:28 [initandlisten] connection accepted from 10.29.160.141:50994 #10 (7 connections now open) m30999| Tue Oct 16 12:45:28 [ReplicaSetMonitorWatcher] reconnect gcov1:31102 ok m31102| Tue Oct 16 12:45:28 [initandlisten] connection accepted from 10.29.160.141:50995 #11 (8 connections now open) m31102| Tue Oct 16 12:45:28 [conn11] authenticate db: local { authenticate: 1, nonce: "c723d313af64c93c", user: "__system", key: "ee99f42e3f0c0481797bcac81d1d32e1" } Awaiting gcov1:31102 to be { "ok" : true, "secondary" : true } for connection to gcov1:30999 (rs: undefined) { "test-rs0" : { "hosts" : [ { "addr" : "gcov1:31100", "ok" : true, "ismaster" : true, "hidden" : false, "secondary" : false, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31101", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 }, { "addr" : "gcov1:31102", "ok" : true, "ismaster" : false, "hidden" : false, "secondary" : true, "pingTimeMillis" : 0 } ], "master" : 0, "nextSlave" : 0 } } ---- Final query to SEC ---- m31100| Tue Oct 16 12:45:28 [initandlisten] connection accepted from 10.29.160.141:45630 #18 (17 connections now open) assert failed Error("Printing Stack Trace")@:0 ()@src/mongo/shell/utils.js:37 ("assert failed")@src/mongo/shell/utils.js:58 (false)@src/mongo/shell/utils.js:66 @/data/buildslaves/Linux_64bit_GCov_Nightly/mongo/jstests/sharding/auth_slaveok_routing.js:95 Tue Oct 16 12:45:28 uncaught exception: assert failed failed to load: /data/buildslaves/Linux_64bit_GCov_Nightly/mongo/jstests/sharding/auth_slaveok_routing.js m29000| Tue Oct 16 12:45:28 got signal 15 (Terminated), will terminate after current cmd ends m29000| Tue Oct 16 12:45:28 [interruptThread] now exiting m29000| Tue Oct 16 12:45:28 dbexit: m29000| Tue Oct 16 12:45:28 [interruptThread] shutdown: going to close listening sockets... m29000| Tue Oct 16 12:45:28 [interruptThread] closing listening socket: 29 m29000| Tue Oct 16 12:45:28 [interruptThread] closing listening socket: 30 m29000| Tue Oct 16 12:45:28 [interruptThread] closing listening socket: 31 m29000| Tue Oct 16 12:45:28 [interruptThread] removing socket file: /tmp/mongodb-29000.sock m29000| Tue Oct 16 12:45:28 [interruptThread] shutdown: going to flush diaglog... m29000| Tue Oct 16 12:45:28 [interruptThread] shutdown: going to close sockets... m29000| Tue Oct 16 12:45:28 [interruptThread] shutdown: waiting for fs preallocator... m29000| Tue Oct 16 12:45:28 [interruptThread] shutdown: lock for final commit... m29000| Tue Oct 16 12:45:28 [interruptThread] shutdown: final commit... m29000| Tue Oct 16 12:45:28 [interruptThread] shutdown: closing all files... m29000| Tue Oct 16 12:45:28 [interruptThread] closeAllFiles() finished m29000| Tue Oct 16 12:45:28 [interruptThread] journalCleanup... m29000| Tue Oct 16 12:45:28 [interruptThread] removeJournalFiles m29000| Tue Oct 16 12:45:28 [interruptThread] shutdown: removing fs lock... m29000| Tue Oct 16 12:45:28 dbexit: really exiting now m30999| Tue Oct 16 12:45:29 [mongosMain] dbexit: received signal 15 rc:0 received signal 15 m31100| Tue Oct 16 12:45:29 [conn2] end connection 10.29.160.141:45411 (16 connections now open) m31100| Tue Oct 16 12:45:29 [conn12] end connection 10.29.160.141:45609 (15 connections now open) m31100| Tue Oct 16 12:45:29 [conn6] end connection 10.29.160.141:45482 (14 connections now open) m31101| Tue Oct 16 12:45:29 [conn7] end connection 10.29.160.141:35543 (8 connections now open) m31101| Tue Oct 16 12:45:29 [conn8] end connection 10.29.160.141:35544 (7 connections now open) m31102| Tue Oct 16 12:45:29 [conn11] end connection 10.29.160.141:50995 (7 connections now open) m31100| Tue Oct 16 12:45:29 [conn14] end connection 10.29.160.141:45614 (13 connections now open) m31102| Tue Oct 16 12:45:29 [conn10] end connection 10.29.160.141:50994 (6 connections now open) m31100| Tue Oct 16 12:45:30 got signal 15 (Terminated), will terminate after current cmd ends m31100| Tue Oct 16 12:45:30 [interruptThread] now exiting m31100| Tue Oct 16 12:45:30 dbexit: m31100| Tue Oct 16 12:45:30 [interruptThread] shutdown: going to close listening sockets... m31100| Tue Oct 16 12:45:30 [interruptThread] closing listening socket: 36 m31100| Tue Oct 16 12:45:30 [interruptThread] closing listening socket: 37 m31100| Tue Oct 16 12:45:30 [interruptThread] closing listening socket: 38 m31100| Tue Oct 16 12:45:30 [interruptThread] removing socket file: /tmp/mongodb-31100.sock m31100| Tue Oct 16 12:45:30 [interruptThread] shutdown: going to flush diaglog... m31100| Tue Oct 16 12:45:30 [interruptThread] shutdown: going to close sockets... m31101| Tue Oct 16 12:45:30 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: gcov1:31100 m31101| Tue Oct 16 12:45:30 [conn5] end connection 10.29.160.141:35524 (6 connections now open) m31102| Tue Oct 16 12:45:30 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: gcov1:31100 m31102| Tue Oct 16 12:45:30 [rsBackgroundSync] replSet db exception in producer: 10278 dbclient error communicating with server: gcov1:31100 m31102| Tue Oct 16 12:45:30 [conn9] end connection 10.29.160.141:50993 (5 connections now open) m31100| Tue Oct 16 12:45:30 [interruptThread] shutdown: waiting for fs preallocator... m31100| Tue Oct 16 12:45:30 [interruptThread] shutdown: lock for final commit... m31100| Tue Oct 16 12:45:30 [interruptThread] shutdown: final commit... m31100| Tue Oct 16 12:45:30 [interruptThread] shutdown: closing all files... m31100| Tue Oct 16 12:45:30 [conn1] end connection 10.29.160.141:45407 (12 connections now open) m31100| Tue Oct 16 12:45:30 [interruptThread] closeAllFiles() finished m31100| Tue Oct 16 12:45:30 [interruptThread] journalCleanup... m31100| Tue Oct 16 12:45:30 [interruptThread] removeJournalFiles m31100| Tue Oct 16 12:45:30 [interruptThread] shutdown: removing fs lock... m31100| Tue Oct 16 12:45:30 dbexit: really exiting now m31101| Tue Oct 16 12:45:30 [rsHealthPoll] DBClientCursor::init call() failed m31101| Tue Oct 16 12:45:30 [rsHealthPoll] replSet info gcov1:31100 is down (or slow to respond): DBClientBase::findN: transport error: gcov1:31100 ns: admin.$cmd query: { replSetHeartbeat: "test-rs0", v: 1, pv: 1, checkEmpty: false, from: "gcov1:31101", $auth: { local: { __system: 2 } } } m31101| Tue Oct 16 12:45:30 [rsHealthPoll] replSet member gcov1:31100 is now in state DOWN m31102| Tue Oct 16 12:45:30 [conn6] command denied: { replSetFresh: 1, set: "test-rs0", opTime: new Date(5799887260576907284), who: "gcov1:31101", cfgver: 1, id: 1 } m31101| Tue Oct 16 12:45:30 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes m31102| Tue Oct 16 12:45:31 [rsHealthPoll] DBClientCursor::init call() failed m31102| Tue Oct 16 12:45:31 [rsHealthPoll] replSet info gcov1:31100 is down (or slow to respond): DBClientBase::findN: transport error: gcov1:31100 ns: admin.$cmd query: { replSetHeartbeat: "test-rs0", v: 1, pv: 1, checkEmpty: false, from: "gcov1:31102", $auth: { local: { __system: 2 } } } m31102| Tue Oct 16 12:45:31 [rsHealthPoll] replSet member gcov1:31100 is now in state DOWN m31102| Tue Oct 16 12:45:31 [rsMgr] replSet info electSelf 2 m31101| Tue Oct 16 12:45:31 [conn10] replSet voting no for gcov1:31102 already voted for another m31102| Tue Oct 16 12:45:31 [rsMgr] replSet couldn't elect self, only received 1 votes m31101| Tue Oct 16 12:45:31 got signal 15 (Terminated), will terminate after current cmd ends m31101| Tue Oct 16 12:45:31 [interruptThread] now exiting m31101| Tue Oct 16 12:45:31 dbexit: m31101| Tue Oct 16 12:45:31 [interruptThread] shutdown: going to close listening sockets... m31101| Tue Oct 16 12:45:31 [interruptThread] closing listening socket: 36 m31101| Tue Oct 16 12:45:31 [interruptThread] closing listening socket: 37 m31101| Tue Oct 16 12:45:31 [interruptThread] closing listening socket: 38 m31101| Tue Oct 16 12:45:31 [interruptThread] removing socket file: /tmp/mongodb-31101.sock m31101| Tue Oct 16 12:45:31 [interruptThread] shutdown: going to flush diaglog... m31101| Tue Oct 16 12:45:31 [interruptThread] shutdown: going to close sockets... m31101| Tue Oct 16 12:45:31 [interruptThread] shutdown: waiting for fs preallocator... m31101| Tue Oct 16 12:45:31 [interruptThread] shutdown: lock for final commit... m31101| Tue Oct 16 12:45:31 [interruptThread] shutdown: final commit... m31101| Tue Oct 16 12:45:31 [interruptThread] shutdown: closing all files... m31102| Tue Oct 16 12:45:31 [conn6] end connection 10.29.160.141:50973 (4 connections now open) m31101| Tue Oct 16 12:45:31 [conn1] end connection 10.29.160.141:35493 (5 connections now open) m31101| Tue Oct 16 12:45:31 [interruptThread] closeAllFiles() finished m31101| Tue Oct 16 12:45:31 [interruptThread] journalCleanup... m31101| Tue Oct 16 12:45:31 [interruptThread] removeJournalFiles m31101| Tue Oct 16 12:45:31 [interruptThread] shutdown: removing fs lock... m31101| Tue Oct 16 12:45:31 dbexit: really exiting now m31102| Tue Oct 16 12:45:32 got signal 15 (Terminated), will terminate after current cmd ends m31102| Tue Oct 16 12:45:32 [interruptThread] now exiting m31102| Tue Oct 16 12:45:32 dbexit: m31102| Tue Oct 16 12:45:32 [interruptThread] shutdown: going to close listening sockets... m31102| Tue Oct 16 12:45:32 [interruptThread] closing listening socket: 40 m31102| Tue Oct 16 12:45:32 [interruptThread] closing listening socket: 41 m31102| Tue Oct 16 12:45:32 [interruptThread] closing listening socket: 42 m31102| Tue Oct 16 12:45:32 [interruptThread] removing socket file: /tmp/mongodb-31102.sock m31102| Tue Oct 16 12:45:32 [interruptThre Tue Oct 16 12:45:33 got signal 15 (Terminated), will terminate after current cmd ends Tue Oct 16 12:45:33 [interruptThread] now exiting Tue Oct 16 12:45:33 dbexit: Tue Oct 16 12:45:33 [interruptThread] shutdown: going to close listening sockets... Tue Oct 16 12:45:33 [interruptThread] closing listening socket: 8 Tue Oct 16 12:45:33 [interruptThread] closing listening socket: 9 Tue Oct 16 12:45:33 [interruptThread] closing listening socket: 10 Tue Oct 16 12:45:33 [interruptThread] removing socket file: /tmp/mongodb-27999.sock Tue Oct 16 12:45:33 [interruptThread] shutdown: going to flush diaglog... Tue Oct 16 12:45:33 [interruptThread] shutdown: going to close sockets... Tue Oct 16 12:45:33 [interruptThread] shutdown: waiting for fs preallocator... Tue Oct 16 12:45:33 [interruptThread] shutdown: lock for final commit... Tue Oct 16 12:45:33 [interruptThread] shutdown: final commit... Tue Oct 16 12:45:33 [interruptThread] shutdown: closing all files... Tue Oct 16 12:45:33 [interruptThread] closeAllFiles() finished Tue Oct 16 12:45:33 [interruptThread] journalCleanup... Tue Oct 16 12:45:33 [interruptThread] removeJournalFiles Tue Oct 16 12:45:33 [interruptThread] shutdown: removing fs lock... Tue Oct 16 12:45:33 dbexit: really exiting now