MongoDB shell version: 2.5.4 connecting to: test ReplSetTest Starting Set ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 31000, 31001, 31002 ] 31000 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : "jstests/libs/key1", "port" : 31000, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "rs1", "dbpath" : "$set-$node", "binVersion" : "248", "restart" : undefined, "pathOpts" : { "node" : 0, "set" : "rs1" } } ReplSetTest Starting.... Resetting db path '/data/db/rs1-0' 2013-12-17T11:39:22.761-0500 shell: started program /Users/vkarpov/qa/QA/QA-424/mongod-248 --oplogSize 40 --keyFile jstests/libs/key1 --port 31000 --noprealloc --smallfiles --rest --replSet rs1 --dbpath /data/db/rs1-0 2013-12-17T11:39:22.762-0500 warning: Failed to connect to 127.0.0.1, reason: errno:61 Connection refused m31000| note: noprealloc may hurt performance in many applications m31000| Tue Dec 17 11:39:22.904 [initandlisten] MongoDB starting : pid=2228 port=31000 dbpath=/data/db/rs1-0 64-bit host=specter.local m31000| Tue Dec 17 11:39:22.905 [initandlisten] m31000| Tue Dec 17 11:39:22.905 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000 m31000| Tue Dec 17 11:39:22.905 [initandlisten] db version v2.4.8 m31000| Tue Dec 17 11:39:22.905 [initandlisten] git version: a350fc38922fbda2cec8d5dd842237b904eafc14 m31000| Tue Dec 17 11:39:22.905 [initandlisten] build info: Darwin bs-osx-106-x86-64-2.10gen.cc 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:32:41 PDT 2011; root:xnu-1504.15.3~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49 m31000| Tue Dec 17 11:39:22.905 [initandlisten] allocator: system m31000| Tue Dec 17 11:39:22.905 [initandlisten] options: { dbpath: "/data/db/rs1-0", keyFile: "jstests/libs/key1", noprealloc: true, oplogSize: 40, port: 31000, replSet: "rs1", rest: true, smallfiles: true } m31000| Tue Dec 17 11:39:22.905 [initandlisten] journal dir=/data/db/rs1-0/journal m31000| Tue Dec 17 11:39:22.905 [initandlisten] recover : no journal files present, no recovery needed m31000| Tue Dec 17 11:39:22.919 [FileAllocator] allocating new datafile /data/db/rs1-0/local.ns, filling with zeroes... m31000| Tue Dec 17 11:39:22.919 [FileAllocator] creating directory /data/db/rs1-0/_tmp 2013-12-17T11:39:22.964-0500 warning: Failed to connect to 127.0.0.1, reason: errno:61 Connection refused m31000| Tue Dec 17 11:39:22.975 [FileAllocator] done allocating datafile /data/db/rs1-0/local.ns, size: 16MB, took 0.055 secs m31000| Tue Dec 17 11:39:22.977 [FileAllocator] allocating new datafile /data/db/rs1-0/local.0, filling with zeroes... m31000| Tue Dec 17 11:39:23.006 [FileAllocator] done allocating datafile /data/db/rs1-0/local.0, size: 16MB, took 0.029 secs m31000| Tue Dec 17 11:39:23.030 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 110ms m31000| Tue Dec 17 11:39:23.031 [websvr] admin web console waiting for connections on port 32000 m31000| Tue Dec 17 11:39:23.031 [initandlisten] waiting for connections on port 31000 m31000| Tue Dec 17 11:39:23.032 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31000| Tue Dec 17 11:39:23.032 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31000| Tue Dec 17 11:39:23.165 [initandlisten] connection accepted from 127.0.0.1:65102 #1 (1 connection now open) [ connection to specter.local:31000 ] ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 31000, 31001, 31002 ] 31001 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : "jstests/libs/key1", "port" : 31001, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "rs1", "dbpath" : "$set-$node", "binVersion" : "248", "restart" : undefined, "pathOpts" : { "node" : 1, "set" : "rs1" } } ReplSetTest Starting.... Resetting db path '/data/db/rs1-1' 2013-12-17T11:39:23.177-0500 shell: started program /Users/vkarpov/qa/QA/QA-424/mongod-248 --oplogSize 40 --keyFile jstests/libs/key1 --port 31001 --noprealloc --smallfiles --rest --replSet rs1 --dbpath /data/db/rs1-1 2013-12-17T11:39:23.177-0500 warning: Failed to connect to 127.0.0.1, reason: errno:61 Connection refused m31001| note: noprealloc may hurt performance in many applications m31001| Tue Dec 17 11:39:23.205 [initandlisten] MongoDB starting : pid=2229 port=31001 dbpath=/data/db/rs1-1 64-bit host=specter.local m31001| Tue Dec 17 11:39:23.206 [initandlisten] m31001| Tue Dec 17 11:39:23.206 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000 m31001| Tue Dec 17 11:39:23.206 [initandlisten] db version v2.4.8 m31001| Tue Dec 17 11:39:23.206 [initandlisten] git version: a350fc38922fbda2cec8d5dd842237b904eafc14 m31001| Tue Dec 17 11:39:23.206 [initandlisten] build info: Darwin bs-osx-106-x86-64-2.10gen.cc 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:32:41 PDT 2011; root:xnu-1504.15.3~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49 m31001| Tue Dec 17 11:39:23.206 [initandlisten] allocator: system m31001| Tue Dec 17 11:39:23.206 [initandlisten] options: { dbpath: "/data/db/rs1-1", keyFile: "jstests/libs/key1", noprealloc: true, oplogSize: 40, port: 31001, replSet: "rs1", rest: true, smallfiles: true } m31001| Tue Dec 17 11:39:23.206 [initandlisten] journal dir=/data/db/rs1-1/journal m31001| Tue Dec 17 11:39:23.206 [initandlisten] recover : no journal files present, no recovery needed m31001| Tue Dec 17 11:39:23.218 [FileAllocator] allocating new datafile /data/db/rs1-1/local.ns, filling with zeroes... m31001| Tue Dec 17 11:39:23.218 [FileAllocator] creating directory /data/db/rs1-1/_tmp m31001| Tue Dec 17 11:39:23.271 [FileAllocator] done allocating datafile /data/db/rs1-1/local.ns, size: 16MB, took 0.052 secs m31001| Tue Dec 17 11:39:23.273 [FileAllocator] allocating new datafile /data/db/rs1-1/local.0, filling with zeroes... m31001| Tue Dec 17 11:39:23.369 [FileAllocator] done allocating datafile /data/db/rs1-1/local.0, size: 16MB, took 0.095 secs 2013-12-17T11:39:23.380-0500 warning: Failed to connect to 127.0.0.1, reason: errno:61 Connection refused m31001| Tue Dec 17 11:39:23.394 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 175ms m31001| Tue Dec 17 11:39:23.394 [websvr] admin web console waiting for connections on port 32001 m31001| Tue Dec 17 11:39:23.394 [initandlisten] waiting for connections on port 31001 m31001| Tue Dec 17 11:39:23.395 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31001| Tue Dec 17 11:39:23.395 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31001| Tue Dec 17 11:39:23.581 [initandlisten] connection accepted from 127.0.0.1:65105 #1 (1 connection now open) [ connection to specter.local:31000, connection to specter.local:31001 ] ReplSetTest n is : 2 ReplSetTest n: 2 ports: [ 31000, 31001, 31002 ] 31002 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : "jstests/libs/key1", "port" : 31002, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "rs1", "dbpath" : "$set-$node", "binVersion" : "248", "restart" : undefined, "pathOpts" : { "node" : 2, "set" : "rs1" } } ReplSetTest Starting.... Resetting db path '/data/db/rs1-2' 2013-12-17T11:39:23.588-0500 shell: started program /Users/vkarpov/qa/QA/QA-424/mongod-248 --oplogSize 40 --keyFile jstests/libs/key1 --port 31002 --noprealloc --smallfiles --rest --replSet rs1 --dbpath /data/db/rs1-2 2013-12-17T11:39:23.588-0500 warning: Failed to connect to 127.0.0.1, reason: errno:61 Connection refused m31002| note: noprealloc may hurt performance in many applications m31002| Tue Dec 17 11:39:23.616 [initandlisten] MongoDB starting : pid=2230 port=31002 dbpath=/data/db/rs1-2 64-bit host=specter.local m31002| Tue Dec 17 11:39:23.616 [initandlisten] m31002| Tue Dec 17 11:39:23.616 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000 m31002| Tue Dec 17 11:39:23.616 [initandlisten] db version v2.4.8 m31002| Tue Dec 17 11:39:23.616 [initandlisten] git version: a350fc38922fbda2cec8d5dd842237b904eafc14 m31002| Tue Dec 17 11:39:23.616 [initandlisten] build info: Darwin bs-osx-106-x86-64-2.10gen.cc 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:32:41 PDT 2011; root:xnu-1504.15.3~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49 m31002| Tue Dec 17 11:39:23.616 [initandlisten] allocator: system m31002| Tue Dec 17 11:39:23.616 [initandlisten] options: { dbpath: "/data/db/rs1-2", keyFile: "jstests/libs/key1", noprealloc: true, oplogSize: 40, port: 31002, replSet: "rs1", rest: true, smallfiles: true } m31002| Tue Dec 17 11:39:23.617 [initandlisten] journal dir=/data/db/rs1-2/journal m31002| Tue Dec 17 11:39:23.617 [initandlisten] recover : no journal files present, no recovery needed m31002| Tue Dec 17 11:39:23.630 [FileAllocator] allocating new datafile /data/db/rs1-2/local.ns, filling with zeroes... m31002| Tue Dec 17 11:39:23.630 [FileAllocator] creating directory /data/db/rs1-2/_tmp m31002| Tue Dec 17 11:39:23.653 [FileAllocator] done allocating datafile /data/db/rs1-2/local.ns, size: 16MB, took 0.021 secs m31002| Tue Dec 17 11:39:23.679 [FileAllocator] allocating new datafile /data/db/rs1-2/local.0, filling with zeroes... m31002| Tue Dec 17 11:39:23.703 [FileAllocator] done allocating datafile /data/db/rs1-2/local.0, size: 16MB, took 0.023 secs m31002| Tue Dec 17 11:39:23.732 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 101ms m31002| Tue Dec 17 11:39:23.733 [websvr] admin web console waiting for connections on port 32002 m31002| Tue Dec 17 11:39:23.733 [initandlisten] waiting for connections on port 31002 m31002| Tue Dec 17 11:39:23.736 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31002| Tue Dec 17 11:39:23.736 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31002| Tue Dec 17 11:39:23.790 [initandlisten] connection accepted from 127.0.0.1:65107 #1 (1 connection now open) [ connection to specter.local:31000, connection to specter.local:31001, connection to specter.local:31002 ] { "replSetInitiate" : { "_id" : "rs1", "members" : [ { "_id" : 0, "host" : "specter.local:31000" }, { "_id" : 1, "host" : "specter.local:31001" }, { "_id" : 2, "host" : "specter.local:31002" } ] } } m31000| Tue Dec 17 11:39:23.792 [conn1] note: no users configured in admin.system.users, allowing localhost access m31000| Tue Dec 17 11:39:23.793 [conn1] replSet replSetInitiate admin command received from client m31000| Tue Dec 17 11:39:23.793 [conn1] replSet replSetInitiate config object parses ok, 3 members specified m31001| Tue Dec 17 11:39:23.955 [initandlisten] connection accepted from 10.4.101.171:65108 #2 (2 connections now open) m31001| Tue Dec 17 11:39:23.955 [conn2] authenticate db: local { authenticate: 1, nonce: "6083facf8fe31020", user: "__system", key: "91bdf01eb3c1c883f03b7ffde6cdf264" } m31002| Tue Dec 17 11:39:23.956 [initandlisten] connection accepted from 10.4.101.171:65109 #2 (2 connections now open) m31002| Tue Dec 17 11:39:23.957 [conn2] authenticate db: local { authenticate: 1, nonce: "94af49fd20e88939", user: "__system", key: "adbe24ce2a896bf451490097b0be3595" } m31000| Tue Dec 17 11:39:23.957 [conn1] replSet replSetInitiate all members seem up m31000| Tue Dec 17 11:39:23.958 [conn1] ****** m31000| Tue Dec 17 11:39:23.958 [conn1] creating replication oplog of size: 40MB... m31000| Tue Dec 17 11:39:23.958 [FileAllocator] allocating new datafile /data/db/rs1-0/local.1, filling with zeroes... m31000| Tue Dec 17 11:39:24.128 [FileAllocator] done allocating datafile /data/db/rs1-0/local.1, size: 64MB, took 0.169 secs m31000| Tue Dec 17 11:39:24.166 [conn1] ****** m31000| Tue Dec 17 11:39:24.167 [conn1] replSet info saving a newer config version to local.system.replset m31000| Tue Dec 17 11:39:24.185 [conn1] replSet saveConfigLocally done m31000| Tue Dec 17 11:39:24.185 [conn1] replSet replSetInitiate config now saved locally. Should come online in about a minute. m31000| Tue Dec 17 11:39:24.185 [conn1] command admin.$cmd command: { replSetInitiate: { _id: "rs1", members: [ { _id: 0.0, host: "specter.local:31000" }, { _id: 1.0, host: "specter.local:31001" }, { _id: 2.0, host: "specter.local:31002" } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:228241 reslen:112 392ms { "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1 } m31001| Tue Dec 17 11:39:24.199 [conn1] note: no users configured in admin.system.users, allowing localhost access m31002| Tue Dec 17 11:39:24.200 [conn1] note: no users configured in admin.system.users, allowing localhost access m31000| Tue Dec 17 11:39:33.033 [rsStart] replSet I am specter.local:31000 m31000| Tue Dec 17 11:39:33.033 [rsStart] replSet STARTUP2 m31001| Tue Dec 17 11:39:33.396 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31002| Tue Dec 17 11:39:33.737 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31000| Tue Dec 17 11:39:34.034 [rsSync] replSet SECONDARY m31000| Tue Dec 17 11:39:35.035 [rsHealthPoll] replSet member specter.local:31001 is up m31000| Tue Dec 17 11:39:35.035 [rsHealthPoll] replSet member specter.local:31002 is up m31000| Tue Dec 17 11:39:35.035 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes m31000| Tue Dec 17 11:39:35.035 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes m31001| Tue Dec 17 11:39:40.241 [rsStart] trying to contact specter.local:31000 m31000| Tue Dec 17 11:39:40.242 [initandlisten] connection accepted from 10.4.101.171:65111 #2 (2 connections now open) m31000| Tue Dec 17 11:39:40.242 [conn2] authenticate db: local { authenticate: 1, nonce: "6e47ae711e01c614", user: "__system", key: "519cd091e58ee7a831f01caffb666dae" } m31001| Tue Dec 17 11:39:40.243 [rsStart] replSet I am specter.local:31001 m31001| Tue Dec 17 11:39:40.243 [rsStart] replSet got config version 1 from a remote, saving locally m31001| Tue Dec 17 11:39:40.243 [rsStart] replSet info saving a newer config version to local.system.replset m31001| Tue Dec 17 11:39:40.247 [rsStart] replSet saveConfigLocally done m31001| Tue Dec 17 11:39:40.247 [rsStart] replSet STARTUP2 m31001| Tue Dec 17 11:39:40.248 [rsSync] ****** m31001| Tue Dec 17 11:39:40.248 [rsSync] creating replication oplog of size: 40MB... m31001| Tue Dec 17 11:39:40.248 [FileAllocator] allocating new datafile /data/db/rs1-1/local.1, filling with zeroes... m31001| Tue Dec 17 11:39:40.398 [FileAllocator] done allocating datafile /data/db/rs1-1/local.1, size: 64MB, took 0.149 secs m31001| Tue Dec 17 11:39:40.435 [rsSync] ****** m31001| Tue Dec 17 11:39:40.435 [rsSync] replSet initial sync pending m31001| Tue Dec 17 11:39:40.435 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31002| Tue Dec 17 11:39:40.582 [rsStart] trying to contact specter.local:31000 m31000| Tue Dec 17 11:39:40.583 [initandlisten] connection accepted from 10.4.101.171:65112 #3 (3 connections now open) m31000| Tue Dec 17 11:39:40.583 [conn3] authenticate db: local { authenticate: 1, nonce: "53d035a59e1a5794", user: "__system", key: "f4aae2ebbc4aa4a409592a06af33ddb8" } m31002| Tue Dec 17 11:39:40.584 [rsStart] replSet I am specter.local:31002 m31002| Tue Dec 17 11:39:40.584 [rsStart] replSet got config version 1 from a remote, saving locally m31002| Tue Dec 17 11:39:40.584 [rsStart] replSet info saving a newer config version to local.system.replset m31002| Tue Dec 17 11:39:40.587 [rsStart] replSet saveConfigLocally done m31002| Tue Dec 17 11:39:40.588 [rsStart] replSet STARTUP2 m31002| Tue Dec 17 11:39:40.588 [rsSync] ****** m31002| Tue Dec 17 11:39:40.588 [rsSync] creating replication oplog of size: 40MB... m31002| Tue Dec 17 11:39:40.588 [FileAllocator] allocating new datafile /data/db/rs1-2/local.1, filling with zeroes... m31002| Tue Dec 17 11:39:40.872 [FileAllocator] done allocating datafile /data/db/rs1-2/local.1, size: 64MB, took 0.283 secs m31002| Tue Dec 17 11:39:41.002 [rsSync] ****** m31002| Tue Dec 17 11:39:41.002 [rsSync] replSet initial sync pending m31002| Tue Dec 17 11:39:41.002 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31000| Tue Dec 17 11:39:41.884 [rsHealthPoll] replset info specter.local:31001 thinks that we are down m31000| Tue Dec 17 11:39:41.884 [rsHealthPoll] replset info specter.local:31002 thinks that we are down m31000| Tue Dec 17 11:39:41.884 [rsHealthPoll] replSet member specter.local:31001 is now in state STARTUP2 m31000| Tue Dec 17 11:39:41.884 [rsHealthPoll] replSet member specter.local:31002 is now in state STARTUP2 m31000| Tue Dec 17 11:39:41.884 [rsMgr] not electing self, specter.local:31002 would veto with 'I don't think specter.local:31000 is electable' m31000| Tue Dec 17 11:39:41.885 [rsMgr] not electing self, specter.local:31002 would veto with 'I don't think specter.local:31000 is electable' m31001| Tue Dec 17 11:39:42.244 [rsHealthPoll] replSet member specter.local:31000 is up m31001| Tue Dec 17 11:39:42.244 [rsHealthPoll] replSet member specter.local:31000 is now in state SECONDARY m31002| Tue Dec 17 11:39:42.244 [initandlisten] connection accepted from 10.4.101.171:65113 #3 (3 connections now open) m31002| Tue Dec 17 11:39:42.244 [conn3] authenticate db: local { authenticate: 1, nonce: "a58b36f659e5511f", user: "__system", key: "02e62d410478df124e29d29e423ec09f" } m31001| Tue Dec 17 11:39:42.245 [rsHealthPoll] replset info specter.local:31002 thinks that we are down m31001| Tue Dec 17 11:39:42.245 [rsHealthPoll] replSet member specter.local:31002 is up m31001| Tue Dec 17 11:39:42.245 [rsHealthPoll] replSet member specter.local:31002 is now in state STARTUP2 m31002| Tue Dec 17 11:39:42.585 [rsHealthPoll] replSet member specter.local:31000 is up m31002| Tue Dec 17 11:39:42.586 [rsHealthPoll] replSet member specter.local:31000 is now in state SECONDARY m31001| Tue Dec 17 11:39:42.586 [initandlisten] connection accepted from 10.4.101.171:65114 #3 (3 connections now open) m31001| Tue Dec 17 11:39:42.586 [conn3] authenticate db: local { authenticate: 1, nonce: "91866132b64998f3", user: "__system", key: "590d8ba6250dad7015d7683bc30361b3" } m31002| Tue Dec 17 11:39:42.586 [rsHealthPoll] replSet member specter.local:31001 is up m31002| Tue Dec 17 11:39:42.586 [rsHealthPoll] replSet member specter.local:31001 is now in state STARTUP2 m31001| Tue Dec 17 11:39:45.887 [initandlisten] connection accepted from 10.4.101.171:65115 #4 (4 connections now open) m31001| Tue Dec 17 11:39:45.887 [conn4] authenticate db: local { authenticate: 1, nonce: "41d173da51362195", user: "__system", key: "31143a5f301a9f334e7c9c21c7b04c66" } m31001| Tue Dec 17 11:39:45.894 [conn2] end connection 10.4.101.171:65108 (3 connections now open) m31000| Tue Dec 17 11:39:47.888 [rsMgr] replSet info electSelf 0 m31001| Tue Dec 17 11:39:47.888 [conn4] replSet RECOVERING m31001| Tue Dec 17 11:39:47.888 [conn4] replSet info voting yea for specter.local:31000 (0) m31002| Tue Dec 17 11:39:47.888 [conn2] replSet RECOVERING m31002| Tue Dec 17 11:39:47.888 [conn2] replSet info voting yea for specter.local:31000 (0) m31000| Tue Dec 17 11:39:47.898 [rsMgr] replSet PRIMARY m31001| Tue Dec 17 11:39:48.248 [rsHealthPoll] replSet member specter.local:31000 is now in state PRIMARY m31001| Tue Dec 17 11:39:48.248 [rsHealthPoll] replSet member specter.local:31002 is now in state RECOVERING m31002| Tue Dec 17 11:39:48.588 [rsHealthPoll] replSet member specter.local:31000 is now in state PRIMARY m31002| Tue Dec 17 11:39:48.588 [rsHealthPoll] replSet member specter.local:31001 is now in state RECOVERING m31000| Tue Dec 17 11:39:49.889 [rsHealthPoll] replSet member specter.local:31001 is now in state RECOVERING m31000| Tue Dec 17 11:39:49.889 [rsHealthPoll] replSet member specter.local:31002 is now in state RECOVERING m31000| Tue Dec 17 11:39:56.252 [conn2] end connection 10.4.101.171:65111 (2 connections now open) m31000| Tue Dec 17 11:39:56.253 [initandlisten] connection accepted from 10.4.101.171:65116 #4 (3 connections now open) m31000| Tue Dec 17 11:39:56.253 [conn4] authenticate db: local { authenticate: 1, nonce: "f2620c0da908a7f8", user: "__system", key: "3dac860ac1baa2e470359a1e09e02612" } m31001| Tue Dec 17 11:39:56.436 [rsSync] replSet initial sync pending m31001| Tue Dec 17 11:39:56.436 [rsSync] replSet syncing to: specter.local:31000 m31000| Tue Dec 17 11:39:56.437 [initandlisten] connection accepted from 10.4.101.171:65117 #5 (4 connections now open) m31000| Tue Dec 17 11:39:56.437 [conn5] authenticate db: local { authenticate: 1, nonce: "3c0a0bc15a2d4b4f", user: "__system", key: "ade4b5e093ae0d31c1343f874783f085" } m31001| Tue Dec 17 11:39:56.439 [rsSync] build index local.me { _id: 1 } m31001| Tue Dec 17 11:39:56.439 [rsSync] build index done. scanned 0 total records. 0 secs m31001| Tue Dec 17 11:39:56.441 [rsSync] build index local.replset.minvalid { _id: 1 } m31001| Tue Dec 17 11:39:56.441 [rsSync] build index done. scanned 0 total records. 0 secs m31001| Tue Dec 17 11:39:56.442 [rsSync] replSet initial sync drop all databases m31001| Tue Dec 17 11:39:56.442 [rsSync] dropAllDatabasesExceptLocal 1 m31001| Tue Dec 17 11:39:56.442 [rsSync] replSet initial sync clone all databases m31001| Tue Dec 17 11:39:56.442 [rsSync] replSet initial sync cloning db: admin m31000| Tue Dec 17 11:39:56.443 [initandlisten] connection accepted from 10.4.101.171:65118 #6 (5 connections now open) m31000| Tue Dec 17 11:39:56.443 [conn6] authenticate db: local { authenticate: 1, nonce: "228ba392affe61c0", user: "__system", key: "093d3af35ff1f8051db1a0506afc9a20" } m31001| Tue Dec 17 11:39:56.443 [rsSync] replSet initial sync data copy, starting syncup m31001| Tue Dec 17 11:39:56.443 [rsSync] oplog sync 1 of 3 m31001| Tue Dec 17 11:39:56.443 [rsSync] oplog sync 2 of 3 m31001| Tue Dec 17 11:39:56.443 [rsSync] replSet initial sync building indexes m31001| Tue Dec 17 11:39:56.443 [rsSync] replSet initial sync cloning indexes for : admin m31001| Tue Dec 17 11:39:56.444 [rsSync] oplog sync 3 of 3 m31000| Tue Dec 17 11:39:56.444 [conn6] end connection 10.4.101.171:65118 (4 connections now open) m31001| Tue Dec 17 11:39:56.444 [rsSync] replSet initial sync finishing up m31001| Tue Dec 17 11:39:56.452 [rsSync] replSet set minValid=52b07e3c:1 m31001| Tue Dec 17 11:39:56.460 [rsSync] replSet initial sync done m31000| Tue Dec 17 11:39:56.460 [conn5] end connection 10.4.101.171:65117 (3 connections now open) m31000| Tue Dec 17 11:39:56.593 [conn3] end connection 10.4.101.171:65112 (2 connections now open) m31000| Tue Dec 17 11:39:56.594 [initandlisten] connection accepted from 10.4.101.171:65119 #7 (3 connections now open) m31000| Tue Dec 17 11:39:56.594 [conn7] authenticate db: local { authenticate: 1, nonce: "b2b34f0ed5b6e550", user: "__system", key: "782925a217e34647e4788c7dc769e085" } m31002| Tue Dec 17 11:39:57.004 [rsSync] replSet initial sync pending m31002| Tue Dec 17 11:39:57.004 [rsSync] replSet syncing to: specter.local:31000 m31000| Tue Dec 17 11:39:57.005 [initandlisten] connection accepted from 10.4.101.171:65120 #8 (4 connections now open) m31000| Tue Dec 17 11:39:57.005 [conn8] authenticate db: local { authenticate: 1, nonce: "bb17b58094190dca", user: "__system", key: "2795ce226a1a9560882c5b7a8636db0e" } m31002| Tue Dec 17 11:39:57.006 [rsSync] build index local.me { _id: 1 } m31002| Tue Dec 17 11:39:57.007 [rsSync] build index done. scanned 0 total records. 0 secs m31002| Tue Dec 17 11:39:57.008 [rsSync] build index local.replset.minvalid { _id: 1 } m31002| Tue Dec 17 11:39:57.009 [rsSync] build index done. scanned 0 total records. 0 secs m31002| Tue Dec 17 11:39:57.009 [rsSync] replSet initial sync drop all databases m31002| Tue Dec 17 11:39:57.009 [rsSync] dropAllDatabasesExceptLocal 1 m31002| Tue Dec 17 11:39:57.009 [rsSync] replSet initial sync clone all databases m31002| Tue Dec 17 11:39:57.009 [rsSync] replSet initial sync cloning db: admin m31000| Tue Dec 17 11:39:57.009 [initandlisten] connection accepted from 10.4.101.171:65121 #9 (5 connections now open) m31000| Tue Dec 17 11:39:57.010 [conn9] authenticate db: local { authenticate: 1, nonce: "5fe8198f09d1e44e", user: "__system", key: "7e379cf900aac0e3bdbb4ceb838bb88c" } m31002| Tue Dec 17 11:39:57.010 [rsSync] replSet initial sync data copy, starting syncup m31002| Tue Dec 17 11:39:57.010 [rsSync] oplog sync 1 of 3 m31002| Tue Dec 17 11:39:57.010 [rsSync] oplog sync 2 of 3 m31002| Tue Dec 17 11:39:57.010 [rsSync] replSet initial sync building indexes m31002| Tue Dec 17 11:39:57.010 [rsSync] replSet initial sync cloning indexes for : admin m31002| Tue Dec 17 11:39:57.011 [rsSync] oplog sync 3 of 3 m31000| Tue Dec 17 11:39:57.011 [conn9] end connection 10.4.101.171:65121 (4 connections now open) m31002| Tue Dec 17 11:39:57.011 [rsSync] replSet initial sync finishing up m31002| Tue Dec 17 11:39:57.025 [rsSync] replSet set minValid=52b07e3c:1 m31002| Tue Dec 17 11:39:57.033 [rsSync] replSet initial sync done m31000| Tue Dec 17 11:39:57.033 [conn8] end connection 10.4.101.171:65120 (3 connections now open) m31001| Tue Dec 17 11:39:57.261 [rsBackgroundSync] replSet syncing to: specter.local:31000 m31000| Tue Dec 17 11:39:57.261 [initandlisten] connection accepted from 10.4.101.171:65122 #10 (4 connections now open) m31000| Tue Dec 17 11:39:57.261 [conn10] authenticate db: local { authenticate: 1, nonce: "8e01ca2d747ed59a", user: "__system", key: "4a498191c8af3cdcc1a8c2ca8fdd5a05" } m31001| Tue Dec 17 11:39:57.461 [rsSyncNotifier] replset setting oplog notifier to specter.local:31000 m31000| Tue Dec 17 11:39:57.461 [initandlisten] connection accepted from 10.4.101.171:65123 #11 (5 connections now open) m31000| Tue Dec 17 11:39:57.461 [conn11] authenticate db: local { authenticate: 1, nonce: "bdd6625051d2f0ab", user: "__system", key: "c830e483c004d327577f8522ff07d1ad" } m31002| Tue Dec 17 11:39:57.601 [rsBackgroundSync] replSet syncing to: specter.local:31000 m31000| Tue Dec 17 11:39:57.602 [initandlisten] connection accepted from 10.4.101.171:65124 #12 (6 connections now open) m31000| Tue Dec 17 11:39:57.602 [conn12] authenticate db: local { authenticate: 1, nonce: "9eecd01393fbbc72", user: "__system", key: "38207075d0910a695a1abcddb7f138c0" } m31002| Tue Dec 17 11:39:58.034 [rsSyncNotifier] replset setting oplog notifier to specter.local:31000 m31000| Tue Dec 17 11:39:58.034 [initandlisten] connection accepted from 10.4.101.171:65125 #13 (7 connections now open) m31000| Tue Dec 17 11:39:58.035 [conn13] authenticate db: local { authenticate: 1, nonce: "6a768ee68a87bd0f", user: "__system", key: "0d61fe87f92bb4483c62b0f0cd3e39a8" } m31001| Tue Dec 17 11:39:58.462 [rsSync] replSet SECONDARY m31000| Tue Dec 17 11:39:58.464 [slaveTracking] build index local.slaves { _id: 1 } m31000| Tue Dec 17 11:39:58.465 [slaveTracking] build index done. scanned 0 total records. 0 secs m31002| Tue Dec 17 11:39:58.594 [rsHealthPoll] replSet member specter.local:31001 is now in state SECONDARY m31002| Tue Dec 17 11:39:59.035 [rsSync] replSet SECONDARY WARNING: The 'addUser' shell helper is DEPRECATED. Please use 'createUser' instead WARNING: connected to a server which does not support write commands. m31000| Tue Dec 17 11:39:59.226 [FileAllocator] allocating new datafile /data/db/rs1-0/admin.ns, filling with zeroes... Successfully added user: { "user" : "admin", "roles" : [ "userAdminAnyDatabase", "readWriteAnyDatabase", "dbAdminAnyDatabase", "clusterAdmin" ], "_id" : ObjectId("52b07e5f43854624c8940e7f") } m31000| Tue Dec 17 11:39:59.254 [FileAllocator] done allocating datafile /data/db/rs1-0/admin.ns, size: 16MB, took 0.027 secs m31000| Tue Dec 17 11:39:59.279 [FileAllocator] allocating new datafile /data/db/rs1-0/admin.0, filling with zeroes... m31000| Tue Dec 17 11:39:59.300 [FileAllocator] done allocating datafile /data/db/rs1-0/admin.0, size: 16MB, took 0.02 secs m31000| Tue Dec 17 11:39:59.327 [conn1] build index admin.system.users { _id: 1 } m31000| Tue Dec 17 11:39:59.328 [conn1] build index done. scanned 0 total records. 0 secs m31000| Tue Dec 17 11:39:59.328 [conn1] build index admin.system.users { user: 1, userSource: 1 } m31000| Tue Dec 17 11:39:59.329 [conn1] build index done. scanned 0 total records. 0 secs m31000| Tue Dec 17 11:39:59.329 [conn1] insert admin.system.users ninserted:1 keyUpdates:0 103ms m31002| Tue Dec 17 11:39:59.330 [FileAllocator] allocating new datafile /data/db/rs1-2/admin.ns, filling with zeroes... m31001| Tue Dec 17 11:39:59.330 [FileAllocator] allocating new datafile /data/db/rs1-1/admin.ns, filling with zeroes... m31002| Tue Dec 17 11:39:59.353 [FileAllocator] done allocating datafile /data/db/rs1-2/admin.ns, size: 16MB, took 0.023 secs m31001| Tue Dec 17 11:39:59.376 [FileAllocator] done allocating datafile /data/db/rs1-1/admin.ns, size: 16MB, took 0.046 secs m31002| Tue Dec 17 11:39:59.401 [FileAllocator] allocating new datafile /data/db/rs1-2/admin.0, filling with zeroes... m31001| Tue Dec 17 11:39:59.415 [FileAllocator] allocating new datafile /data/db/rs1-1/admin.0, filling with zeroes... m31002| Tue Dec 17 11:39:59.428 [FileAllocator] done allocating datafile /data/db/rs1-2/admin.0, size: 16MB, took 0.026 secs m31001| Tue Dec 17 11:39:59.458 [FileAllocator] done allocating datafile /data/db/rs1-1/admin.0, size: 16MB, took 0.043 secs m31002| Tue Dec 17 11:39:59.478 [repl writer worker 1] build index admin.system.users { _id: 1 } m31002| Tue Dec 17 11:39:59.479 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31002| Tue Dec 17 11:39:59.479 [repl writer worker 1] build index admin.system.users { user: 1, userSource: 1 } m31002| Tue Dec 17 11:39:59.479 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31000| Tue Dec 17 11:39:59.490 [conn1] command admin.$cmd command: { getlasterror: 1.0, w: "majority", wtimeout: 30000.0 } ntoreturn:1 keyUpdates:0 reslen:204 161ms -------- Upgrading set to 26 m31001| Tue Dec 17 11:39:59.493 [repl writer worker 1] build index admin.system.users { _id: 1 } m31001| Tue Dec 17 11:39:59.494 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31001| Tue Dec 17 11:39:59.494 [repl writer worker 1] build index admin.system.users { user: 1, userSource: 1 } m31001| Tue Dec 17 11:39:59.495 [repl writer worker 1] build index done. scanned 0 total records. 0 secs { "n0" : { "binVersion" : "26", "keyFile" : "jstests/libs/key1", "clusterAuthMode" : "keyFile" }, "n1" : { "binVersion" : "26", "keyFile" : "jstests/libs/key1", "clusterAuthMode" : "keyFile" }, "n2" : { "binVersion" : "26", "keyFile" : "jstests/libs/key1", "clusterAuthMode" : "keyFile" } } ReplSetTest n: 1 ports: [ 31000, 31001, 31002 ] 31001 number ReplSetTest stop *** Shutting down mongod in port 31001 *** m31001| Tue Dec 17 11:39:59.498 [signalProcessingThread] got signal 15 (Terminated: 15), will terminate after current cmd ends m31001| Tue Dec 17 11:39:59.498 [signalProcessingThread] now exiting m31001| Tue Dec 17 11:39:59.498 dbexit: m31001| Tue Dec 17 11:39:59.498 [signalProcessingThread] shutdown: going to close listening sockets... m31001| Tue Dec 17 11:39:59.498 [signalProcessingThread] closing listening socket: 16 m31001| Tue Dec 17 11:39:59.498 [signalProcessingThread] closing listening socket: 17 m31001| Tue Dec 17 11:39:59.499 [signalProcessingThread] closing listening socket: 18 m31001| Tue Dec 17 11:39:59.499 [signalProcessingThread] removing socket file: /tmp/mongodb-31001.sock m31001| Tue Dec 17 11:39:59.499 [signalProcessingThread] shutdown: going to flush diaglog... m31001| Tue Dec 17 11:39:59.499 [signalProcessingThread] shutdown: going to close sockets... m31001| Tue Dec 17 11:39:59.499 [signalProcessingThread] shutdown: waiting for fs preallocator... m31001| Tue Dec 17 11:39:59.499 [signalProcessingThread] shutdown: lock for final commit... m31001| Tue Dec 17 11:39:59.499 [signalProcessingThread] shutdown: final commit... m31001| Tue Dec 17 11:39:59.502 [rsBackgroundSync] Socket recv() errno:9 Bad file descriptor 10.4.101.171:31000 m31001| Tue Dec 17 11:39:59.502 [rsBackgroundSync] SocketException: remote: 10.4.101.171:31000 error: 9001 socket exception [RECV_ERROR] server [10.4.101.171:31000] m31001| Tue Dec 17 11:39:59.502 [rsSyncNotifier] Socket recv() errno:9 Bad file descriptor 10.4.101.171:31000 m31001| Tue Dec 17 11:39:59.502 [rsSyncNotifier] SocketException: remote: 10.4.101.171:31000 error: 9001 socket exception [RECV_ERROR] server [10.4.101.171:31000] m31001| Tue Dec 17 11:39:59.502 [conn3] end connection 10.4.101.171:65114 (2 connections now open) m31001| Tue Dec 17 11:39:59.503 [conn1] end connection 127.0.0.1:65105 (2 connections now open) m31001| Tue Dec 17 11:39:59.503 [conn4] end connection 10.4.101.171:65115 (2 connections now open) m31000| Tue Dec 17 11:39:59.503 [conn4] end connection 10.4.101.171:65116 (6 connections now open) m31002| Tue Dec 17 11:39:59.503 [conn3] end connection 10.4.101.171:65113 (2 connections now open) m31001| Tue Dec 17 11:39:59.519 [signalProcessingThread] shutdown: closing all files... m31001| Tue Dec 17 11:39:59.519 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: specter.local:31000 m31001| Tue Dec 17 11:39:59.519 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: specter.local:31000 m31001| Tue Dec 17 11:39:59.527 [signalProcessingThread] closeAllFiles() finished m31001| Tue Dec 17 11:39:59.527 [signalProcessingThread] journalCleanup... m31001| Tue Dec 17 11:39:59.527 [signalProcessingThread] removeJournalFiles m31001| Tue Dec 17 11:39:59.528 [signalProcessingThread] shutdown: removing fs lock... m31001| Tue Dec 17 11:39:59.528 dbexit: really exiting now m31000| Tue Dec 17 11:39:59.895 [rsHealthPoll] DBClientCursor::init call() failed m31000| Tue Dec 17 11:39:59.895 [rsHealthPoll] replset info specter.local:31001 heartbeat failed, retrying m31002| Tue Dec 17 11:39:59.895 [conn2] end connection 10.4.101.171:65109 (1 connection now open) m31002| Tue Dec 17 11:39:59.896 [initandlisten] connection accepted from 10.4.101.171:65127 #4 (2 connections now open) m31000| Tue Dec 17 11:39:59.896 [rsHealthPoll] replSet info specter.local:31001 is down (or slow to respond): m31000| Tue Dec 17 11:39:59.896 [rsHealthPoll] replSet member specter.local:31001 is now in state DOWN m31002| Tue Dec 17 11:39:59.896 [conn4] authenticate db: local { authenticate: 1, nonce: "abe5ef84d6eed90d", user: "__system", key: "b8eb692229fd2a388a817da9e40fce0f" } m31000| Tue Dec 17 11:39:59.896 [rsHealthPoll] replSet member specter.local:31002 is now in state SECONDARY 2013-12-17T11:40:00.499-0500 shell: stopped mongo program on port 31001 ReplSetTest n is : connection to specter.local:31001 ReplSetTest n: 1 ports: [ 31000, 31001, 31002 ] 31001 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : "jstests/libs/key1", "port" : 31001, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "rs1", "dbpath" : "$set-$node", "binVersion" : "26", "clusterAuthMode" : "keyFile", "restart" : true, "pathOpts" : { "node" : 1, "set" : "rs1" } } ReplSetTest (Re)Starting.... 2013-12-17T11:40:00.501-0500 shell: started program /Users/vkarpov/qa/QA/QA-424/mongod-26 --oplogSize 40 --keyFile jstests/libs/key1 --port 31001 --noprealloc --smallfiles --rest --replSet rs1 --dbpath /data/db/rs1-1 --clusterAuthMode keyFile 2013-12-17T11:40:00.502-0500 warning: Failed to connect to 127.0.0.1, reason: errno:61 Connection refused m31001| 2013-12-17T11:40:00.525-0500 ** WARNING: --rest is specified without --httpinterface, m31001| 2013-12-17T11:40:00.525-0500 ** enabling http interface m31001| note: noprealloc may hurt performance in many applications m31001| 2013-12-17T11:40:00.528-0500 [initandlisten] MongoDB starting : pid=2231 port=31001 dbpath=/data/db/rs1-1 64-bit host=specter.local m31001| 2013-12-17T11:40:00.528-0500 [initandlisten] m31001| 2013-12-17T11:40:00.528-0500 [initandlisten] ** NOTE: This is a development version (2.5.4) of MongoDB. m31001| 2013-12-17T11:40:00.528-0500 [initandlisten] ** Not recommended for production. m31001| 2013-12-17T11:40:00.528-0500 [initandlisten] m31001| 2013-12-17T11:40:00.528-0500 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000 m31001| 2013-12-17T11:40:00.528-0500 [initandlisten] m31001| 2013-12-17T11:40:00.528-0500 [initandlisten] db version v2.5.4 m31001| 2013-12-17T11:40:00.528-0500 [initandlisten] git version: ffd52e5f46cf2ba74ba931c78da62d4a7f480d8e m31001| 2013-12-17T11:40:00.528-0500 [initandlisten] build info: Darwin bs-osx-108-x86-64-2.10gen.cc 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan 6 22:37:10 PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49 m31001| 2013-12-17T11:40:00.528-0500 [initandlisten] allocator: system m31001| 2013-12-17T11:40:00.528-0500 [initandlisten] options: { clusterAuthMode: "keyFile", dbpath: "/data/db/rs1-1", keyFile: "jstests/libs/key1", noprealloc: true, oplogSize: 40, port: 31001, replSet: "rs1", rest: true, smallfiles: true } m31001| 2013-12-17T11:40:00.528-0500 [initandlisten] journal dir=/data/db/rs1-1/journal m31001| 2013-12-17T11:40:00.529-0500 [initandlisten] recover : no journal files present, no recovery needed m31001| 2013-12-17T11:40:00.587-0500 [initandlisten] waiting for connections on port 31001 m31001| 2013-12-17T11:40:00.587-0500 [websvr] admin web console waiting for connections on port 32001 m31001| 2013-12-17T11:40:00.588-0500 [rsStart] replSet I am specter.local:31001 m31002| Tue Dec 17 11:40:00.595 [rsHealthPoll] DBClientCursor::init call() failed m31002| Tue Dec 17 11:40:00.595 [rsHealthPoll] replset info specter.local:31001 heartbeat failed, retrying m31001| 2013-12-17T11:40:00.596-0500 [initandlisten] connection accepted from 10.4.101.171:65129 #1 (1 connection now open) m31001| 2013-12-17T11:40:00.596-0500 [rsStart] replSet STARTUP2 m31001| 2013-12-17T11:40:00.596-0500 [conn1] authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" } m31001| 2013-12-17T11:40:00.597-0500 [rsSync] replSet SECONDARY m31002| Tue Dec 17 11:40:00.597 [rsHealthPoll] replset info specter.local:31001 thinks that we are down m31001| 2013-12-17T11:40:00.703-0500 [initandlisten] connection accepted from 127.0.0.1:65130 #2 (2 connections now open) [ connection to specter.local:31000, connection to specter.local:31001, connection to specter.local:31002 ] ReplSetTest waitForIndicator state on connection to specter.local:31001 [ 1, 2, 7 ] ReplSetTest waitForIndicator from node connection to specter.local:31001 m31000| Tue Dec 17 11:40:00.706 [conn1] command denied: { replSetGetStatus: 1.0 } ReplSetTest waitForIndicator Initial status ( timeout : 30000 ) : { "ok" : 0, "errmsg" : "unauthorized" } m31000| Tue Dec 17 11:40:00.909 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:40:01.110 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:40:01.313 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:40:01.515 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:40:01.718 [conn1] command denied: { replSetGetStatus: 1.0 } m31001| 2013-12-17T11:40:01.896-0500 [initandlisten] connection accepted from 10.4.101.171:65131 #3 (3 connections now open) m31001| 2013-12-17T11:40:01.897-0500 [conn3] authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" } m31000| Tue Dec 17 11:40:01.897 [rsHealthPoll] replset info specter.local:31001 thinks that we are down m31000| Tue Dec 17 11:40:01.897 [rsHealthPoll] replSet member specter.local:31001 is up m31000| Tue Dec 17 11:40:01.897 [rsHealthPoll] replSet member specter.local:31001 is now in state SECONDARY m31000| Tue Dec 17 11:40:01.920 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:40:02.123 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:40:02.325 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:40:02.528 [conn1] command denied: { replSetGetStatus: 1.0 } m31002| Tue Dec 17 11:40:02.590 [initandlisten] connection accepted from 10.4.101.171:65133 #5 (3 connections now open) m31000| Tue Dec 17 11:40:02.590 [initandlisten] connection accepted from 10.4.101.171:65132 #14 (7 connections now open) m31000| Tue Dec 17 11:40:02.590 [conn14] authenticate db: local { authenticate: 1, nonce: "30df081a1512fd27", user: "__system", key: "132b4a686b909a3742585fa83df4768d" } m31002| Tue Dec 17 11:40:02.590 [conn5] authenticate db: local { authenticate: 1, nonce: "1ac9e3786def1741", user: "__system", key: "cae21ab1111421d275d49f92fb8e2341" } m31001| 2013-12-17T11:40:02.591-0500 [rsHealthPoll] replSet member specter.local:31000 is up m31001| 2013-12-17T11:40:02.591-0500 [rsHealthPoll] replSet member specter.local:31002 is up m31001| 2013-12-17T11:40:02.591-0500 [rsHealthPoll] replSet member specter.local:31000 is now in state PRIMARY m31001| 2013-12-17T11:40:02.591-0500 [rsHealthPoll] replSet member specter.local:31002 is now in state SECONDARY m31000| Tue Dec 17 11:40:02.730 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:40:02.932 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:40:03.135 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:40:03.338 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:40:03.540 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:40:03.743 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:40:03.945 [conn1] command denied: { replSetGetStatus: 1.0 } ^Cspecter:QA-424 vkarpov$ ./mongo-26 repl_upgrade_normal.js MongoDB shell version: 2.5.4 connecting to: test ReplSetTest Starting Set ReplSetTest n is : 0 ReplSetTest n: 0 ports: [ 31000, 31001, 31002 ] 31000 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : "jstests/libs/key1", "port" : 31000, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "rs1", "dbpath" : "$set-$node", "binVersion" : "248", "restart" : undefined, "pathOpts" : { "node" : 0, "set" : "rs1" } } ReplSetTest Starting.... Resetting db path '/data/db/rs1-0' 2013-12-17T11:52:12.772-0500 shell: started program /Users/vkarpov/qa/QA/QA-424/mongod-248 --oplogSize 40 --keyFile jstests/libs/key1 --port 31000 --noprealloc --smallfiles --rest --replSet rs1 --dbpath /data/db/rs1-0 2013-12-17T11:52:12.774-0500 warning: Failed to connect to 127.0.0.1, reason: errno:61 Connection refused m31000| note: noprealloc may hurt performance in many applications m31000| Tue Dec 17 11:52:12.802 [initandlisten] MongoDB starting : pid=2240 port=31000 dbpath=/data/db/rs1-0 64-bit host=specter.local m31000| Tue Dec 17 11:52:12.802 [initandlisten] m31000| Tue Dec 17 11:52:12.802 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000 m31000| Tue Dec 17 11:52:12.802 [initandlisten] db version v2.4.8 m31000| Tue Dec 17 11:52:12.802 [initandlisten] git version: a350fc38922fbda2cec8d5dd842237b904eafc14 m31000| Tue Dec 17 11:52:12.802 [initandlisten] build info: Darwin bs-osx-106-x86-64-2.10gen.cc 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:32:41 PDT 2011; root:xnu-1504.15.3~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49 m31000| Tue Dec 17 11:52:12.802 [initandlisten] allocator: system m31000| Tue Dec 17 11:52:12.802 [initandlisten] options: { dbpath: "/data/db/rs1-0", keyFile: "jstests/libs/key1", noprealloc: true, oplogSize: 40, port: 31000, replSet: "rs1", rest: true, smallfiles: true } m31000| Tue Dec 17 11:52:12.809 [initandlisten] journal dir=/data/db/rs1-0/journal m31000| Tue Dec 17 11:52:12.809 [initandlisten] recover : no journal files present, no recovery needed m31000| Tue Dec 17 11:52:12.821 [FileAllocator] allocating new datafile /data/db/rs1-0/local.ns, filling with zeroes... m31000| Tue Dec 17 11:52:12.821 [FileAllocator] creating directory /data/db/rs1-0/_tmp m31000| Tue Dec 17 11:52:12.843 [FileAllocator] done allocating datafile /data/db/rs1-0/local.ns, size: 16MB, took 0.021 secs m31000| Tue Dec 17 11:52:12.870 [FileAllocator] allocating new datafile /data/db/rs1-0/local.0, filling with zeroes... m31000| Tue Dec 17 11:52:12.893 [FileAllocator] done allocating datafile /data/db/rs1-0/local.0, size: 16MB, took 0.023 secs m31000| Tue Dec 17 11:52:12.923 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31000| Tue Dec 17 11:52:12.923 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31000| Tue Dec 17 11:52:12.923 [websvr] admin web console waiting for connections on port 32000 m31000| Tue Dec 17 11:52:12.923 [initandlisten] waiting for connections on port 31000 m31000| Tue Dec 17 11:52:12.976 [initandlisten] connection accepted from 127.0.0.1:65257 #1 (1 connection now open) [ connection to specter.local:31000 ] ReplSetTest n is : 1 ReplSetTest n: 1 ports: [ 31000, 31001, 31002 ] 31001 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : "jstests/libs/key1", "port" : 31001, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "rs1", "dbpath" : "$set-$node", "binVersion" : "248", "restart" : undefined, "pathOpts" : { "node" : 1, "set" : "rs1" } } ReplSetTest Starting.... Resetting db path '/data/db/rs1-1' 2013-12-17T11:52:13.022-0500 shell: started program /Users/vkarpov/qa/QA/QA-424/mongod-248 --oplogSize 40 --keyFile jstests/libs/key1 --port 31001 --noprealloc --smallfiles --rest --replSet rs1 --dbpath /data/db/rs1-1 2013-12-17T11:52:13.022-0500 warning: Failed to connect to 127.0.0.1, reason: errno:61 Connection refused m31001| note: noprealloc may hurt performance in many applications m31001| Tue Dec 17 11:52:13.058 [initandlisten] MongoDB starting : pid=2241 port=31001 dbpath=/data/db/rs1-1 64-bit host=specter.local m31001| Tue Dec 17 11:52:13.058 [initandlisten] m31001| Tue Dec 17 11:52:13.058 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000 m31001| Tue Dec 17 11:52:13.058 [initandlisten] db version v2.4.8 m31001| Tue Dec 17 11:52:13.058 [initandlisten] git version: a350fc38922fbda2cec8d5dd842237b904eafc14 m31001| Tue Dec 17 11:52:13.058 [initandlisten] build info: Darwin bs-osx-106-x86-64-2.10gen.cc 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:32:41 PDT 2011; root:xnu-1504.15.3~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49 m31001| Tue Dec 17 11:52:13.058 [initandlisten] allocator: system m31001| Tue Dec 17 11:52:13.058 [initandlisten] options: { dbpath: "/data/db/rs1-1", keyFile: "jstests/libs/key1", noprealloc: true, oplogSize: 40, port: 31001, replSet: "rs1", rest: true, smallfiles: true } m31001| Tue Dec 17 11:52:13.059 [initandlisten] journal dir=/data/db/rs1-1/journal m31001| Tue Dec 17 11:52:13.060 [initandlisten] recover : no journal files present, no recovery needed m31001| Tue Dec 17 11:52:13.106 [FileAllocator] allocating new datafile /data/db/rs1-1/local.ns, filling with zeroes... m31001| Tue Dec 17 11:52:13.106 [FileAllocator] creating directory /data/db/rs1-1/_tmp m31001| Tue Dec 17 11:52:13.128 [FileAllocator] done allocating datafile /data/db/rs1-1/local.ns, size: 16MB, took 0.021 secs m31001| Tue Dec 17 11:52:13.154 [FileAllocator] allocating new datafile /data/db/rs1-1/local.0, filling with zeroes... m31001| Tue Dec 17 11:52:13.179 [FileAllocator] done allocating datafile /data/db/rs1-1/local.0, size: 16MB, took 0.024 secs m31001| Tue Dec 17 11:52:13.205 [websvr] admin web console waiting for connections on port 32001 m31001| Tue Dec 17 11:52:13.205 [initandlisten] waiting for connections on port 31001 m31001| Tue Dec 17 11:52:13.206 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31001| Tue Dec 17 11:52:13.206 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31001| Tue Dec 17 11:52:13.224 [initandlisten] connection accepted from 127.0.0.1:65259 #1 (1 connection now open) [ connection to specter.local:31000, connection to specter.local:31001 ] ReplSetTest n is : 2 ReplSetTest n: 2 ports: [ 31000, 31001, 31002 ] 31002 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : "jstests/libs/key1", "port" : 31002, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "rs1", "dbpath" : "$set-$node", "binVersion" : "248", "restart" : undefined, "pathOpts" : { "node" : 2, "set" : "rs1" } } ReplSetTest Starting.... Resetting db path '/data/db/rs1-2' 2013-12-17T11:52:13.239-0500 shell: started program /Users/vkarpov/qa/QA/QA-424/mongod-248 --oplogSize 40 --keyFile jstests/libs/key1 --port 31002 --noprealloc --smallfiles --rest --replSet rs1 --dbpath /data/db/rs1-2 2013-12-17T11:52:13.241-0500 warning: Failed to connect to 127.0.0.1, reason: errno:61 Connection refused m31002| note: noprealloc may hurt performance in many applications m31002| Tue Dec 17 11:52:13.271 [initandlisten] MongoDB starting : pid=2242 port=31002 dbpath=/data/db/rs1-2 64-bit host=specter.local m31002| Tue Dec 17 11:52:13.271 [initandlisten] m31002| Tue Dec 17 11:52:13.271 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000 m31002| Tue Dec 17 11:52:13.271 [initandlisten] db version v2.4.8 m31002| Tue Dec 17 11:52:13.271 [initandlisten] git version: a350fc38922fbda2cec8d5dd842237b904eafc14 m31002| Tue Dec 17 11:52:13.271 [initandlisten] build info: Darwin bs-osx-106-x86-64-2.10gen.cc 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:32:41 PDT 2011; root:xnu-1504.15.3~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49 m31002| Tue Dec 17 11:52:13.271 [initandlisten] allocator: system m31002| Tue Dec 17 11:52:13.271 [initandlisten] options: { dbpath: "/data/db/rs1-2", keyFile: "jstests/libs/key1", noprealloc: true, oplogSize: 40, port: 31002, replSet: "rs1", rest: true, smallfiles: true } m31002| Tue Dec 17 11:52:13.274 [initandlisten] journal dir=/data/db/rs1-2/journal m31002| Tue Dec 17 11:52:13.275 [initandlisten] recover : no journal files present, no recovery needed m31002| Tue Dec 17 11:52:13.328 [FileAllocator] allocating new datafile /data/db/rs1-2/local.ns, filling with zeroes... m31002| Tue Dec 17 11:52:13.328 [FileAllocator] creating directory /data/db/rs1-2/_tmp m31002| Tue Dec 17 11:52:13.365 [FileAllocator] done allocating datafile /data/db/rs1-2/local.ns, size: 16MB, took 0.036 secs m31002| Tue Dec 17 11:52:13.390 [FileAllocator] allocating new datafile /data/db/rs1-2/local.0, filling with zeroes... 2013-12-17T11:52:13.442-0500 warning: Failed to connect to 127.0.0.1, reason: errno:61 Connection refused m31002| Tue Dec 17 11:52:13.455 [FileAllocator] done allocating datafile /data/db/rs1-2/local.0, size: 16MB, took 0.065 secs m31002| Tue Dec 17 11:52:13.482 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 154ms m31002| Tue Dec 17 11:52:13.483 [websvr] admin web console waiting for connections on port 32002 m31002| Tue Dec 17 11:52:13.483 [initandlisten] waiting for connections on port 31002 m31002| Tue Dec 17 11:52:13.484 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31002| Tue Dec 17 11:52:13.484 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done m31002| Tue Dec 17 11:52:13.643 [initandlisten] connection accepted from 127.0.0.1:65262 #1 (1 connection now open) [ connection to specter.local:31000, connection to specter.local:31001, connection to specter.local:31002 ] { "replSetInitiate" : { "_id" : "rs1", "members" : [ { "_id" : 0, "host" : "specter.local:31000" }, { "_id" : 1, "host" : "specter.local:31001" }, { "_id" : 2, "host" : "specter.local:31002" } ] } } m31000| Tue Dec 17 11:52:13.647 [conn1] note: no users configured in admin.system.users, allowing localhost access m31000| Tue Dec 17 11:52:13.648 [conn1] replSet replSetInitiate admin command received from client m31000| Tue Dec 17 11:52:13.648 [conn1] replSet replSetInitiate config object parses ok, 3 members specified m31001| Tue Dec 17 11:52:13.796 [initandlisten] connection accepted from 10.4.101.171:65263 #2 (2 connections now open) m31001| Tue Dec 17 11:52:13.797 [conn2] authenticate db: local { authenticate: 1, nonce: "ba7288ffc0648983", user: "__system", key: "30dc431a552a04785414ba91ed2e3132" } m31002| Tue Dec 17 11:52:13.798 [initandlisten] connection accepted from 10.4.101.171:65264 #2 (2 connections now open) m31002| Tue Dec 17 11:52:13.799 [conn2] authenticate db: local { authenticate: 1, nonce: "acf2be53062c44b5", user: "__system", key: "bdd81a5fa08377eccfe02f50fcd1b79b" } m31000| Tue Dec 17 11:52:13.799 [conn1] replSet replSetInitiate all members seem up m31000| Tue Dec 17 11:52:13.800 [conn1] ****** m31000| Tue Dec 17 11:52:13.800 [conn1] creating replication oplog of size: 40MB... m31000| Tue Dec 17 11:52:13.800 [FileAllocator] allocating new datafile /data/db/rs1-0/local.1, filling with zeroes... m31000| Tue Dec 17 11:52:13.946 [FileAllocator] done allocating datafile /data/db/rs1-0/local.1, size: 64MB, took 0.145 secs m31000| Tue Dec 17 11:52:13.984 [conn1] ****** m31000| Tue Dec 17 11:52:13.984 [conn1] replSet info saving a newer config version to local.system.replset m31000| Tue Dec 17 11:52:14.001 [conn1] replSet saveConfigLocally done m31000| Tue Dec 17 11:52:14.001 [conn1] replSet replSetInitiate config now saved locally. Should come online in about a minute. m31000| Tue Dec 17 11:52:14.001 [conn1] command admin.$cmd command: { replSetInitiate: { _id: "rs1", members: [ { _id: 0.0, host: "specter.local:31000" }, { _id: 1.0, host: "specter.local:31001" }, { _id: 2.0, host: "specter.local:31002" } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:202259 reslen:112 353ms { "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1 } m31001| Tue Dec 17 11:52:14.013 [conn1] note: no users configured in admin.system.users, allowing localhost access m31002| Tue Dec 17 11:52:14.013 [conn1] note: no users configured in admin.system.users, allowing localhost access m31000| Tue Dec 17 11:52:22.924 [rsStart] replSet I am specter.local:31000 m31000| Tue Dec 17 11:52:22.925 [rsStart] replSet STARTUP2 m31001| Tue Dec 17 11:52:23.208 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31002| Tue Dec 17 11:52:23.486 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG) m31000| Tue Dec 17 11:52:23.926 [rsSync] replSet SECONDARY m31000| Tue Dec 17 11:52:24.926 [rsHealthPoll] replSet member specter.local:31002 is up m31000| Tue Dec 17 11:52:24.926 [rsHealthPoll] replSet member specter.local:31001 is up m31000| Tue Dec 17 11:52:24.926 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes m31000| Tue Dec 17 11:52:24.927 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes m31000| Tue Dec 17 11:52:30.930 [rsMgr] replSet not electing self, not all members up and we have been up less than 5 minutes m31001| Tue Dec 17 11:52:33.209 [rsStart] trying to contact specter.local:31000 m31000| Tue Dec 17 11:52:33.210 [initandlisten] connection accepted from 10.4.101.171:65269 #2 (2 connections now open) m31000| Tue Dec 17 11:52:33.210 [conn2] authenticate db: local { authenticate: 1, nonce: "c59bcf319b100291", user: "__system", key: "358914a3e691517e5f8959f15d361142" } m31001| Tue Dec 17 11:52:33.211 [rsStart] replSet I am specter.local:31001 m31001| Tue Dec 17 11:52:33.211 [rsStart] replSet got config version 1 from a remote, saving locally m31001| Tue Dec 17 11:52:33.211 [rsStart] replSet info saving a newer config version to local.system.replset m31001| Tue Dec 17 11:52:33.221 [rsStart] replSet saveConfigLocally done m31001| Tue Dec 17 11:52:33.221 [rsStart] replSet STARTUP2 m31001| Tue Dec 17 11:52:33.222 [rsSync] ****** m31001| Tue Dec 17 11:52:33.222 [rsSync] creating replication oplog of size: 40MB... m31001| Tue Dec 17 11:52:33.222 [FileAllocator] allocating new datafile /data/db/rs1-1/local.1, filling with zeroes... m31001| Tue Dec 17 11:52:33.374 [FileAllocator] done allocating datafile /data/db/rs1-1/local.1, size: 64MB, took 0.151 secs m31001| Tue Dec 17 11:52:33.411 [rsSync] ****** m31001| Tue Dec 17 11:52:33.411 [rsSync] replSet initial sync pending m31001| Tue Dec 17 11:52:33.411 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31002| Tue Dec 17 11:52:33.487 [rsStart] trying to contact specter.local:31000 m31000| Tue Dec 17 11:52:33.488 [initandlisten] connection accepted from 10.4.101.171:65270 #3 (3 connections now open) m31000| Tue Dec 17 11:52:33.488 [conn3] authenticate db: local { authenticate: 1, nonce: "ac1bc7464fa2e677", user: "__system", key: "b30a13d56978765aa5f280b679530512" } m31002| Tue Dec 17 11:52:33.489 [rsStart] replSet I am specter.local:31002 m31002| Tue Dec 17 11:52:33.489 [rsStart] replSet got config version 1 from a remote, saving locally m31002| Tue Dec 17 11:52:33.489 [rsStart] replSet info saving a newer config version to local.system.replset m31002| Tue Dec 17 11:52:33.494 [rsStart] replSet saveConfigLocally done m31002| Tue Dec 17 11:52:33.495 [rsStart] replSet STARTUP2 m31002| Tue Dec 17 11:52:33.495 [rsSync] ****** m31002| Tue Dec 17 11:52:33.495 [rsSync] creating replication oplog of size: 40MB... m31002| Tue Dec 17 11:52:33.495 [FileAllocator] allocating new datafile /data/db/rs1-2/local.1, filling with zeroes... m31002| Tue Dec 17 11:52:33.642 [FileAllocator] done allocating datafile /data/db/rs1-2/local.1, size: 64MB, took 0.146 secs m31002| Tue Dec 17 11:52:33.679 [rsSync] ****** m31002| Tue Dec 17 11:52:33.680 [rsSync] replSet initial sync pending m31002| Tue Dec 17 11:52:33.680 [rsSync] replSet initial sync need a member to be primary or secondary to do our initial sync m31000| Tue Dec 17 11:52:34.932 [rsHealthPoll] replset info specter.local:31002 thinks that we are down m31000| Tue Dec 17 11:52:34.932 [rsHealthPoll] replset info specter.local:31001 thinks that we are down m31000| Tue Dec 17 11:52:34.933 [rsHealthPoll] replSet member specter.local:31002 is now in state STARTUP2 m31000| Tue Dec 17 11:52:34.933 [rsHealthPoll] replSet member specter.local:31001 is now in state STARTUP2 m31000| Tue Dec 17 11:52:34.933 [rsMgr] not electing self, specter.local:31002 would veto with 'I don't think specter.local:31000 is electable' m31000| Tue Dec 17 11:52:34.933 [rsMgr] not electing self, specter.local:31002 would veto with 'I don't think specter.local:31000 is electable' m31001| Tue Dec 17 11:52:35.212 [rsHealthPoll] replSet member specter.local:31000 is up m31001| Tue Dec 17 11:52:35.212 [rsHealthPoll] replSet member specter.local:31000 is now in state SECONDARY m31002| Tue Dec 17 11:52:35.212 [initandlisten] connection accepted from 10.4.101.171:65271 #3 (3 connections now open) m31002| Tue Dec 17 11:52:35.212 [conn3] authenticate db: local { authenticate: 1, nonce: "6743cc4a2f495470", user: "__system", key: "1217fecb88257ed2bc452821a5dacc95" } m31001| Tue Dec 17 11:52:35.212 [rsHealthPoll] replset info specter.local:31002 thinks that we are down m31001| Tue Dec 17 11:52:35.212 [rsHealthPoll] replSet member specter.local:31002 is up m31001| Tue Dec 17 11:52:35.212 [rsHealthPoll] replSet member specter.local:31002 is now in state STARTUP2 m31002| Tue Dec 17 11:52:35.490 [rsHealthPoll] replSet member specter.local:31000 is up m31002| Tue Dec 17 11:52:35.490 [rsHealthPoll] replSet member specter.local:31000 is now in state SECONDARY m31001| Tue Dec 17 11:52:35.491 [initandlisten] connection accepted from 10.4.101.171:65272 #3 (3 connections now open) m31001| Tue Dec 17 11:52:35.491 [conn3] authenticate db: local { authenticate: 1, nonce: "6c49afc054d946e1", user: "__system", key: "182cd9521fc09241732b7f5ad8235060" } m31002| Tue Dec 17 11:52:35.491 [rsHealthPoll] replSet member specter.local:31001 is up m31002| Tue Dec 17 11:52:35.491 [rsHealthPoll] replSet member specter.local:31001 is now in state STARTUP2 m31001| Tue Dec 17 11:52:38.936 [conn2] end connection 10.4.101.171:65263 (2 connections now open) m31001| Tue Dec 17 11:52:38.937 [initandlisten] connection accepted from 10.4.101.171:65273 #4 (3 connections now open) m31001| Tue Dec 17 11:52:38.937 [conn4] authenticate db: local { authenticate: 1, nonce: "f5962e51c8a6409", user: "__system", key: "4e5e875839e49c3972ca0d78fee90dc1" } m31000| Tue Dec 17 11:52:40.938 [rsMgr] replSet info electSelf 0 m31002| Tue Dec 17 11:52:40.939 [conn2] replSet RECOVERING m31002| Tue Dec 17 11:52:40.939 [conn2] replSet info voting yea for specter.local:31000 (0) m31001| Tue Dec 17 11:52:40.939 [conn4] replSet RECOVERING m31001| Tue Dec 17 11:52:40.939 [conn4] replSet info voting yea for specter.local:31000 (0) m31000| Tue Dec 17 11:52:40.942 [rsMgr] replSet PRIMARY m31001| Tue Dec 17 11:52:41.215 [rsHealthPoll] replSet member specter.local:31000 is now in state PRIMARY m31001| Tue Dec 17 11:52:41.216 [rsHealthPoll] replSet member specter.local:31002 is now in state RECOVERING m31002| Tue Dec 17 11:52:41.495 [rsHealthPoll] replSet member specter.local:31001 is now in state RECOVERING m31002| Tue Dec 17 11:52:41.495 [rsHealthPoll] replSet member specter.local:31000 is now in state PRIMARY m31000| Tue Dec 17 11:52:42.940 [rsHealthPoll] replSet member specter.local:31002 is now in state RECOVERING m31000| Tue Dec 17 11:52:42.940 [rsHealthPoll] replSet member specter.local:31001 is now in state RECOVERING m31000| Tue Dec 17 11:52:49.221 [conn2] end connection 10.4.101.171:65269 (2 connections now open) m31000| Tue Dec 17 11:52:49.221 [initandlisten] connection accepted from 10.4.101.171:65274 #4 (3 connections now open) m31000| Tue Dec 17 11:52:49.221 [conn4] authenticate db: local { authenticate: 1, nonce: "4d03f39558040631", user: "__system", key: "1e9dfdd980617b6153aa22220cfdc7e6" } m31001| Tue Dec 17 11:52:49.413 [rsSync] replSet initial sync pending m31001| Tue Dec 17 11:52:49.413 [rsSync] replSet syncing to: specter.local:31000 m31000| Tue Dec 17 11:52:49.413 [initandlisten] connection accepted from 10.4.101.171:65275 #5 (4 connections now open) m31000| Tue Dec 17 11:52:49.414 [conn5] authenticate db: local { authenticate: 1, nonce: "70d049df5d4d95cd", user: "__system", key: "e49c0008af72acd8b0553b25f2fef335" } m31001| Tue Dec 17 11:52:49.415 [rsSync] build index local.me { _id: 1 } m31001| Tue Dec 17 11:52:49.416 [rsSync] build index done. scanned 0 total records. 0.001 secs m31001| Tue Dec 17 11:52:49.417 [rsSync] build index local.replset.minvalid { _id: 1 } m31001| Tue Dec 17 11:52:49.418 [rsSync] build index done. scanned 0 total records. 0 secs m31001| Tue Dec 17 11:52:49.418 [rsSync] replSet initial sync drop all databases m31001| Tue Dec 17 11:52:49.418 [rsSync] dropAllDatabasesExceptLocal 1 m31001| Tue Dec 17 11:52:49.418 [rsSync] replSet initial sync clone all databases m31001| Tue Dec 17 11:52:49.418 [rsSync] replSet initial sync cloning db: admin m31000| Tue Dec 17 11:52:49.419 [initandlisten] connection accepted from 10.4.101.171:65276 #6 (5 connections now open) m31000| Tue Dec 17 11:52:49.419 [conn6] authenticate db: local { authenticate: 1, nonce: "642088a3389b3e2e", user: "__system", key: "34a804afab00321977b107bbc6cd9b5e" } m31001| Tue Dec 17 11:52:49.419 [rsSync] replSet initial sync data copy, starting syncup m31001| Tue Dec 17 11:52:49.419 [rsSync] oplog sync 1 of 3 m31001| Tue Dec 17 11:52:49.419 [rsSync] oplog sync 2 of 3 m31001| Tue Dec 17 11:52:49.420 [rsSync] replSet initial sync building indexes m31001| Tue Dec 17 11:52:49.420 [rsSync] replSet initial sync cloning indexes for : admin m31001| Tue Dec 17 11:52:49.420 [rsSync] oplog sync 3 of 3 m31000| Tue Dec 17 11:52:49.420 [conn6] end connection 10.4.101.171:65276 (4 connections now open) m31001| Tue Dec 17 11:52:49.420 [rsSync] replSet initial sync finishing up m31001| Tue Dec 17 11:52:49.435 [rsSync] replSet set minValid=52b0813d:1 m31001| Tue Dec 17 11:52:49.442 [rsSync] replSet initial sync done m31000| Tue Dec 17 11:52:49.442 [conn5] end connection 10.4.101.171:65275 (3 connections now open) m31000| Tue Dec 17 11:52:49.500 [conn3] end connection 10.4.101.171:65270 (2 connections now open) m31000| Tue Dec 17 11:52:49.501 [initandlisten] connection accepted from 10.4.101.171:65277 #7 (3 connections now open) m31000| Tue Dec 17 11:52:49.501 [conn7] authenticate db: local { authenticate: 1, nonce: "68a58bd0d78759f5", user: "__system", key: "3f8824ac010fb08b5ceeb10739f48ccf" } m31002| Tue Dec 17 11:52:49.681 [rsSync] replSet initial sync pending m31002| Tue Dec 17 11:52:49.681 [rsSync] replSet syncing to: specter.local:31000 m31000| Tue Dec 17 11:52:49.682 [initandlisten] connection accepted from 10.4.101.171:65278 #8 (4 connections now open) m31000| Tue Dec 17 11:52:49.682 [conn8] authenticate db: local { authenticate: 1, nonce: "b15504df99b8f68e", user: "__system", key: "837b33f1b22c97148f5845ebba828219" } m31002| Tue Dec 17 11:52:49.684 [rsSync] build index local.me { _id: 1 } m31002| Tue Dec 17 11:52:49.685 [rsSync] build index done. scanned 0 total records. 0.001 secs m31002| Tue Dec 17 11:52:49.687 [rsSync] build index local.replset.minvalid { _id: 1 } m31002| Tue Dec 17 11:52:49.687 [rsSync] build index done. scanned 0 total records. 0 secs m31002| Tue Dec 17 11:52:49.688 [rsSync] replSet initial sync drop all databases m31002| Tue Dec 17 11:52:49.688 [rsSync] dropAllDatabasesExceptLocal 1 m31002| Tue Dec 17 11:52:49.688 [rsSync] replSet initial sync clone all databases m31002| Tue Dec 17 11:52:49.688 [rsSync] replSet initial sync cloning db: admin m31000| Tue Dec 17 11:52:49.689 [initandlisten] connection accepted from 10.4.101.171:65279 #9 (5 connections now open) m31000| Tue Dec 17 11:52:49.689 [conn9] authenticate db: local { authenticate: 1, nonce: "acfdb24b9b69ecac", user: "__system", key: "27c1959b313efaa18f00bfe1398435ac" } m31002| Tue Dec 17 11:52:49.689 [rsSync] replSet initial sync data copy, starting syncup m31002| Tue Dec 17 11:52:49.689 [rsSync] oplog sync 1 of 3 m31002| Tue Dec 17 11:52:49.689 [rsSync] oplog sync 2 of 3 m31002| Tue Dec 17 11:52:49.690 [rsSync] replSet initial sync building indexes m31002| Tue Dec 17 11:52:49.690 [rsSync] replSet initial sync cloning indexes for : admin m31002| Tue Dec 17 11:52:49.690 [rsSync] oplog sync 3 of 3 m31000| Tue Dec 17 11:52:49.690 [conn9] end connection 10.4.101.171:65279 (4 connections now open) m31002| Tue Dec 17 11:52:49.690 [rsSync] replSet initial sync finishing up m31002| Tue Dec 17 11:52:49.698 [rsSync] replSet set minValid=52b0813d:1 m31002| Tue Dec 17 11:52:49.706 [rsSync] replSet initial sync done m31000| Tue Dec 17 11:52:49.706 [conn8] end connection 10.4.101.171:65278 (3 connections now open) m31001| Tue Dec 17 11:52:50.237 [rsBackgroundSync] replSet syncing to: specter.local:31000 m31000| Tue Dec 17 11:52:50.238 [initandlisten] connection accepted from 10.4.101.171:65280 #10 (4 connections now open) m31000| Tue Dec 17 11:52:50.238 [conn10] authenticate db: local { authenticate: 1, nonce: "314aa85c1bfe2765", user: "__system", key: "cd2712ec4f6ce2303559f1b13aed5d9a" } m31001| Tue Dec 17 11:52:50.443 [rsSyncNotifier] replset setting oplog notifier to specter.local:31000 m31000| Tue Dec 17 11:52:50.444 [initandlisten] connection accepted from 10.4.101.171:65281 #11 (5 connections now open) m31000| Tue Dec 17 11:52:50.444 [conn11] authenticate db: local { authenticate: 1, nonce: "71a46ae1d7d49dce", user: "__system", key: "a6df54f4940e82c5d8eacc0e673ff8ca" } m31002| Tue Dec 17 11:52:50.509 [rsBackgroundSync] replSet syncing to: specter.local:31000 m31000| Tue Dec 17 11:52:50.510 [initandlisten] connection accepted from 10.4.101.171:65282 #12 (6 connections now open) m31000| Tue Dec 17 11:52:50.510 [conn12] authenticate db: local { authenticate: 1, nonce: "772befd538e27510", user: "__system", key: "813dadc196cd1d5d225950c953582cd7" } m31002| Tue Dec 17 11:52:50.707 [rsSyncNotifier] replset setting oplog notifier to specter.local:31000 m31000| Tue Dec 17 11:52:50.708 [initandlisten] connection accepted from 10.4.101.171:65283 #13 (7 connections now open) m31000| Tue Dec 17 11:52:50.708 [conn13] authenticate db: local { authenticate: 1, nonce: "7d2ed9c0001ef813", user: "__system", key: "f502e174544f0e69cf1f4c9bf7faa33b" } m31001| Tue Dec 17 11:52:51.446 [rsSync] replSet SECONDARY m31000| Tue Dec 17 11:52:51.448 [slaveTracking] build index local.slaves { _id: 1 } m31000| Tue Dec 17 11:52:51.449 [slaveTracking] build index done. scanned 0 total records. 0 secs m31002| Tue Dec 17 11:52:51.502 [rsHealthPoll] replSet member specter.local:31001 is now in state SECONDARY m31002| Tue Dec 17 11:52:51.710 [rsSync] replSet SECONDARY WARNING: The 'addUser' shell helper is DEPRECATED. Please use 'createUser' instead WARNING: connected to a server which does not support write commands. m31000| Tue Dec 17 11:52:51.739 [FileAllocator] allocating new datafile /data/db/rs1-0/admin.ns, filling with zeroes... Successfully added user: { "user" : "admin", "roles" : [ "userAdminAnyDatabase", "readWriteAnyDatabase", "dbAdminAnyDatabase", "clusterAdmin" ], "_id" : ObjectId("52b08163c642bd49b3fa0065") } m31000| Tue Dec 17 11:52:51.761 [FileAllocator] done allocating datafile /data/db/rs1-0/admin.ns, size: 16MB, took 0.021 secs m31000| Tue Dec 17 11:52:51.786 [FileAllocator] allocating new datafile /data/db/rs1-0/admin.0, filling with zeroes... m31000| Tue Dec 17 11:52:51.807 [FileAllocator] done allocating datafile /data/db/rs1-0/admin.0, size: 16MB, took 0.02 secs m31000| Tue Dec 17 11:52:51.834 [conn1] build index admin.system.users { _id: 1 } m31000| Tue Dec 17 11:52:51.835 [conn1] build index done. scanned 0 total records. 0 secs m31000| Tue Dec 17 11:52:51.835 [conn1] build index admin.system.users { user: 1, userSource: 1 } m31000| Tue Dec 17 11:52:51.836 [conn1] build index done. scanned 0 total records. 0 secs m31001| Tue Dec 17 11:52:51.837 [FileAllocator] allocating new datafile /data/db/rs1-1/admin.ns, filling with zeroes... m31002| Tue Dec 17 11:52:51.837 [FileAllocator] allocating new datafile /data/db/rs1-2/admin.ns, filling with zeroes... m31002| Tue Dec 17 11:52:51.865 [FileAllocator] done allocating datafile /data/db/rs1-2/admin.ns, size: 16MB, took 0.027 secs m31001| Tue Dec 17 11:52:51.884 [FileAllocator] done allocating datafile /data/db/rs1-1/admin.ns, size: 16MB, took 0.046 secs m31002| Tue Dec 17 11:52:51.913 [FileAllocator] allocating new datafile /data/db/rs1-2/admin.0, filling with zeroes... m31001| Tue Dec 17 11:52:51.923 [FileAllocator] allocating new datafile /data/db/rs1-1/admin.0, filling with zeroes... m31002| Tue Dec 17 11:52:51.936 [FileAllocator] done allocating datafile /data/db/rs1-2/admin.0, size: 16MB, took 0.022 secs m31001| Tue Dec 17 11:52:51.968 [FileAllocator] done allocating datafile /data/db/rs1-1/admin.0, size: 16MB, took 0.045 secs m31002| Tue Dec 17 11:52:51.985 [repl writer worker 1] build index admin.system.users { _id: 1 } m31002| Tue Dec 17 11:52:51.986 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31002| Tue Dec 17 11:52:51.986 [repl writer worker 1] build index admin.system.users { user: 1, userSource: 1 } m31002| Tue Dec 17 11:52:51.987 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31000| Tue Dec 17 11:52:51.998 [conn1] command admin.$cmd command: { getlasterror: 1.0, w: "majority", wtimeout: 30000.0 } ntoreturn:1 keyUpdates:0 reslen:204 162ms -------- Upgrading set to 26 m31001| Tue Dec 17 11:52:52.000 [repl writer worker 1] build index admin.system.users { _id: 1 } m31001| Tue Dec 17 11:52:52.001 [repl writer worker 1] build index done. scanned 0 total records. 0 secs m31001| Tue Dec 17 11:52:52.001 [repl writer worker 1] build index admin.system.users { user: 1, userSource: 1 } m31001| Tue Dec 17 11:52:52.001 [repl writer worker 1] build index done. scanned 0 total records. 0 secs { "n0" : { "binVersion" : "26", "keyFile" : "jstests/libs/key1", "clusterAuthMode" : "keyFile" }, "n1" : { "binVersion" : "26", "keyFile" : "jstests/libs/key1", "clusterAuthMode" : "keyFile" }, "n2" : { "binVersion" : "26", "keyFile" : "jstests/libs/key1", "clusterAuthMode" : "keyFile" } } ReplSetTest n: 1 ports: [ 31000, 31001, 31002 ] 31001 number ReplSetTest stop *** Shutting down mongod in port 31001 *** m31001| Tue Dec 17 11:52:52.003 [signalProcessingThread] got signal 15 (Terminated: 15), will terminate after current cmd ends m31001| Tue Dec 17 11:52:52.003 [signalProcessingThread] now exiting m31001| Tue Dec 17 11:52:52.004 dbexit: m31001| Tue Dec 17 11:52:52.004 [signalProcessingThread] shutdown: going to close listening sockets... m31001| Tue Dec 17 11:52:52.004 [signalProcessingThread] closing listening socket: 16 m31001| Tue Dec 17 11:52:52.004 [signalProcessingThread] closing listening socket: 17 m31001| Tue Dec 17 11:52:52.004 [signalProcessingThread] closing listening socket: 18 m31001| Tue Dec 17 11:52:52.004 [signalProcessingThread] removing socket file: /tmp/mongodb-31001.sock m31001| Tue Dec 17 11:52:52.004 [signalProcessingThread] shutdown: going to flush diaglog... m31001| Tue Dec 17 11:52:52.004 [signalProcessingThread] shutdown: going to close sockets... m31001| Tue Dec 17 11:52:52.004 [signalProcessingThread] shutdown: waiting for fs preallocator... m31001| Tue Dec 17 11:52:52.004 [signalProcessingThread] shutdown: lock for final commit... m31001| Tue Dec 17 11:52:52.004 [signalProcessingThread] shutdown: final commit... m31001| Tue Dec 17 11:52:52.004 [rsBackgroundSync] Socket recv() errno:9 Bad file descriptor 10.4.101.171:31000 m31001| Tue Dec 17 11:52:52.004 [rsBackgroundSync] SocketException: remote: 10.4.101.171:31000 error: 9001 socket exception [RECV_ERROR] server [10.4.101.171:31000] m31001| Tue Dec 17 11:52:52.004 [conn1] end connection 127.0.0.1:65259 (2 connections now open) m31001| Tue Dec 17 11:52:52.004 [conn4] end connection 10.4.101.171:65273 (2 connections now open) m31002| Tue Dec 17 11:52:52.004 [conn3] end connection 10.4.101.171:65271 (2 connections now open) m31001| Tue Dec 17 11:52:52.004 [rsSyncNotifier] Socket recv() errno:9 Bad file descriptor 10.4.101.171:31000 m31001| Tue Dec 17 11:52:52.004 [rsSyncNotifier] SocketException: remote: 10.4.101.171:31000 error: 9001 socket exception [RECV_ERROR] server [10.4.101.171:31000] m31001| Tue Dec 17 11:52:52.004 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: specter.local:31000 m31000| Tue Dec 17 11:52:52.005 [conn4] end connection 10.4.101.171:65274 (6 connections now open) m31001| Tue Dec 17 11:52:52.005 [conn3] end connection 10.4.101.171:65272 (2 connections now open) m31001| Tue Dec 17 11:52:52.005 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: specter.local:31000 m31001| Tue Dec 17 11:52:52.023 [signalProcessingThread] shutdown: closing all files... m31001| Tue Dec 17 11:52:52.030 [signalProcessingThread] closeAllFiles() finished m31001| Tue Dec 17 11:52:52.030 [signalProcessingThread] journalCleanup... m31001| Tue Dec 17 11:52:52.030 [signalProcessingThread] removeJournalFiles m31001| Tue Dec 17 11:52:52.030 [signalProcessingThread] shutdown: removing fs lock... m31001| Tue Dec 17 11:52:52.030 dbexit: really exiting now m31000| Tue Dec 17 11:52:52.946 [rsHealthPoll] DBClientCursor::init call() failed m31000| Tue Dec 17 11:52:52.946 [rsHealthPoll] replset info specter.local:31001 heartbeat failed, retrying m31002| Tue Dec 17 11:52:52.946 [conn2] end connection 10.4.101.171:65264 (1 connection now open) m31002| Tue Dec 17 11:52:52.947 [initandlisten] connection accepted from 10.4.101.171:65285 #4 (2 connections now open) m31000| Tue Dec 17 11:52:52.947 [rsHealthPoll] replSet info specter.local:31001 is down (or slow to respond): m31000| Tue Dec 17 11:52:52.947 [rsHealthPoll] replSet member specter.local:31001 is now in state DOWN m31002| Tue Dec 17 11:52:52.947 [conn4] authenticate db: local { authenticate: 1, nonce: "907010cb2735a2e0", user: "__system", key: "3e52944ec90b9e2004fbc1132dce32d4" } m31000| Tue Dec 17 11:52:52.948 [rsHealthPoll] replSet member specter.local:31002 is now in state SECONDARY 2013-12-17T11:52:53.004-0500 shell: stopped mongo program on port 31001 ReplSetTest n is : connection to specter.local:31001 ReplSetTest n: 1 ports: [ 31000, 31001, 31002 ] 31001 number { "useHostName" : true, "oplogSize" : 40, "keyFile" : "jstests/libs/key1", "port" : 31001, "noprealloc" : "", "smallfiles" : "", "rest" : "", "replSet" : "rs1", "dbpath" : "$set-$node", "binVersion" : "26", "clusterAuthMode" : "keyFile", "restart" : true, "pathOpts" : { "node" : 1, "set" : "rs1" } } ReplSetTest (Re)Starting.... 2013-12-17T11:52:53.007-0500 shell: started program /Users/vkarpov/qa/QA/QA-424/mongod-26 --oplogSize 40 --keyFile jstests/libs/key1 --port 31001 --noprealloc --smallfiles --rest --replSet rs1 --dbpath /data/db/rs1-1 --clusterAuthMode keyFile 2013-12-17T11:52:53.007-0500 warning: Failed to connect to 127.0.0.1, reason: errno:61 Connection refused m31001| 2013-12-17T11:52:53.037-0500 ** WARNING: --rest is specified without --httpinterface, m31001| 2013-12-17T11:52:53.038-0500 ** enabling http interface m31001| note: noprealloc may hurt performance in many applications m31001| 2013-12-17T11:52:53.042-0500 [initandlisten] MongoDB starting : pid=2243 port=31001 dbpath=/data/db/rs1-1 64-bit host=specter.local m31001| 2013-12-17T11:52:53.042-0500 [initandlisten] m31001| 2013-12-17T11:52:53.042-0500 [initandlisten] ** NOTE: This is a development version (2.5.4) of MongoDB. m31001| 2013-12-17T11:52:53.042-0500 [initandlisten] ** Not recommended for production. m31001| 2013-12-17T11:52:53.042-0500 [initandlisten] m31001| 2013-12-17T11:52:53.042-0500 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000 m31001| 2013-12-17T11:52:53.042-0500 [initandlisten] m31001| 2013-12-17T11:52:53.042-0500 [initandlisten] db version v2.5.4 m31001| 2013-12-17T11:52:53.042-0500 [initandlisten] git version: ffd52e5f46cf2ba74ba931c78da62d4a7f480d8e m31001| 2013-12-17T11:52:53.042-0500 [initandlisten] build info: Darwin bs-osx-108-x86-64-2.10gen.cc 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan 6 22:37:10 PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49 m31001| 2013-12-17T11:52:53.042-0500 [initandlisten] allocator: system m31001| 2013-12-17T11:52:53.042-0500 [initandlisten] options: { clusterAuthMode: "keyFile", dbpath: "/data/db/rs1-1", keyFile: "jstests/libs/key1", noprealloc: true, oplogSize: 40, port: 31001, replSet: "rs1", rest: true, smallfiles: true } m31001| 2013-12-17T11:52:53.049-0500 [initandlisten] journal dir=/data/db/rs1-1/journal m31001| 2013-12-17T11:52:53.049-0500 [initandlisten] recover : no journal files present, no recovery needed m31001| 2013-12-17T11:52:53.108-0500 [initandlisten] waiting for connections on port 31001 m31001| 2013-12-17T11:52:53.109-0500 [websvr] admin web console waiting for connections on port 32001 m31001| 2013-12-17T11:52:53.110-0500 [rsStart] replSet I am specter.local:31001 m31001| 2013-12-17T11:52:53.120-0500 [rsStart] replSet STARTUP2 m31001| 2013-12-17T11:52:53.120-0500 [rsSync] replSet SECONDARY m31001| 2013-12-17T11:52:53.208-0500 [initandlisten] connection accepted from 127.0.0.1:65287 #1 (1 connection now open) [ connection to specter.local:31000, connection to specter.local:31001, connection to specter.local:31002 ] ReplSetTest waitForIndicator state on connection to specter.local:31001 [ 1, 2, 7 ] ReplSetTest waitForIndicator from node connection to specter.local:31001 m31000| Tue Dec 17 11:52:53.212 [conn1] command denied: { replSetGetStatus: 1.0 } ReplSetTest waitForIndicator Initial status ( timeout : 30000 ) : { "ok" : 0, "errmsg" : "unauthorized" } m31000| Tue Dec 17 11:52:53.414 [conn1] command denied: { replSetGetStatus: 1.0 } m31002| Tue Dec 17 11:52:53.503 [rsHealthPoll] DBClientCursor::init call() failed m31002| Tue Dec 17 11:52:53.503 [rsHealthPoll] replset info specter.local:31001 heartbeat failed, retrying m31001| 2013-12-17T11:52:53.503-0500 [initandlisten] connection accepted from 10.4.101.171:65288 #2 (2 connections now open) m31001| 2013-12-17T11:52:53.504-0500 [conn2] authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" } m31002| Tue Dec 17 11:52:53.504 [rsHealthPoll] replset info specter.local:31001 thinks that we are down m31000| Tue Dec 17 11:52:53.617 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:53.820 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:54.023 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:54.226 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:54.429 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:54.632 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:54.835 [conn1] command denied: { replSetGetStatus: 1.0 } m31001| 2013-12-17T11:52:54.949-0500 [initandlisten] connection accepted from 10.4.101.171:65289 #3 (3 connections now open) m31001| 2013-12-17T11:52:54.949-0500 [conn3] authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" } m31000| Tue Dec 17 11:52:54.950 [rsHealthPoll] replset info specter.local:31001 thinks that we are down m31000| Tue Dec 17 11:52:54.950 [rsHealthPoll] replSet member specter.local:31001 is up m31000| Tue Dec 17 11:52:54.950 [rsHealthPoll] replSet member specter.local:31001 is now in state SECONDARY m31000| Tue Dec 17 11:52:55.038 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:55.112 [initandlisten] connection accepted from 10.4.101.171:65291 #14 (7 connections now open) m31002| Tue Dec 17 11:52:55.112 [initandlisten] connection accepted from 10.4.101.171:65292 #5 (3 connections now open) m31000| Tue Dec 17 11:52:55.113 [conn14] authenticate db: local { authenticate: 1, nonce: "89cb7cf0e00b349b", user: "__system", key: "e86baab1db55669a3c07436c14570e66" } m31002| Tue Dec 17 11:52:55.113 [conn5] authenticate db: local { authenticate: 1, nonce: "8b17459d4e6d6f0d", user: "__system", key: "c4bb188860a17ede1223f46a58951eda" } m31001| 2013-12-17T11:52:55.113-0500 [rsHealthPoll] replSet member specter.local:31000 is up m31001| 2013-12-17T11:52:55.113-0500 [rsHealthPoll] replSet member specter.local:31000 is now in state PRIMARY m31001| 2013-12-17T11:52:55.113-0500 [rsHealthPoll] replSet member specter.local:31002 is up m31001| 2013-12-17T11:52:55.113-0500 [rsHealthPoll] replSet member specter.local:31002 is now in state SECONDARY m31000| Tue Dec 17 11:52:55.241 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:55.443 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:55.646 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:55.848 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:56.051 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:56.253 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:56.456 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:56.660 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:56.853 [conn10] end connection 10.4.101.171:65280 (6 connections now open) m31000| Tue Dec 17 11:52:56.862 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:57.020 [conn11] end connection 10.4.101.171:65281 (5 connections now open) m31000| Tue Dec 17 11:52:57.064 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:57.267 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:57.469 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:57.672 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:57.875 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:58.078 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:58.281 [conn1] command denied: { replSetGetStatus: 1.0 } { "ok" : 0, "errmsg" : "unauthorized" } m31000| Tue Dec 17 11:52:58.485 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:58.687 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:58.890 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:59.093 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:59.295 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:59.499 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:59.702 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:52:59.906 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:00.110 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:00.314 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:00.517 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:00.720 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:00.923 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:01.126 [conn1] command denied: { replSetGetStatus: 1.0 } m31001| 2013-12-17T11:53:01.128-0500 [rsBackgroundSync] replSet syncing to: specter.local:31000 m31000| Tue Dec 17 11:53:01.128 [initandlisten] connection accepted from 10.4.101.171:65293 #15 (6 connections now open) m31000| Tue Dec 17 11:53:01.129 [conn15] authenticate db: local { authenticate: 1, nonce: "e87f4a121f984a89", user: "__system", key: "018012c94d52f7b127d1ffce3f0a0ec9" } m31000| Tue Dec 17 11:53:01.129 [initandlisten] connection accepted from 10.4.101.171:65294 #16 (7 connections now open) m31000| Tue Dec 17 11:53:01.129 [conn16] authenticate db: local { authenticate: 1, nonce: "4cb514274fc246d3", user: "__system", key: "1ded5dc456acdd0acd8b03b88e35ca1a" } m31001| 2013-12-17T11:53:01.130-0500 [rsBackgroundSync] upstream updater is unsupported on this version m31000| Tue Dec 17 11:53:01.130 [conn16] end connection 10.4.101.171:65294 (6 connections now open) m31000| Tue Dec 17 11:53:01.130 [initandlisten] connection accepted from 10.4.101.171:65295 #17 (7 connections now open) m31000| Tue Dec 17 11:53:01.130 [conn17] authenticate db: local { authenticate: 1, nonce: "3ee196361c7fb1df", user: "__system", key: "763800fe34ff6a3591705325d8cda336" } m31001| 2013-12-17T11:53:01.134-0500 [rsSyncNotifier] replset setting oplog notifier to specter.local:31000 m31000| Tue Dec 17 11:53:01.134 [conn17] end connection 10.4.101.171:65295 (6 connections now open) m31000| Tue Dec 17 11:53:01.135 [initandlisten] connection accepted from 10.4.101.171:65296 #18 (7 connections now open) m31000| Tue Dec 17 11:53:01.135 [conn18] authenticate db: local { authenticate: 1, nonce: "fb0bf1ba718c892b", user: "__system", key: "3aae784ea51941d5bffd3bc4c5d747af" } m31001| 2013-12-17T11:53:01.136-0500 [rsSyncNotifier] upstream updater is unsupported on this version m31000| Tue Dec 17 11:53:01.136 [conn18] end connection 10.4.101.171:65296 (6 connections now open) m31000| Tue Dec 17 11:53:01.136 [initandlisten] connection accepted from 10.4.101.171:65297 #19 (7 connections now open) m31000| Tue Dec 17 11:53:01.136 [conn19] authenticate db: local { authenticate: 1, nonce: "4de44f4c65f9eabd", user: "__system", key: "038ed89aea399e0a2730ec98ebcd97b9" } m31000| Tue Dec 17 11:53:01.329 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:01.532 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:01.735 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:01.954 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:02.157 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:02.359 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:02.562 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:02.764 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:02.967 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:03.170 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:03.373 [conn1] command denied: { replSetGetStatus: 1.0 } { "ok" : 0, "errmsg" : "unauthorized" } m31001| 2013-12-17T11:53:03.510-0500 [conn2] end connection 10.4.101.171:65288 (2 connections now open) m31001| 2013-12-17T11:53:03.511-0500 [initandlisten] connection accepted from 10.4.101.171:65298 #4 (3 connections now open) m31001| 2013-12-17T11:53:03.511-0500 [conn4] authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" } m31000| Tue Dec 17 11:53:03.577 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:03.780 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:03.982 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:04.185 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:04.388 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:04.590 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:04.792 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:04.995 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:05.198 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:05.400 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:05.603 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:05.806 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:06.008 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:06.211 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:06.414 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:06.616 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:06.819 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:07.021 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:07.224 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:07.426 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:07.629 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:07.832 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:08.034 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:08.237 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:08.440 [conn1] command denied: { replSetGetStatus: 1.0 } { "ok" : 0, "errmsg" : "unauthorized" } m31000| Tue Dec 17 11:53:08.643 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:08.845 [conn1] command denied: { replSetGetStatus: 1.0 } m31001| 2013-12-17T11:53:08.959-0500 [conn3] end connection 10.4.101.171:65289 (2 connections now open) m31001| 2013-12-17T11:53:08.960-0500 [initandlisten] connection accepted from 10.4.101.171:65299 #5 (3 connections now open) m31001| 2013-12-17T11:53:08.960-0500 [conn5] authenticate db: local { authenticate: 1, nonce: "xxx", user: "__system", key: "xxx" } m31000| Tue Dec 17 11:53:09.048 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:09.122 [conn14] end connection 10.4.101.171:65291 (6 connections now open) m31000| Tue Dec 17 11:53:09.123 [initandlisten] connection accepted from 10.4.101.171:65300 #20 (7 connections now open) m31000| Tue Dec 17 11:53:09.123 [conn20] authenticate db: local { authenticate: 1, nonce: "6937a3c792389a50", user: "__system", key: "5e112dd63407835020831407b419fa79" } m31000| Tue Dec 17 11:53:09.251 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:09.454 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:09.657 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:09.860 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:10.063 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:10.266 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:10.469 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:10.671 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:10.873 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:11.075 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:11.279 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:11.482 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:11.685 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:11.887 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:12.090 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:12.293 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:12.495 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:12.698 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:12.900 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:13.103 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:13.306 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:13.509 [conn1] command denied: { replSetGetStatus: 1.0 } { "ok" : 0, "errmsg" : "unauthorized" } m31000| Tue Dec 17 11:53:13.712 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:13.915 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:14.118 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:14.321 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:14.524 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:14.727 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:14.930 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:15.132 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:15.335 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:15.537 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:15.740 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:15.943 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:16.145 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:16.349 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:16.551 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:16.755 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:16.958 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:17.161 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:17.364 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:17.567 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:17.770 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:17.973 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:18.176 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:18.378 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:18.581 [conn1] command denied: { replSetGetStatus: 1.0 } { "ok" : 0, "errmsg" : "unauthorized" } m31000| Tue Dec 17 11:53:18.784 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:18.987 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:19.190 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:19.393 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:19.518 [conn7] end connection 10.4.101.171:65277 (6 connections now open) m31000| Tue Dec 17 11:53:19.519 [initandlisten] connection accepted from 10.4.101.171:65301 #21 (7 connections now open) m31000| Tue Dec 17 11:53:19.519 [conn21] authenticate db: local { authenticate: 1, nonce: "32164190dac07f0c", user: "__system", key: "3655652dfb1fb4273ed49ee032525039" } m31000| Tue Dec 17 11:53:19.596 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:19.798 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:20.001 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:20.203 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:20.406 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:20.609 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:20.812 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:21.015 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:21.217 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:21.420 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:21.623 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:21.825 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:22.028 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:22.230 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:22.433 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:22.636 [conn1] command denied: { replSetGetStatus: 1.0 } m31000| Tue Dec 17 11:53:22.839 [conn1] command denied: { replSetGetStatus: 1.0 } m31002| Tue Dec 17 11:53:22.968 [conn4] end connection 10.4.101.171:65285 (2 connections now open) m31002| Tue Dec 17 11:53:22.969 [initandlisten] connection accepted from 10.4.101.171:65302 #6 (3 connections now open) m31002| Tue Dec 17 11:53:22.969 [conn6] authenticate db: local { authenticate: 1, nonce: "699169e977342c71", user: "__system", key: "a078842dcae3a2783e6496f667bed1e0" } m31000| Tue Dec 17 11:53:23.042 [conn1] command denied: { replSetGetStatus: 1.0 } m31002| Tue Dec 17 11:53:23.132 [conn5] end connection 10.4.101.171:65292 (2 connections now open) m31002| Tue Dec 17 11:53:23.132 [initandlisten] connection accepted from 10.4.101.171:65303 #7 (3 connections now open) m31002| Tue Dec 17 11:53:23.132 [conn7] authenticate db: local { authenticate: 1, nonce: "e1b6ae7fa7c6f405", user: "__system", key: "b50a02dcbe55c2beebe58f235ebb4e23" } m31000| Tue Dec 17 11:53:23.245 [conn1] command denied: { replSetGetStatus: 1.0 } assert.soon failed: function () { try { status = self.status(); } catch ( ex ) { print( "ReplSetTest waitForIndicator could not get status: " + tojson( ex ) ); return false; } var printStatus = false if( lastTime == null || ( currTime = new Date().getTime() ) - (1000 * 5) > lastTime ){ if( lastTime == null ) print( "ReplSetTest waitForIndicator Initial status ( timeout : " + timeout + " ) :" ) printjson( status ) lastTime = new Date().getTime() printStatus = true } if (typeof status.members == 'undefined') { return false; } for( var i = 0; i < status.members.length; i++ ){ if( printStatus ) print( "Status for : " + status.members[i].name + ", checking " + node.host + "/" + node.name ) if( status.members[i].name == node.host || status.members[i].name == node.name ){ for( var j = 0; j < states.length; j++ ){ if( printStatus ) print( "Status " + " : " + status.members[i][ind] + " target state : " + states[j] ) if( status.members[i][ind] == states[j] ) return true; } } } return false }, msg:waiting for state indicator state for 30000ms Error: Printing Stack Trace at printStackTrace (src/mongo/shell/utils.js:38:15) at doassert (src/mongo/shell/assert.js:6:5) at Function.assert.soon (src/mongo/shell/assert.js:174:60) at ReplSetTest.waitForIndicator (src/mongo/shell/replsettest.js:921:12) at ReplSetTest.waitForState (src/mongo/shell/replsettest.js:878:10) at ReplSetTest.upgradeNode (x509_repl_upgrade.js:58:14) at ReplSetTest.upgradeSet (x509_repl_upgrade.js:41:14) at repl_upgrade_normal.js:31:11 at repl_upgrade_normal.js:38:3 2013-12-17T11:53:23.250-0500 assert.soon failed: function () { try { status = self.status(); } catch ( ex ) { print( "ReplSetTest waitForIndicator could not get status: " + tojson( ex ) ); return false; } var printStatus = false if( lastTime == null || ( currTime = new Date().getTime() ) - (1000 * 5) > lastTime ){ if( lastTime == null ) print( "ReplSetTest waitForIndicator Initial status ( timeout : " + timeout + " ) :" ) printjson( status ) lastTime = new Date().getTime() printStatus = true } if (typeof status.members == 'undefined') { return false; } for( var i = 0; i < status.members.length; i++ ){ if( printStatus ) print( "Status for : " + status.members[i].name + ", checking " + node.host + "/" + node.name ) if( status.members[i].name == node.host || status.members[i].name == node.name ){ for( var j = 0; j < states.length; j++ ){ if( printStatus ) print( "Status " + " : " + status.members[i][ind] + " target state : " + states[j] ) if( status.members[i][ind] == states[j] ) return true; } } } return false }, msg:waiting for state indicator state for 30000ms at src/mongo/shell/assert.js:7 failed to load: repl_upgrade_normal.js m31000| Tue Dec 17 11:53:23.250 [signalProcessingThread] got signal 15 (Terminated: 15), will terminate after current cmd ends m31000| Tue Dec 17 11:53:23.250 [signalProcessingThread] now exiting m31000| Tue Dec 17 11:53:23.250 dbexit: m31000| Tue Dec 17 11:53:23.250 [signalProcessingThread] shutdown: going to close listening sockets... m31000| Tue Dec 17 11:53:23.250 [signalProcessingThread] closing listening socket: 13 m31000| Tue Dec 17 11:53:23.250 [signalProcessingThread] closing listening socket: 14 m31000| Tue Dec 17 11:53:23.250 [signalProcessingThread] closing listening socket: 15 m31000| Tue Dec 17 11:53:23.250 [signalProcessingThread] removing socket file: /tmp/mongodb-31000.sock m31000| Tue Dec 17 11:53:23.250 [signalProcessingThread] shutdown: going to flush diaglog... m31000| Tue Dec 17 11:53:23.250 [signalProcessingThread] shutdown: going to close sockets... m31000| Tue Dec 17 11:53:23.251 [signalProcessingThread] shutdown: waiting for fs preallocator... m31000| Tue Dec 17 11:53:23.251 [signalProcessingThread] shutdown: lock for final commit... m31000| Tue Dec 17 11:53:23.251 [signalProcessingThread] shutdown: final commit... m31000| Tue Dec 17 11:53:23.251 [conn1] end connection 127.0.0.1:65257 (6 connections now open) m31000| Tue Dec 17 11:53:23.251 [conn13] end connection 10.4.101.171:65283 (6 connections now open) m31000| Tue Dec 17 11:53:23.251 [conn21] end connection 10.4.101.171:65301 (6 connections now open) m31000| Tue Dec 17 11:53:23.251 [conn19] end connection 10.4.101.171:65297 (6 connections now open) m31002| Tue Dec 17 11:53:23.251 [conn6] end connection 10.4.101.171:65302 (2 connections now open) m31002| Tue Dec 17 11:53:23.251 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: specter.local:31000 m31002| Tue Dec 17 11:53:23.251 [rsBackgroundSync] replSet syncing to: specter.local:31000 m31001| 2013-12-17T11:53:23.251-0500 [conn5] end connection 10.4.101.171:65299 (2 connections now open) m31000| Tue Dec 17 11:53:23.252 [conn20] end connection 10.4.101.171:65300 (6 connections now open) m31002| Tue Dec 17 11:53:23.252 [rsBackgroundSync] repl: couldn't connect to server specter.local:31000 m31001| 2013-12-17T11:53:23.259-0500 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: specter.local:31000 m31001| 2013-12-17T11:53:23.259-0500 [rsBackgroundSync] replSet syncing to: specter.local:31000 m31001| 2013-12-17T11:53:23.259-0500 [rsBackgroundSync] warning: Failed to connect to 10.4.101.171, reason: errno:61 Connection refused m31001| 2013-12-17T11:53:23.259-0500 [rsBackgroundSync] repl: couldn't connect to server specter.local:31000 m31000| Tue Dec 17 11:53:23.264 [signalProcessingThread] shutdown: closing all files... m31000| Tue Dec 17 11:53:23.264 [signalProcessingThread] closeAllFiles() finished m31000| Tue Dec 17 11:53:23.264 [signalProcessingThread] journalCleanup... m31000| Tue Dec 17 11:53:23.264 [signalProcessingThread] removeJournalFiles m31000| Tue Dec 17 11:53:23.264 [signalProcessingThread] shutdown: removing fs lock... m31000| Tue Dec 17 11:53:23.264 dbexit: really exiting now m31002| Tue Dec 17 11:53:23.522 [rsHealthPoll] DBClientCursor::init call() failed m31002| Tue Dec 17 11:53:23.522 [rsHealthPoll] replset info specter.local:31000 heartbeat failed, retrying m31002| Tue Dec 17 11:53:23.523 [rsHealthPoll] replSet info specter.local:31000 is down (or slow to respond): m31002| Tue Dec 17 11:53:23.523 [rsHealthPoll] replSet member specter.local:31000 is now in state DOWN m31002| Tue Dec 17 11:53:23.523 [rsMgr] not electing self, specter.local:31001 would veto with 'specter.local:31002 is trying to elect itself but specter.local:31000 is already primary and more up-to-date' m31001| 2013-12-17T11:53:24.251-0500 [signalProcessingThread] got signal 15 (Terminated: 15), will terminate after current cmd ends m31001| 2013-12-17T11:53:24.252-0500 [signalProcessingThread] now exiting m31001| dbexit: 2013-12-17T11:53:24.252-0500 [signalProcessingThread] shutdown: going to close listening sockets... m31001| 2013-12-17T11:53:24.252-0500 [signalProcessingThread] closing listening socket: 22 m31001| 2013-12-17T11:53:24.252-0500 [signalProcessingThread] closing listening socket: 23 m31001| 2013-12-17T11:53:24.252-0500 [signalProcessingThread] closing listening socket: 26 m31001| 2013-12-17T11:53:24.252-0500 [signalProcessingThread] removing socket file: /tmp/mongodb-31001.sock m31001| 2013-12-17T11:53:24.252-0500 [signalProcessingThread] shutdown: going to flush diaglog... m31001| 2013-12-17T11:53:24.252-0500 [signalProcessingThread] shutdown: going to close sockets... m31001| 2013-12-17T11:53:24.252-0500 [signalProcessingThread] shutdown: waiting for fs preallocator... m31001| 2013-12-17T11:53:24.252-0500 [signalProcessingThread] shutdown: lock for final commit... m31001| 2013-12-17T11:53:24.252-0500 [signalProcessingThread] shutdown: final commit... m31001| 2013-12-17T11:53:24.253-0500 [conn4] end connection 10.4.101.171:65298 (1 connection now open) m31002| Tue Dec 17 11:53:24.253 [conn7] end connection 10.4.101.171:65303 (1 connection now open) m31001| 2013-12-17T11:53:24.253-0500 [conn1] end connection 127.0.0.1:65287 (1 connection now open) m31001| 2013-12-17T11:53:24.272-0500 [signalProcessingThread] shutdown: closing all files... m31001| 2013-12-17T11:53:24.272-0500 [signalProcessingThread] closeAllFiles() finished m31001| 2013-12-17T11:53:24.272-0500 [signalProcessingThread] journalCleanup... m31001| 2013-12-17T11:53:24.272-0500 [signalProcessingThread] removeJournalFiles m31001| 2013-12-17T11:53:24.273-0500 [signalProcessingThread] shutdown: removing fs lock... m31002| Tue Dec 17 11:53:25.253 [signalProcessingThread] got signal 15 (Terminated: 15), will terminate after current cmd ends m31002| Tue Dec 17 11:53:25.253 [signalProcessingThread] now exiting m31002| Tue Dec 17 11:53:25.253 dbexit: m31002| Tue Dec 17 11:53:25.253 [signalProcessingThread] shutdown: going to close listening sockets... m31002| Tue Dec 17 11:53:25.253 [signalProcessingThread] closing listening socket: 19 m31002| Tue Dec 17 11:53:25.253 [signalProcessingThread] closing listening socket: 20 m31002| Tue Dec 17 11:53:25.253 [signalProcessingThread] closing listening socket: 21 m31002| Tue Dec 17 11:53:25.253 [signalProcessingThread] removing socket file: /tmp/mongodb-31002.sock m31002| Tue Dec 17 11:53:25.253 [signalProcessingThread] shutdown: going to flush diaglog... m31002| Tue Dec 17 11:53:25.253 [signalProcessingThread] shutdown: going to close sockets... m31002| Tue Dec 17 11:53:25.253 [signalProcessingThread] shutdown: waiting for fs preallocator... m31002| Tue Dec 17 11:53:25.253 [signalProcessingThread] shutdown: lock for final commit... m31002| Tue Dec 17 11:53:25.253 [signalProcessingThread] shutdown: final commit... m31002| Tue Dec 17 11:53:25.254 [conn1] end connection 127.0.0.1:65262 (0 connections now open) m31002| Tue Dec 17 11:53:25.265 [signalProcessingThread] shutdown: closing all files... m31002| Tue Dec 17 11:53:25.266 [signalProcessingThread] closeAllFiles() finished m31002| Tue Dec 17 11:53:25.266 [signalProcessingThread] journalCleanup... m31002| Tue Dec 17 11:53:25.266 [signalProcessingThread] removeJournalFiles m31002| Tue Dec 17 11:53:25.266 [signalProcessingThread] shutdown: removing fs lock... m31002| Tue Dec 17 11:53:25.266 dbexit: really exiting now