2016-02-01T05:34:32.829+0000 I CONTROL [main] ***** SERVER RESTARTED ***** 2016-02-01T05:34:32.836+0000 I CONTROL [initandlisten] MongoDB starting : pid=12232 port=27017 dbpath=/data 64-bit host=ip-10-0-12-86 2016-02-01T05:34:32.836+0000 I CONTROL [initandlisten] db version v3.2.1 2016-02-01T05:34:32.836+0000 I CONTROL [initandlisten] git version: a14d55980c2cdc565d4704a7e3ad37e4e535c1b2 2016-02-01T05:34:32.836+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.0-fips 29 Mar 2010 2016-02-01T05:34:32.836+0000 I CONTROL [initandlisten] allocator: tcmalloc 2016-02-01T05:34:32.836+0000 I CONTROL [initandlisten] modules: none 2016-02-01T05:34:32.836+0000 I CONTROL [initandlisten] build environment: 2016-02-01T05:34:32.836+0000 I CONTROL [initandlisten] distmod: amazon 2016-02-01T05:34:32.836+0000 I CONTROL [initandlisten] distarch: x86_64 2016-02-01T05:34:32.836+0000 I CONTROL [initandlisten] target_arch: x86_64 2016-02-01T05:34:32.836+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017 }, operationProfiling: { slowOpThresholdMs: 5000 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, replication: { replSetName: "rsAppsee2" }, storage: { dbPath: "/data", engine: "mmapv1", indexBuildRetry: false }, systemLog: { component: { replication: { verbosity: 5 } }, destination: "file", logAppend: true, logRotate: "rename", path: "/data/mongod.log" } } 2016-02-01T05:34:32.869+0000 I JOURNAL [initandlisten] journal dir=/data/journal 2016-02-01T05:34:32.869+0000 I JOURNAL [initandlisten] recover : no journal files present, no recovery needed 2016-02-01T05:34:33.139+0000 I JOURNAL [initandlisten] preallocateIsFaster=true 3.56 2016-02-01T05:34:33.262+0000 I JOURNAL [durability] Durability thread started 2016-02-01T05:34:33.263+0000 I JOURNAL [journal writer] Journal writer thread started 2016-02-01T05:34:33.287+0000 D REPL [initandlisten] returning minvalid: (term: -1, timestamp: Jan 1 00:00:00:0)({ ts: Timestamp 0|0, t: -1 }) -> (term: -1, timestamp: Jan 21 09:03:36:430)({ ts: Timestamp 1453367016000|1072, t: -1 }) 2016-02-01T05:34:33.287+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/diagnostic.data' 2016-02-01T05:34:33.287+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker 2016-02-01T05:34:33.288+0000 I NETWORK [initandlisten] waiting for connections on port 27017 2016-02-01T05:34:33.289+0000 W NETWORK [HostnameCanonicalizationWorker] Failed to obtain address information for hostname ip-10-0-12-86: Name or service not known 2016-02-01T05:34:33.331+0000 D REPL [ReplicationExecutor] Updated term in topology coordinator to 0 due to new config 2016-02-01T05:34:33.331+0000 I REPL [ReplicationExecutor] New replica set config in use: { _id: "rsAppsee2", version: 77032, protocolVersion: 1, members: [ { _id: 0, host: "in.db2m1.mydomain.com:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 2.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "in.db2m2.mydomain.com:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "in.db2arb.mydomain.com:27017", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 3, host: "10.0.12.86:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } } 2016-02-01T05:34:33.331+0000 I REPL [ReplicationExecutor] This node is 10.0.12.86:27017 in the config 2016-02-01T05:34:33.331+0000 I REPL [ReplicationExecutor] transition to STARTUP2 2016-02-01T05:34:33.331+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.mydomain.com:27017 at 2016-02-01T05:34:33.331Z 2016-02-01T05:34:33.331+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m2.mydomain.com:27017 at 2016-02-01T05:34:33.331Z 2016-02-01T05:34:33.331+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.mydomain.com:27017 at 2016-02-01T05:34:33.331Z 2016-02-01T05:34:33.331+0000 D REPL [ReplicationExecutor] slaveinfo lastupdate is: 2016-02-01T05:34:33.331+0000 2016-02-01T05:34:33.331+0000 D REPL [ReplicationExecutor] slaveinfo lastupdate is: 2016-02-01T05:34:33.331+0000 2016-02-01T05:34:33.331+0000 D REPL [ReplicationExecutor] slaveinfo lastupdate is: 2016-02-01T05:34:33.331+0000 2016-02-01T05:34:33.331+0000 D REPL [ReplicationExecutor] earliest member 0 date: 2016-02-01T05:34:33.331+0000 2016-02-01T05:34:33.331+0000 D REPL [ReplicationExecutor] scheduling next check at 2016-02-01T05:34:43.331+0000 2016-02-01T05:34:33.331+0000 D REPL [ReplicationExecutor] Current term is now -1 2016-02-01T05:34:33.331+0000 I REPL [ReplicationExecutor] Starting replication applier threads 2016-02-01T05:34:33.331+0000 D REPL [rsSync] return initial flag value of 0 2016-02-01T05:34:33.331+0000 I REPL [ReplicationExecutor] transition to RECOVERING 2016-02-01T05:34:33.331+0000 D REPL [rsBackgroundSync] bgsync fetch queue set to: (term: -1, timestamp: Jan 21 09:03:36:430) -6286199844409637543 2016-02-01T05:34:33.332+0000 D REPL [rsSync] returning minvalid: (term: -1, timestamp: Jan 1 00:00:00:0)({ ts: Timestamp 0|0, t: -1 }) -> (term: -1, timestamp: Jan 21 09:03:36:430)({ ts: Timestamp 1453367016000|1072, t: -1 }) 2016-02-01T05:34:33.333+0000 I REPL [ReplicationExecutor] transition to SECONDARY 2016-02-01T05:34:33.333+0000 D REPL [ReplicationExecutor] Scheduling election timeout callback at 2016-02-01T05:34:43.443+0000 2016-02-01T05:34:33.333+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to in.db2m1.mydomain.com:27017 2016-02-01T05:34:33.333+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to in.db2m2.mydomain.com:27017 2016-02-01T05:34:33.333+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to in.db2arb.mydomain.com:27017 2016-02-01T05:34:33.333+0000 D REPL [ReplicationExecutor] Canceling election timeout callback at 2016-02-01T05:34:43.443+0000 2016-02-01T05:34:33.333+0000 D REPL [ReplicationExecutor] Scheduling election timeout callback at 2016-02-01T05:34:43.609+0000 2016-02-01T05:34:33.333+0000 D REPL [ReplicationExecutor] Canceling election timeout callback at 2016-02-01T05:34:43.609+0000 2016-02-01T05:34:33.333+0000 D REPL [ReplicationExecutor] Scheduling election timeout callback at 2016-02-01T05:34:44.556+0000 2016-02-01T05:34:33.333+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:0, msg: 2016-02-01T05:34:33.333+0000 I REPL [ReplicationExecutor] Member in.db2m1.mydomain.com:27017 is now in state PRIMARY 2016-02-01T05:34:33.333+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.mydomain.com:27017 at 2016-02-01T05:34:38.333Z 2016-02-01T05:34:33.333+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:1, msg: 2016-02-01T05:34:33.334+0000 I REPL [ReplicationExecutor] Member in.db2m2.mydomain.com:27017 is now in state SECONDARY 2016-02-01T05:34:33.334+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m2.mydomain.com:27017 at 2016-02-01T05:34:38.333Z 2016-02-01T05:34:33.334+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:2, msg: 2016-02-01T05:34:33.334+0000 I REPL [ReplicationExecutor] Member in.db2arb.mydomain.com:27017 is now in state ARBITER 2016-02-01T05:34:33.334+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.mydomain.com:27017 at 2016-02-01T05:34:38.334Z 2016-02-01T05:34:33.479+0000 I NETWORK [initandlisten] connection accepted from 10.0.209.231:40336 #1 (1 connection now open) 2016-02-01T05:34:33.529+0000 I NETWORK [initandlisten] connection accepted from 10.0.97.218:56189 #2 (2 connections now open) 2016-02-01T05:34:34.387+0000 I NETWORK [initandlisten] connection accepted from 10.0.86.213:33661 #3 (3 connections now open) 2016-02-01T05:34:34.388+0000 I NETWORK [initandlisten] connection accepted from 10.0.208.168:52254 #4 (4 connections now open) 2016-02-01T05:34:34.389+0000 I NETWORK [initandlisten] connection accepted from 10.0.51.223:56013 #5 (5 connections now open) 2016-02-01T05:34:36.382+0000 I NETWORK [initandlisten] connection accepted from 10.0.43.87:48038 #6 (6 connections now open) 2016-02-01T05:34:36.763+0000 I NETWORK [initandlisten] connection accepted from 10.0.208.168:52256 #7 (7 connections now open) 2016-02-01T05:34:38.334+0000 D REPL [ReplicationExecutor] Canceling election timeout callback at 2016-02-01T05:34:44.556+0000 2016-02-01T05:34:38.334+0000 D REPL [ReplicationExecutor] Scheduling election timeout callback at 2016-02-01T05:34:49.066+0000 2016-02-01T05:34:38.334+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:0, msg: 2016-02-01T05:34:38.334+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.mydomain.com:27017 at 2016-02-01T05:34:43.334Z 2016-02-01T05:34:38.334+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:1, msg: 2016-02-01T05:34:38.334+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m2.mydomain.com:27017 at 2016-02-01T05:34:43.334Z 2016-02-01T05:34:38.335+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:2, msg: 2016-02-01T05:34:38.335+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.mydomain.com:27017 at 2016-02-01T05:34:43.335Z 2016-02-01T05:34:39.269+0000 I NETWORK [initandlisten] connection accepted from 10.0.51.223:56015 #8 (8 connections now open) 2016-02-01T05:34:39.332+0000 I REPL [ReplicationExecutor] syncing from: in.db2m2.mydomain.com:27017 2016-02-01T05:34:39.335+0000 D REPL [SyncSourceFeedback] resetting connection in sync source feedback 2016-02-01T05:34:39.335+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m2.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-01T05:34:39.335+0000 I REPL [SyncSourceFeedback] setting syncSourceFeedback to in.db2m2.mydomain.com:27017 2016-02-01T05:34:39.336+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454304875000|17, t: 4 }, memberId: 1, cfgver: 77032 }, { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454256910000|1, t: 0 }, memberId: 2, cfgver: 77032 }, { _id: ObjectId('56ae3208451e6bf36cf9be0b'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 77032 } ] } 2016-02-01T05:34:39.336+0000 I ASIO [NetworkInterfaceASIO-BGSync-0] Successfully connected to in.db2m2.mydomain.com:27017 2016-02-01T05:34:43.270+0000 I NETWORK [initandlisten] connection accepted from 10.0.158.156:50929 #9 (9 connections now open) 2016-02-01T05:34:43.331+0000 D REPL [ReplicationExecutor] slaveinfo lastupdate is: 2016-02-01T05:34:42.392+0000 2016-02-01T05:34:43.331+0000 D REPL [ReplicationExecutor] slaveinfo lastupdate is: 2016-02-01T05:34:42.394+0000 2016-02-01T05:34:43.331+0000 D REPL [ReplicationExecutor] slaveinfo lastupdate is: 2016-02-01T05:34:39.389+0000 2016-02-01T05:34:43.331+0000 D REPL [ReplicationExecutor] earliest member 2 date: 2016-02-01T05:34:39.389+0000 2016-02-01T05:34:43.332+0000 D REPL [ReplicationExecutor] scheduling next check at 2016-02-01T05:34:49.389+0000 2016-02-01T05:34:43.334+0000 D REPL [ReplicationExecutor] Canceling election timeout callback at 2016-02-01T05:34:49.066+0000 2016-02-01T05:34:43.334+0000 D REPL [ReplicationExecutor] Scheduling election timeout callback at 2016-02-01T05:34:54.220+0000 2016-02-01T05:34:43.334+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:0, msg: 2016-02-01T05:34:43.334+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.mydomain.com:27017 at 2016-02-01T05:34:45.334Z 2016-02-01T05:34:43.335+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:1, msg: 2016-02-01T05:34:43.335+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m2.mydomain.com:27017 at 2016-02-01T05:34:45.335Z 2016-02-01T05:34:43.335+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:2, msg: 2016-02-01T05:34:43.335+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.mydomain.com:27017 at 2016-02-01T05:34:45.335Z 2016-02-01T05:34:44.336+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454304880000|11, t: 4 }, memberId: 1, cfgver: 77032 }, { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454256910000|1, t: 0 }, memberId: 2, cfgver: 77032 }, { _id: ObjectId('56ae3208451e6bf36cf9be0b'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 77032 } ] } 2016-02-01T05:34:44.402+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m2.mydomain.com:27017 2016-02-01T05:34:44.402+0000 I REPL [ReplicationExecutor] syncing from: in.db2m2.mydomain.com:27017 2016-02-01T05:34:44.404+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454304880000|11, t: 4 }, memberId: 1, cfgver: 77032 }, { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454256910000|1, t: 0 }, memberId: 2, cfgver: 77032 }, { _id: ObjectId('56ae3208451e6bf36cf9be0b'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 77032 } ] } 2016-02-01T05:34:44.404+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m2.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-01T05:34:45.335+0000 D REPL [ReplicationExecutor] Canceling election timeout callback at 2016-02-01T05:34:54.220+0000 2016-02-01T05:34:45.336+0000 D REPL [ReplicationExecutor] Scheduling election timeout callback at 2016-02-01T05:34:56.503+0000 2016-02-01T05:34:45.336+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:0, msg: 2016-02-01T05:34:45.336+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.mydomain.com:27017 at 2016-02-01T05:34:47.335Z 2016-02-01T05:34:45.336+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:1, msg: 2016-02-01T05:34:45.336+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m2.mydomain.com:27017 at 2016-02-01T05:34:47.336Z 2016-02-01T05:34:45.336+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:2, msg: 2016-02-01T05:34:45.336+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.mydomain.com:27017 at 2016-02-01T05:34:47.336Z 2016-02-01T05:34:47.336+0000 D REPL [ReplicationExecutor] Canceling election timeout callback at 2016-02-01T05:34:56.503+0000 2016-02-01T05:34:47.336+0000 D REPL [ReplicationExecutor] Scheduling election timeout callback at 2016-02-01T05:34:57.767+0000 2016-02-01T05:34:47.336+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:0, msg: 2016-02-01T05:34:47.337+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.mydomain.com:27017 at 2016-02-01T05:34:49.336Z 2016-02-01T05:34:47.337+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:1, msg: 2016-02-01T05:34:47.337+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m2.mydomain.com:27017 at 2016-02-01T05:34:49.337Z 2016-02-01T05:34:47.337+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:2, msg: 2016-02-01T05:34:47.337+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.mydomain.com:27017 at 2016-02-01T05:34:49.337Z 2016-02-01T05:34:49.337+0000 D REPL [ReplicationExecutor] Canceling election timeout callback at 2016-02-01T05:34:57.767+0000 2016-02-01T05:34:49.337+0000 D REPL [ReplicationExecutor] Scheduling election timeout callback at 2016-02-01T05:35:00.099+0000 2016-02-01T05:34:49.337+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:0, msg: 2016-02-01T05:34:49.337+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.mydomain.com:27017 at 2016-02-01T05:34:51.337Z 2016-02-01T05:34:49.338+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:1, msg: 2016-02-01T05:34:49.338+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m2.mydomain.com:27017 at 2016-02-01T05:34:51.338Z 2016-02-01T05:34:49.338+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:2, msg: 2016-02-01T05:34:49.338+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.mydomain.com:27017 at 2016-02-01T05:34:51.338Z 2016-02-01T05:34:49.389+0000 D REPL [ReplicationExecutor] slaveinfo lastupdate is: 2016-02-01T05:34:48.395+0000 2016-02-01T05:34:49.389+0000 D REPL [ReplicationExecutor] slaveinfo lastupdate is: 2016-02-01T05:34:49.338+0000 2016-02-01T05:34:49.389+0000 D REPL [ReplicationExecutor] slaveinfo lastupdate is: 2016-02-01T05:34:44.390+0000 2016-02-01T05:34:49.389+0000 D REPL [ReplicationExecutor] earliest member 2 date: 2016-02-01T05:34:44.390+0000 2016-02-01T05:34:49.389+0000 D REPL [ReplicationExecutor] scheduling next check at 2016-02-01T05:34:54.390+0000 2016-02-01T05:34:49.405+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454304888000|16, t: 4 }, memberId: 1, cfgver: 77032 }, { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454256910000|1, t: 0 }, memberId: 2, cfgver: 77032 }, { _id: ObjectId('56ae3208451e6bf36cf9be0b'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 77032 } ] } 2016-02-01T05:34:49.471+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m2.mydomain.com:27017 2016-02-01T05:34:49.471+0000 I REPL [ReplicationExecutor] syncing from: in.db2m2.mydomain.com:27017 2016-02-01T05:34:49.473+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454304888000|16, t: 4 }, memberId: 1, cfgver: 77032 }, { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454256910000|1, t: 0 }, memberId: 2, cfgver: 77032 }, { _id: ObjectId('56ae3208451e6bf36cf9be0b'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 77032 } ] } 2016-02-01T05:34:49.473+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m2.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-01T05:34:51.339+0000 D REPL [ReplicationExecutor] Canceling election timeout callback at 2016-02-01T05:35:00.099+0000 2016-02-01T05:34:51.339+0000 D REPL [ReplicationExecutor] Scheduling election timeout callback at 2016-02-01T05:35:01.972+0000 2016-02-01T05:34:51.339+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:0, msg: 2016-02-01T05:34:51.339+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.mydomain.com:27017 at 2016-02-01T05:34:53.338Z 2016-02-01T05:34:51.339+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:2, msg: 2016-02-01T05:34:51.339+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.mydomain.com:27017 at 2016-02-01T05:34:53.339Z 2016-02-01T05:34:51.339+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:1, msg: 2016-02-01T05:34:51.339+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m2.mydomain.com:27017 at 2016-02-01T05:34:53.339Z 2016-02-01T05:34:53.339+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:2, msg: 2016-02-01T05:34:53.339+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.mydomain.com:27017 at 2016-02-01T05:34:55.339Z 2016-02-01T05:34:53.339+0000 D REPL [ReplicationExecutor] Canceling election timeout callback at 2016-02-01T05:35:01.972+0000 2016-02-01T05:34:53.339+0000 D REPL [ReplicationExecutor] Scheduling election timeout callback at 2016-02-01T05:35:04.560+0000 2016-02-01T05:34:53.339+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:0, msg: 2016-02-01T05:34:53.339+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.mydomain.com:27017 at 2016-02-01T05:34:55.339Z 2016-02-01T05:34:53.339+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:1, msg: 2016-02-01T05:34:53.339+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m2.mydomain.com:27017 at 2016-02-01T05:34:55.339Z 2016-02-01T05:34:54.391+0000 D REPL [ReplicationExecutor] slaveinfo lastupdate is: 2016-02-01T05:34:52.397+0000 2016-02-01T05:34:54.391+0000 D REPL [ReplicationExecutor] slaveinfo lastupdate is: 2016-02-01T05:34:52.399+0000 2016-02-01T05:34:54.391+0000 D REPL [ReplicationExecutor] slaveinfo lastupdate is: 2016-02-01T05:34:49.391+0000 2016-02-01T05:34:54.391+0000 D REPL [ReplicationExecutor] earliest member 2 date: 2016-02-01T05:34:49.391+0000 2016-02-01T05:34:54.391+0000 D REPL [ReplicationExecutor] scheduling next check at 2016-02-01T05:34:59.391+0000 2016-02-01T05:34:54.474+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454304888000|16, t: 4 }, memberId: 1, cfgver: 77032 }, { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454256910000|1, t: 0 }, memberId: 2, cfgver: 77032 }, { _id: ObjectId('56ae3208451e6bf36cf9be0b'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 77032 } ] } 2016-02-01T05:34:54.539+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m2.mydomain.com:27017 2016-02-01T05:34:54.539+0000 I REPL [ReplicationExecutor] syncing from: in.db2m2.mydomain.com:27017 2016-02-01T05:34:54.541+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454304888000|16, t: 4 }, memberId: 1, cfgver: 77032 }, { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454256910000|1, t: 0 }, memberId: 2, cfgver: 77032 }, { _id: ObjectId('56ae3208451e6bf36cf9be0b'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 77032 } ] } 2016-02-01T05:34:54.541+0000 D REPL [rsBackgroundSync] scheduling fetcher to read remote oplog on in.db2m2.mydomain.com:27017 starting at filter: { ts: { $gte: Timestamp 1453367016000|1072 } } 2016-02-01T05:34:55.340+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:2, msg: 2016-02-01T05:34:55.340+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.mydomain.com:27017 at 2016-02-01T05:34:57.340Z 2016-02-01T05:34:55.340+0000 D REPL [ReplicationExecutor] Canceling election timeout callback at 2016-02-01T05:35:04.560+0000 2016-02-01T05:34:55.341+0000 D REPL [ReplicationExecutor] Scheduling election timeout callback at 2016-02-01T05:35:06.322+0000 2016-02-01T05:34:55.341+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:0, msg: 2016-02-01T05:34:55.341+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.mydomain.com:27017 at 2016-02-01T05:34:57.340Z 2016-02-01T05:34:55.341+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:1, msg: 2016-02-01T05:34:55.341+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m2.mydomain.com:27017 at 2016-02-01T05:34:57.341Z 2016-02-01T05:34:57.341+0000 D REPL [ReplicationExecutor] Canceling election timeout callback at 2016-02-01T05:35:06.322+0000 2016-02-01T05:34:57.341+0000 D REPL [ReplicationExecutor] Scheduling election timeout callback at 2016-02-01T05:35:08.328+0000 2016-02-01T05:34:57.341+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:0, msg: 2016-02-01T05:34:57.341+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.mydomain.com:27017 at 2016-02-01T05:34:59.341Z 2016-02-01T05:34:57.341+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:1, msg: 2016-02-01T05:34:57.342+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m2.mydomain.com:27017 at 2016-02-01T05:34:59.341Z 2016-02-01T05:34:57.342+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:2, msg: 2016-02-01T05:34:57.342+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.mydomain.com:27017 at 2016-02-01T05:34:59.342Z 2016-02-01T05:34:59.342+0000 D REPL [ReplicationExecutor] Canceling election timeout callback at 2016-02-01T05:35:08.328+0000 2016-02-01T05:34:59.342+0000 D REPL [ReplicationExecutor] Scheduling election timeout callback at 2016-02-01T05:35:10.530+0000 2016-02-01T05:34:59.342+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:0, msg: 2016-02-01T05:34:59.342+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m1.mydomain.com:27017 at 2016-02-01T05:35:01.342Z 2016-02-01T05:34:59.342+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:1, msg: 2016-02-01T05:34:59.342+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2m2.mydomain.com:27017 at 2016-02-01T05:35:01.342Z 2016-02-01T05:34:59.343+0000 D REPL [ReplicationExecutor] setUpValues: heartbeat response good for member _id:2, msg: 2016-02-01T05:34:59.343+0000 D REPL [ReplicationExecutor] Scheduling heartbeat to in.db2arb.mydomain.com:27017 at 2016-02-01T05:35:01.343Z 2016-02-01T05:34:59.391+0000 D REPL [ReplicationExecutor] slaveinfo lastupdate is: 2016-02-01T05:34:58.399+0000 2016-02-01T05:34:59.391+0000 D REPL [ReplicationExecutor] slaveinfo lastupdate is: 2016-02-01T05:34:58.402+0000 2016-02-01T05:34:59.391+0000 D REPL [ReplicationExecutor] slaveinfo lastupdate is: 2016-02-01T05:34:54.392+0000 2016-02-01T05:34:59.391+0000 D REPL [ReplicationExecutor] earliest member 2 date: 2016-02-01T05:34:54.392+0000 2016-02-01T05:34:59.391+0000 D REPL [ReplicationExecutor] scheduling next check at 2016-02-01T05:35:04.392+0000 2016-02-01T05:34:59.541+0000 D REPL [SyncSourceFeedback] Sending slave oplog progress to upstream updater: { replSetUpdatePosition: 1, optimes: [ { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454304895000|7, t: 4 }, memberId: 1, cfgver: 77032 }, { _id: ObjectId('000000000000000000000000'), optime: { ts: Timestamp 1454256910000|1, t: 0 }, memberId: 2, cfgver: 77032 }, { _id: ObjectId('56ae3208451e6bf36cf9be0b'), optime: { ts: Timestamp 1453367016000|1072, t: -1 }, memberId: 3, cfgver: 77032 } ] } 2016-02-01T05:34:59.607+0000 D REPL [rsBackgroundSync] fetcher stopped reading remote oplog on in.db2m2.mydomain.com:27017