2017-07-11T16:09:59.219Z I CONTROL  [initandlisten] MongoDB starting : pid=7 port=27017 dbpath=/opt/collaborne/data/persistency 64-bit host=mongodb-arbiter-1674084307-7zp3d
2017-07-11T16:09:59.219Z I CONTROL  [initandlisten] db version v3.4.6
2017-07-11T16:09:59.219Z I CONTROL  [initandlisten] git version: c55eb86ef46ee7aede3b1e2a5d184a7df4bfb5b5
2017-07-11T16:09:59.219Z I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.0-fips 29 Mar 2010
2017-07-11T16:09:59.219Z I CONTROL  [initandlisten] allocator: tcmalloc
2017-07-11T16:09:59.219Z I CONTROL  [initandlisten] modules: none
2017-07-11T16:09:59.219Z I CONTROL  [initandlisten] build environment:
2017-07-11T16:09:59.219Z I CONTROL  [initandlisten]     distmod: amazon
2017-07-11T16:09:59.219Z I CONTROL  [initandlisten]     distarch: x86_64
2017-07-11T16:09:59.219Z I CONTROL  [initandlisten]     target_arch: x86_64
2017-07-11T16:09:59.219Z I CONTROL  [initandlisten] options: { config: "/opt/collaborne/etc/mongod.conf", net: { port: 27017, unixDomainSocket: { enabled: false } }, operationProfiling: { mode: "slowOp", slowOpThresholdMs: 100 }, processManagement: { pidFilePath: "/opt/collaborne/var/persistency-server.pid" }, replication: { oplogSizeMB: 2048, replSet: "rs0/mongodb" }, storage: { dbPath: "/opt/collaborne/data/persistency", directoryPerDB: true, engine: "wiredTiger" }, systemLog: { logAppend: true, timeStampFormat: "iso8601-utc" } }
2017-07-11T16:09:59.241Z I STORAGE  [initandlisten] 
2017-07-11T16:09:59.241Z I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-07-11T16:09:59.241Z I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-07-11T16:09:59.241Z I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1463M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-07-11T16:09:59.776Z I CONTROL  [initandlisten] 
2017-07-11T16:09:59.776Z I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-07-11T16:09:59.776Z I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2017-07-11T16:09:59.776Z I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2017-07-11T16:09:59.776Z I CONTROL  [initandlisten] 
2017-07-11T16:09:59.787Z I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/opt/collaborne/data/persistency/diagnostic.data'
2017-07-11T16:09:59.788Z I NETWORK  [thread1] waiting for connections on port 27017
2017-07-11T16:09:59.804Z W NETWORK  [replExecDBWorker-2] getaddrinfo("mongodb-0.mongodb.staging-external.svc.cluster.local") failed: Name or service not known
2017-07-11T16:09:59.809Z I NETWORK  [replExecDBWorker-2] getaddrinfo("mongodb-0.mongodb.staging-external.svc.cluster.local") failed: Name or service not known
2017-07-11T16:10:02.460Z W NETWORK  [replExecDBWorker-2] Failed to connect to 10.0.0.33:27017, in(checking socket for error after poll), reason: No route to host
2017-07-11T16:10:02.477Z W NETWORK  [replExecDBWorker-2] getaddrinfo("mongodb-1.mongodb.staging-external.svc.cluster.local") failed: Name or service not known
2017-07-11T16:10:02.611Z I NETWORK  [replExecDBWorker-2] getaddrinfo("mongodb-1.mongodb.staging-external.svc.cluster.local") failed: Name or service not known
2017-07-11T16:10:02.611Z W REPL     [replExecDBWorker-2] Locally stored replica set configuration does not have a valid entry for the current node; waiting for reconfig or remote heartbeat; Got "NodeNotFound: No host described in new configuration 3 for replica set rs0 maps to this node" while validating { _id: "rs0", version: 3, protocolVersion: 1, members: [ { _id: 0, host: "mongodb-0.mongodb.staging-external.svc.cluster.local:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "mongodb-arbiter.staging-external.svc.cluster.local:27017", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "mongodb-1.mongodb.staging-external.svc.cluster.local:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: 2000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('595cfae818a40c19332b525b') } }
2017-07-11T16:10:02.612Z W REPL     [replExecDBWorker-2] Failed to load timestamp of most recently applied operation: InitialSyncFailure: In the middle of an initial sync.
2017-07-11T16:10:02.612Z I REPL     [replExecDBWorker-2] New replica set config in use: { _id: "rs0", version: 3, protocolVersion: 1, members: [ { _id: 0, host: "mongodb-0.mongodb.staging-external.svc.cluster.local:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "mongodb-arbiter.staging-external.svc.cluster.local:27017", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "mongodb-1.mongodb.staging-external.svc.cluster.local:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: 2000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('595cfae818a40c19332b525b') } }
2017-07-11T16:10:02.612Z I REPL     [replExecDBWorker-2] This node is not a member of the config
2017-07-11T16:10:02.612Z I REPL     [replExecDBWorker-2] transition to REMOVED
2017-07-11T16:10:02.612Z I NETWORK  [replExecDBWorker-2] legacy transport layer closing all connections
2017-07-11T16:10:02.612Z I REPL     [replExecDBWorker-2] Starting replication storage threads
2017-07-11T16:10:02.628Z I REPL     [replication-0] Starting initial sync (attempt 1 of 10)
2017-07-11T16:10:02.629Z I COMMAND  [ftdc] serverStatus was very slow: { after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after locks: 0, after network: 0, after opLatencies: 0, after opcounters: 0, after opcountersRepl: 0, after repl: 2628, after security: 2628, after storageEngine: 2628, after tcmalloc: 2628, after wiredTiger: 2628, at end: 2628 }
2017-07-11T16:10:12.181Z I REPL     [replication-0] Initial sync attempt finishing up.
2017-07-11T16:10:12.181Z I REPL     [replication-0] Initial Sync Attempt Statistics: { failedInitialSyncAttempts: 0, maxFailedInitialSyncAttempts: 10, initialSyncStart: new Date(1499789402628), initialSyncAttempts: [] }
2017-07-11T16:10:12.181Z E REPL     [replication-0] Initial sync attempt failed -- attempts left: 9 cause: InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.
2017-07-11T16:10:13.182Z I REPL     [replication-1] Starting initial sync (attempt 2 of 10)
2017-07-11T16:10:19.797Z I NETWORK  [thread1] connection accepted from 172.17.0.1:39780 #1 (1 connection now open)
2017-07-11T16:10:19.810Z I -        [conn1] end connection 172.17.0.1:39780 (1 connection now open)
2017-07-11T16:10:22.370Z I REPL     [replication-1] Initial sync attempt finishing up.
2017-07-11T16:10:22.370Z I REPL     [replication-1] Initial Sync Attempt Statistics: { failedInitialSyncAttempts: 1, maxFailedInitialSyncAttempts: 10, initialSyncStart: new Date(1499789402628), initialSyncAttempts: [ { durationMillis: 0, status: "InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.", syncSource: ":27017" } ] }
2017-07-11T16:10:22.370Z E REPL     [replication-1] Initial sync attempt failed -- attempts left: 8 cause: InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.
2017-07-11T16:10:23.386Z I REPL     [replication-0] Starting initial sync (attempt 3 of 10)
2017-07-11T16:10:29.796Z I NETWORK  [thread1] connection accepted from 172.17.0.1:40066 #2 (1 connection now open)
2017-07-11T16:10:29.797Z I -        [conn2] end connection 172.17.0.1:40066 (1 connection now open)
2017-07-11T16:10:32.423Z I REPL     [replication-0] Initial sync attempt finishing up.
2017-07-11T16:10:32.423Z I REPL     [replication-0] Initial Sync Attempt Statistics: { failedInitialSyncAttempts: 2, maxFailedInitialSyncAttempts: 10, initialSyncStart: new Date(1499789402628), initialSyncAttempts: [ { durationMillis: 0, status: "InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.", syncSource: ":27017" }, { durationMillis: 0, status: "InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.", syncSource: ":27017" } ] }
2017-07-11T16:10:32.423Z E REPL     [replication-0] Initial sync attempt failed -- attempts left: 7 cause: InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.
2017-07-11T16:10:33.424Z I REPL     [replication-1] Starting initial sync (attempt 4 of 10)
2017-07-11T16:10:39.289Z I NETWORK  [thread1] connection accepted from 172.17.0.18:43892 #3 (1 connection now open)
2017-07-11T16:10:39.289Z I NETWORK  [conn3] received client metadata from 172.17.0.18:43892 conn3: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.4.6" }, os: { type: "Linux", name: "Amazon Linux AMI release 2017.03", architecture: "x86_64", version: "Kernel 4.7.2" } }
2017-07-11T16:10:39.290Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-0.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:39.301Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-0.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:39.301Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-0.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:39.301Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-0.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:39.301Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-0.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:39.303Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-0.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:39.303Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-0.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:39.303Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-0.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:39.303Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-0.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:39.305Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-0.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:39.305Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-0.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:39.305Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-0.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:39.796Z I NETWORK  [thread1] connection accepted from 172.17.0.1:40172 #4 (2 connections now open)
2017-07-11T16:10:39.799Z I -        [conn4] end connection 172.17.0.1:40172 (2 connections now open)
2017-07-11T16:10:40.350Z I NETWORK  [thread1] connection accepted from 172.17.0.1:54372 #5 (2 connections now open)
2017-07-11T16:10:40.350Z I NETWORK  [conn5] received client metadata from 172.17.0.1:54372 conn5: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.6" }, os: { type: "Linux", name: "Amazon Linux AMI release 2017.03", architecture: "x86_64", version: "Kernel 4.7.2" } }
2017-07-11T16:10:40.353Z I -        [conn5] end connection 172.17.0.1:54372 (2 connections now open)
2017-07-11T16:10:40.411Z I NETWORK  [thread1] connection accepted from 172.17.0.1:54374 #6 (2 connections now open)
2017-07-11T16:10:40.412Z I NETWORK  [conn6] received client metadata from 172.17.0.1:54374 conn6: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.6" }, os: { type: "Linux", name: "Amazon Linux AMI release 2017.03", architecture: "x86_64", version: "Kernel 4.7.2" } }
2017-07-11T16:10:40.414Z I -        [conn6] end connection 172.17.0.1:54374 (2 connections now open)
2017-07-11T16:10:41.220Z I NETWORK  [thread1] connection accepted from 172.17.0.27:53224 #7 (2 connections now open)
2017-07-11T16:10:41.220Z I NETWORK  [conn7] received client metadata from 172.17.0.27:53224 conn7: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.4.6" }, os: { type: "Linux", name: "Amazon Linux AMI release 2017.03", architecture: "x86_64", version: "Kernel 4.7.2" } }
2017-07-11T16:10:41.221Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-1.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:41.224Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:41.224Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:41.224Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-1.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:41.225Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-1.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:41.226Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:41.227Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:41.227Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-1.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:41.227Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-1.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:41.228Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:41.228Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:41.229Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-1.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:42.441Z I REPL     [replication-0] Initial sync attempt finishing up.
2017-07-11T16:10:42.441Z I REPL     [replication-0] Initial Sync Attempt Statistics: { failedInitialSyncAttempts: 3, maxFailedInitialSyncAttempts: 10, initialSyncStart: new Date(1499789402628), initialSyncAttempts: [ { durationMillis: 0, status: "InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.", syncSource: ":27017" }, { durationMillis: 0, status: "InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.", syncSource: ":27017" }, { durationMillis: 0, status: "InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.", syncSource: ":27017" } ] }
2017-07-11T16:10:42.441Z E REPL     [replication-0] Initial sync attempt failed -- attempts left: 6 cause: InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.
2017-07-11T16:10:43.441Z I REPL     [replication-1] Starting initial sync (attempt 5 of 10)
2017-07-11T16:10:44.305Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-0.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:44.308Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-0.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:44.308Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-0.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:44.308Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-0.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:44.308Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-0.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:44.310Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-0.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:44.310Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-0.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:44.310Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-0.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:44.310Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-0.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:44.313Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-0.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:44.313Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-0.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:44.313Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-0.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:46.228Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-1.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:46.230Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:46.230Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:46.230Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-1.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:46.230Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-1.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:46.232Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:46.232Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:46.232Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-1.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:46.232Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-1.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:46.233Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:46.233Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:46.233Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-1.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:46.439Z I NETWORK  [thread1] connection accepted from 172.17.0.32:43300 #8 (3 connections now open)
2017-07-11T16:10:49.314Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-0.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:49.317Z I ASIO     [NetworkInterfaceASIO-Replication-0] Successfully connected to mongodb-0.mongodb.staging-external.svc.cluster.local:27017, took 3ms (1 connections now open to mongodb-0.mongodb.staging-external.svc.cluster.local:27017)
2017-07-11T16:10:49.318Z I REPL     [ReplicationExecutor] Member mongodb-0.mongodb.staging-external.svc.cluster.local:27017 is now in state SECONDARY
2017-07-11T16:10:49.796Z I NETWORK  [thread1] connection accepted from 172.17.0.1:40252 #9 (4 connections now open)
2017-07-11T16:10:49.798Z I -        [conn9] end connection 172.17.0.1:40252 (4 connections now open)
2017-07-11T16:10:51.233Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-1.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:51.236Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:51.236Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:51.236Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-1.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:51.236Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-1.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:51.239Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:51.239Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:51.240Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-1.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:51.240Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-1.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:51.242Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:51.242Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:51.242Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-1.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:52.452Z I REPL     [replication-1] Initial sync attempt finishing up.
2017-07-11T16:10:52.452Z I REPL     [replication-1] Initial Sync Attempt Statistics: { failedInitialSyncAttempts: 4, maxFailedInitialSyncAttempts: 10, initialSyncStart: new Date(1499789402628), initialSyncAttempts: [ { durationMillis: 0, status: "InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.", syncSource: ":27017" }, { durationMillis: 0, status: "InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.", syncSource: ":27017" }, { durationMillis: 0, status: "InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.", syncSource: ":27017" }, { durationMillis: 0, status: "InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.", syncSource: ":27017" } ] }
2017-07-11T16:10:52.452Z E REPL     [replication-1] Initial sync attempt failed -- attempts left: 5 cause: InitialSyncOplogSourceMissing: No valid sync source found in current replica set to do an initial sync.
2017-07-11T16:10:53.348Z I NETWORK  [thread1] connection accepted from 172.17.0.31:59742 #10 (4 connections now open)
2017-07-11T16:10:53.373Z I NETWORK  [thread1] connection accepted from 172.17.0.22:34962 #11 (5 connections now open)
2017-07-11T16:10:53.452Z I REPL     [replication-0] Starting initial sync (attempt 6 of 10)
2017-07-11T16:10:56.243Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-1.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:56.245Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:56.245Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:56.245Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-1.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:56.246Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-1.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:56.247Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:56.247Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:56.247Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-1.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:56.248Z I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb-1.mongodb.staging-external.svc.cluster.local:27017
2017-07-11T16:10:56.249Z I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 - HostUnreachable: HostUnreachable
2017-07-11T16:10:56.250Z I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb-1.mongodb.staging-external.svc.cluster.local:27017 due to failed operation on a connection
2017-07-11T16:10:56.250Z I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb-1.mongodb.staging-external.svc.cluster.local:27017; HostUnreachable: HostUnreachable
2017-07-11T16:10:56.754Z I -        [conn7] Invariant failure i < _members.size() src/mongo/db/repl/repl_set_config.cpp 620
2017-07-11T16:10:56.754Z I -        [conn7] 

***aborting after invariant() failure


2017-07-11T16:10:56.774Z F -        [conn7] Got signal: 6 (Aborted).

 0x55e560b6a921 0x55e560b69b39 0x55e560b6a01d 0x7f23ec4f5370 0x7f23ec1591d7 0x7f23ec15a8c8 0x55e55fe1340c 0x55e5605a447c 0x55e5605a45d9 0x55e56066176f 0x55e5605dbdab 0x55e5605c20ac 0x55e56006541f 0x55e560066b01 0x55e56067edf0 0x55e560284c78 0x55e55fe82c5d 0x55e55fe8358d 0x55e560ad2881 0x7f23ec4eddc5 0x7f23ec21b6ed
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55E55F5F9000","o":"1571921","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55E55F5F9000","o":"1570B39"},{"b":"55E55F5F9000","o":"157101D"},{"b":"7F23EC4E6000","o":"F370"},{"b":"7F23EC124000","o":"351D7","s":"gsignal"},{"b":"7F23EC124000","o":"368C8","s":"abort"},{"b":"55E55F5F9000","o":"81A40C","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"55E55F5F9000","o":"FAB47C"},{"b":"55E55F5F9000","o":"FAB5D9"},{"b":"55E55F5F9000","o":"106876F","s":"_ZN5mongo4repl23TopologyCoordinatorImpl26processReplSetRequestVotesERKNS0_23ReplSetRequestVotesArgsEPNS0_27ReplSetRequestVotesResponseERKNS0_6OpTimeE"},{"b":"55E55F5F9000","o":"FE2DAB","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl26processReplSetRequestVotesEPNS_16OperationContextERKNS0_23ReplSetRequestVotesArgsEPNS0_27ReplSetRequestVotesResponseE"},{"b":"55E55F5F9000","o":"FC90AC","s":"_ZN5mongo4repl22CmdReplSetRequestVotes3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_7BSONObjEiRS9_RNS_14BSONObjBuilderE"},{"b":"55E55F5F9000","o":"A6C41F","s":"_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE"},{"b":"55E55F5F9000","o":"A6DB01","s":"_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE"},{"b":"55E55F5F9000","o":"1085DF0","s":"_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE"},{"b":"55E55F5F9000","o":"C8BC78","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"55E55F5F9000","o":"889C5D","s":"_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE"},{"b":"55E55F5F9000","o":"88A58D"},{"b":"55E55F5F9000","o":"14D9881"},{"b":"7F23EC4E6000","o":"7DC5"},{"b":"7F23EC124000","o":"F76ED","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.6", "gitVersion" : "c55eb86ef46ee7aede3b1e2a5d184a7df4bfb5b5", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.7.2", "version" : "#1 SMP Fri Apr 7 10:38:40 PDT 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "55E55F5F9000", "elfType" : 3, "buildId" : "E7DAD75E0AB1FE58FC79C75D5C934D568B747E3E" }, { "b" : "7FFE584DF000", "elfType" : 3, "buildId" : "E1B7CD9BB3B12443C1D330322068CD7DE34C2D53" }, { "b" : "7F23ED40E000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "FAF738A64DDEFBE202A700A23E88667132CFB755" }, { "b" : "7F23ED026000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "BA85B1BBD34D0502B5648916F409484083D688E7" }, { "b" : "7F23ECE1E000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "84240319CB3644443A8E5D6E265ABE8E023B81E6" }, { "b" : "7F23ECC1A000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "EB575314825D0BB0D64D2251E8B779E52FA8D419" }, { "b" : "7F23EC918000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "417E9F2F89F47EBA68BBFC48EC70020874F5D071" }, { "b" : "7F23EC702000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "3FD5F89DE59E124AB1419A0BD16775B4096E84FD" }, { "b" : "7F23EC4E6000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "9620D82D9ED384A215A25AC280672B8F5B8C8553" }, { "b" : "7F23EC124000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "071017DCC4755642CFC82A38A7EB74B535476A45" }, { "b" : "7F23ED67C000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "B2DC1178D4666475D45C1D9E8A3861C250DD1323" }, { "b" : "7F23EBED6000", "path" : "/usr/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "1BE9E6309ED365E35806E13FA9E23350D71F2513" }, { "b" : "7F23EBBEF000", "path" : "/usr/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "9EE23694485D684651195C7B51766E47D0CB95E3" }, { "b" : "7F23EB9EC000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "5C01209C5AE1B1714F19B07EB58F2A1274B69DC8" }, { "b" : "7F23EB7BA000", "path" : "/usr/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "FD5974E4861D56DFFFFC8BF5DB35E74B1C20ABD5" }, { "b" : "7F23EB5A4000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "89C6AF118B6B4FB6A73AE1813E2C8BDD722956D1" }, { "b" : "7F23EB395000", "path" : "/usr/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "1B55330B231D45AF433F7D9DCA507C5FB0609780" }, { "b" : "7F23EB192000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7F23EAF78000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "13F0C4788711C7B6BC537231CA981ECD345CAE5E" }, { "b" : "7F23EAD57000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "F5054DC94443326819FBF3065CFDF5E4726F57EE" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55e560b6a921]
 mongod(+0x1570B39) [0x55e560b69b39]
 mongod(+0x157101D) [0x55e560b6a01d]
 libpthread.so.0(+0xF370) [0x7f23ec4f5370]
 libc.so.6(gsignal+0x37) [0x7f23ec1591d7]
 libc.so.6(abort+0x148) [0x7f23ec15a8c8]
 mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x0) [0x55e55fe1340c]
 mongod(+0xFAB47C) [0x55e5605a447c]
 mongod(+0xFAB5D9) [0x55e5605a45d9]
 mongod(_ZN5mongo4repl23TopologyCoordinatorImpl26processReplSetRequestVotesERKNS0_23ReplSetRequestVotesArgsEPNS0_27ReplSetRequestVotesResponseERKNS0_6OpTimeE+0x16F) [0x55e56066176f]
 mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl26processReplSetRequestVotesEPNS_16OperationContextERKNS0_23ReplSetRequestVotesArgsEPNS0_27ReplSetRequestVotesResponseE+0x16B) [0x55e5605dbdab]
 mongod(_ZN5mongo4repl22CmdReplSetRequestVotes3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERNS_7BSONObjEiRS9_RNS_14BSONObjBuilderE+0x2DC) [0x55e5605c20ac]
 mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x4FF) [0x55e56006541f]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0xF81) [0x55e560066b01]
 mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x240) [0x55e56067edf0]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xD38) [0x55e560284c78]
 mongod(_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE+0x1FD) [0x55e55fe82c5d]
 mongod(+0x88A58D) [0x55e55fe8358d]
 mongod(+0x14D9881) [0x55e560ad2881]
 libpthread.so.0(+0x7DC5) [0x7f23ec4eddc5]
 libc.so.6(clone+0x6D) [0x7f23ec21b6ed]
-----  END BACKTRACE  -----




----

Happened after playing with minikube start/stop, shortly after services started coming back up. Likely very corrupted state before that, caused by earlier start/stop cycles that definitely didn't
wait until things are good.