2016-02-07T01:01:54.000+0000 I NETWORK  [conn141] end connection 10.206.8.229:52590 (18 connections now open)
2016-02-07T01:18:14.850+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._392
2016-02-07T01:18:14.850+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._393
2016-02-07T01:35:22.874+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._394
2016-02-07T01:46:58.083+0000 I NETWORK  [initandlisten] connection accepted from 10.206.8.229:54593 #142 (19 connections now open)
2016-02-07T01:46:58.084+0000 I COMMAND  [conn142] command: unlock requested
2016-02-07T01:46:58.086+0000 I NETWORK  [conn142] end connection 10.206.8.229:54593 (18 connections now open)
2016-02-07T01:46:58.142+0000 I NETWORK  [initandlisten] connection accepted from 10.206.8.229:54594 #143 (19 connections now open)
2016-02-07T01:46:58.145+0000 I NETWORK  [conn143] end connection 10.206.8.229:54594 (18 connections now open)
2016-02-07T01:52:35.132+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._395
2016-02-07T02:09:32.094+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._396
2016-02-07T02:25:55.412+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._397
2016-02-07T02:43:29.059+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._398
2016-02-07T03:02:45.699+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._399
2016-02-07T03:20:01.553+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._400
2016-02-07T03:37:58.066+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._401
2016-02-07T03:57:27.121+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._402
2016-02-07T04:04:57.695+0000 I STORAGE  [rsSync] passes = 5000 in CappedRecordStoreV1::allocRecord: ns: local.oplog.rs, lenToAlloc: 1107552, maxCappedDocs: 9223372036854775807, nrecords: 558998461, datasize: 119903669380. Continuing to delete old records to make room.
2016-02-07T04:15:03.964+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._403
2016-02-07T04:32:59.426+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._404
2016-02-07T04:52:30.470+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._405
2016-02-07T05:06:11.339+0000 I STORAGE  [FileAllocator] allocating new datafile /data/Appsee.810, filling with zeroes...
2016-02-07T05:06:11.344+0000 I STORAGE  [FileAllocator] done allocating datafile /data/Appsee.810, size: 2047MB,  took 0.005 secs
2016-02-07T05:10:39.165+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._406
2016-02-07T05:28:19.233+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._407
2016-02-07T05:47:48.724+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._408
2016-02-07T06:07:33.820+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._409
2016-02-07T06:25:21.331+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._410
2016-02-07T06:45:30.419+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._411
2016-02-07T06:57:05.153+0000 I STORAGE  [rsSync] passes = 5000 in CappedRecordStoreV1::allocRecord: ns: local.oplog.rs, lenToAlloc: 1156056, maxCappedDocs: 9223372036854775807, nrecords: 558007459, datasize: 119919541172. Continuing to delete old records to make room.
2016-02-07T07:05:50.069+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._412
2016-02-07T07:22:47.375+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._413
2016-02-07T07:41:59.419+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._414
2016-02-07T08:02:25.372+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._415
2016-02-07T08:17:52.156+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._416
2016-02-07T08:35:42.677+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._417
2016-02-07T08:45:19.352+0000 I STORAGE  [rsSync] passes = 5000 in CappedRecordStoreV1::allocRecord: ns: local.oplog.rs, lenToAlloc: 1435188, maxCappedDocs: 9223372036854775807, nrecords: 557621488, datasize: 119925732540. Continuing to delete old records to make room.
2016-02-07T08:55:43.707+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._418
2016-02-07T09:11:58.155+0000 I JOURNAL  [journal writer] old journal file will be removed: /data/journal/j._419
2016-02-07T09:26:56.483+0000 I NETWORK  [conn111] end connection 10.206.8.229:34056 (17 connections now open)
2016-02-07T09:26:56.483+0000 I NETWORK  [conn39] end connection 10.206.8.229:60656 (17 connections now open)
2016-02-07T09:27:10.523+0000 I REPL     [SyncSourceFeedback] SyncSourceFeedback error sending update, response: { configVersion: 77065, ok: 0.0, errmsg: "Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065", code: 93 }
2016-02-07T09:27:10.525+0000 I REPL     [SyncSourceFeedback] updateUpstream failed: InvalidReplicaSetConfig Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065, will retry
2016-02-07T09:27:10.528+0000 I REPL     [SyncSourceFeedback] SyncSourceFeedback error sending update, response: { configVersion: 77065, ok: 0.0, errmsg: "Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065", code: 93 }
2016-02-07T09:27:10.528+0000 I REPL     [SyncSourceFeedback] updateUpstream failed: InvalidReplicaSetConfig Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065, will retry
2016-02-07T09:27:10.554+0000 I REPL     [SyncSourceFeedback] SyncSourceFeedback error sending update, response: { configVersion: 77065, ok: 0.0, errmsg: "Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065", code: 93 }
2016-02-07T09:27:10.554+0000 I REPL     [SyncSourceFeedback] updateUpstream failed: InvalidReplicaSetConfig Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065, will retry
2016-02-07T09:27:10.564+0000 I REPL     [SyncSourceFeedback] SyncSourceFeedback error sending update, response: { configVersion: 77065, ok: 0.0, errmsg: "Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065", code: 93 }
2016-02-07T09:27:10.564+0000 I REPL     [SyncSourceFeedback] updateUpstream failed: InvalidReplicaSetConfig Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065, will retry
2016-02-07T09:27:10.586+0000 I REPL     [SyncSourceFeedback] SyncSourceFeedback error sending update, response: { configVersion: 77065, ok: 0.0, errmsg: "Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065", code: 93 }
2016-02-07T09:27:10.586+0000 I REPL     [SyncSourceFeedback] updateUpstream failed: InvalidReplicaSetConfig Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065, will retry
2016-02-07T09:27:10.598+0000 I REPL     [SyncSourceFeedback] SyncSourceFeedback error sending update, response: { configVersion: 77065, ok: 0.0, errmsg: "Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065", code: 93 }
2016-02-07T09:27:10.598+0000 I REPL     [SyncSourceFeedback] updateUpstream failed: InvalidReplicaSetConfig Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065, will retry
2016-02-07T09:27:10.622+0000 I REPL     [SyncSourceFeedback] SyncSourceFeedback error sending update, response: { configVersion: 77065, ok: 0.0, errmsg: "Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065", code: 93 }
2016-02-07T09:27:10.622+0000 I REPL     [SyncSourceFeedback] updateUpstream failed: InvalidReplicaSetConfig Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065, will retry
2016-02-07T09:27:10.632+0000 I REPL     [SyncSourceFeedback] SyncSourceFeedback error sending update, response: { configVersion: 77065, ok: 0.0, errmsg: "Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065", code: 93 }
2016-02-07T09:27:10.632+0000 I REPL     [SyncSourceFeedback] updateUpstream failed: InvalidReplicaSetConfig Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065, will retry
2016-02-07T09:27:10.657+0000 I REPL     [SyncSourceFeedback] SyncSourceFeedback error sending update, response: { configVersion: 77065, ok: 0.0, errmsg: "Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065", code: 93 }
2016-02-07T09:27:10.657+0000 I REPL     [SyncSourceFeedback] updateUpstream failed: InvalidReplicaSetConfig Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065, will retry
2016-02-07T09:27:10.666+0000 I REPL     [SyncSourceFeedback] SyncSourceFeedback error sending update, response: { configVersion: 77065, ok: 0.0, errmsg: "Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065", code: 93 }
2016-02-07T09:27:10.666+0000 I REPL     [SyncSourceFeedback] updateUpstream failed: InvalidReplicaSetConfig Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065, will retry
2016-02-07T09:27:10.691+0000 I REPL     [SyncSourceFeedback] SyncSourceFeedback error sending update, response: { configVersion: 77065, ok: 0.0, errmsg: "Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065", code: 93 }
2016-02-07T09:27:10.691+0000 I REPL     [SyncSourceFeedback] updateUpstream failed: InvalidReplicaSetConfig Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065, will retry
2016-02-07T09:27:10.700+0000 I NETWORK  [conn91] end connection 10.46.13.176:49926 (15 connections now open)
2016-02-07T09:27:10.725+0000 I REPL     [SyncSourceFeedback] SyncSourceFeedback error sending update, response: { configVersion: 77065, ok: 0.0, errmsg: "Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065", code: 93 }
2016-02-07T09:27:10.725+0000 I REPL     [SyncSourceFeedback] updateUpstream failed: InvalidReplicaSetConfig Received replSetUpdatePosition for node with memberId 2 whose config version of 77064 doesn't match our config version of 77065, will retry
2016-02-07T09:27:10.744+0000 I REPL     [ReplicationExecutor] Cannot find self in new replica set configuration; I must be removed; NodeNotFound No host described in new configuration 77065 for replica set rsAppsee2 maps to this node
2016-02-07T09:27:10.745+0000 I REPL     [ReplicationExecutor] New replica set config in use: { _id: "rsAppsee2", version: 77065, protocolVersion: 1, members: [ { _id: 0, host: "in.db2m1.appsee.com:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 2.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "in.db2arb.appsee.com:27017", arbiterOnly: true, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 4, host: "in.db2m2.appsee.com:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.0, tags: {}, slaveDelay: 0, votes: 0 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
2016-02-07T09:27:10.745+0000 I REPL     [ReplicationExecutor] This node is not a member of the config
2016-02-07T09:27:10.745+0000 I REPL     [ReplicationExecutor] transition to REMOVED
2016-02-07T09:27:10.745+0000 I NETWORK  [conn21] end connection 10.206.8.229:59902 (14 connections now open)
2016-02-07T09:27:10.745+0000 I NETWORK  [conn33] end connection 10.35.154.152:59792 (14 connections now open)
2016-02-07T09:27:10.745+0000 I NETWORK  [conn87] end connection 10.46.13.176:49919 (14 connections now open)
2016-02-07T09:27:10.745+0000 I NETWORK  [conn23] end connection 10.206.8.229:60376 (13 connections now open)
2016-02-07T09:27:10.745+0000 I NETWORK  [conn38] end connection 10.41.58.141:56919 (13 connections now open)
2016-02-07T09:27:10.745+0000 I NETWORK  [conn102] end connection 10.46.13.176:50408 (11 connections now open)
2016-02-07T09:27:10.745+0000 I NETWORK  [conn32] end connection 10.46.2.126:36694 (11 connections now open)
2016-02-07T09:27:10.745+0000 I NETWORK  [conn31] end connection 10.40.7.177:52130 (14 connections now open)
2016-02-07T09:27:10.745+0000 I NETWORK  [conn37] end connection 10.37.137.232:36157 (13 connections now open)
2016-02-07T09:27:10.746+0000 I NETWORK  [conn55] end connection 10.46.2.126:37434 (7 connections now open)
2016-02-07T09:27:10.746+0000 I NETWORK  [initandlisten] connection accepted from 10.206.8.229:41182 #144 (8 connections now open)
2016-02-07T09:27:10.747+0000 I NETWORK  [conn84] end connection 10.46.13.176:49916 (7 connections now open)
2016-02-07T09:27:10.747+0000 I -        [ReplicationExecutor] Invariant failure i < _members.size() src/mongo/db/repl/replica_set_config.cpp 505
2016-02-07T09:27:10.747+0000 I NETWORK  [conn34] end connection 10.229.6.198:44452 (6 connections now open)
2016-02-07T09:27:10.747+0000 I NETWORK  [conn36] end connection 10.41.14.22:59417 (5 connections now open)
2016-02-07T09:27:10.747+0000 I NETWORK  [conn30] end connection 10.251.122.71:52350 (4 connections now open)
2016-02-07T09:27:10.745+0000 I NETWORK  [conn35] end connection 10.43.4.125:45733 (14 connections now open)
2016-02-07T09:27:10.748+0000 I -        [ReplicationExecutor] 

***aborting after invariant() failure


2016-02-07T09:27:10.786+0000 F -        [ReplicationExecutor] Got signal: 6 (Aborted).

 0x12d5772 0x12d46a9 0x12d4eb2 0x7f8141d17100 0x7f814197b5f7 0x7f814197cce8 0x125c83b 0xe7de2a 0xef5fe2 0xea8849 0xebc9d2 0xec19e5 0x1a99100 0x7f8141d0fdc5 0x7f8141a3cbdd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"ED5772"},{"b":"400000","o":"ED46A9"},{"b":"400000","o":"ED4EB2"},{"b":"7F8141D08000","o":"F100"},{"b":"7F8141946000","o":"355F7"},{"b":"7F8141946000","o":"36CE8"},{"b":"400000","o":"E5C83B"},{"b":"400000","o":"A7DE2A"},{"b":"400000","o":"AF5FE2"},{"b":"400000","o":"AA8849"},{"b":"400000","o":"ABC9D2"},{"b":"400000","o":"AC19E5"},{"b":"400000","o":"1699100"},{"b":"7F8141D08000","o":"7DC5"},{"b":"7F8141946000","o":"F6BDD"}],"processInfo":{ "mongodbVersion" : "3.2.1", "gitVersion" : "a14d55980c2cdc565d4704a7e3ad37e4e535c1b2", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.14.44-32.39.amzn1.x86_64", "version" : "#1 SMP Thu Jun 11 20:33:38 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "A931E563057BCC34ED2D1971AAC77D44F47033A9" }, { "b" : "7FFC6FE9C000", "elfType" : 3, "buildId" : "3C9325999495818299EC622C63D9330D234E02C9" }, { "b" : "7F8142F31000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "B949A1349FF0E93409055F00BFD60F758EE8FA02" }, { "b" : "7F8142B4C000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "F444116328F54797379DAA3B0FD53CF6C829F22A" }, { "b" : "7F8142944000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "42833B65941483A611C40EA7D32F56EA83EA6E93" }, { "b" : "7F8142740000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "6335077ACD51527BE9F2F18451A88E2B7350C5B6" }, { "b" : "7F814243C000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "DD6383EEAC49E9BAA9E3D1080AE932F42CF8A385" }, { "b" : "7F814213A000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "BB312C4A65B8FD830C148612CBEACEACC8B08E4F" }, { "b" : "7F8141F24000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "C52958E393BDF8E8D090F36DE0F4E620D8736FBF" }, { "b" : "7F8141D08000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "E5E575776DAD20ADE8FD0CAF17897C9D89020A87" }, { "b" : "7F8141946000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "D84E3AFDFF3E164A09C125F85B5DCABC6F545B5E" }, { "b" : "7F814319E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "7B7BF8FEEF1A9C627EF90CA5C9188EFD4DA2DDD2" }, { "b" : "7F8141703000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "DE5A9F7A11A0881CB64E375F4DDCA58028F0FAF8" }, { "b" : "7F814141E000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "A3E43FC66908AC8B00773707FECA3B1677AFF311" }, { "b" : "7F814121B000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "622F315EB5CB2F791E9B64020692EBA98195D06D" }, { "b" : "7F8140FF0000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "B10FBFEC246C4EAD1719D16090D0BE54904BBFC9" }, { "b" : "7F8140DDA000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "87B4EBF2183C8EA4AB657212203EFFE6340E2F4F" }, { "b" : "7F8140BCF000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "7292C0673D7C116E3389D3FFA67087A6B9287A71" }, { "b" : "7F81409CC000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7F81407B2000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "47EC2C63132D25E4FE83F77023DA1A66457A88F1" }, { "b" : "7F8140591000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "F5054DC94443326819FBF3065CFDF5E4726F57EE" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12d5772]
 mongod(+0xED46A9) [0x12d46a9]
 mongod(+0xED4EB2) [0x12d4eb2]
 libpthread.so.0(+0xF100) [0x7f8141d17100]
 libc.so.6(gsignal+0x37) [0x7f814197b5f7]
 libc.so.6(abort+0x148) [0x7f814197cce8]
 mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xCB) [0x125c83b]
 mongod(+0xA7DE2A) [0xe7de2a]
 mongod(_ZNK5mongo4repl23TopologyCoordinatorImpl22shouldChangeSyncSourceERKNS_11HostAndPortERKNS0_6OpTimeES7_bNS_6Date_tE+0x222) [0xef5fe2]
 mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl23_shouldChangeSyncSourceERKNS_8executor12TaskExecutor12CallbackArgsERKNS_11HostAndPortERKNS0_6OpTimeEbPb+0x89) [0xea8849]
 mongod(+0xABC9D2) [0xebc9d2]
 mongod(_ZN5mongo4repl19ReplicationExecutor3runEv+0x275) [0xec19e5]
 mongod(+0x1699100) [0x1a99100]
 libpthread.so.0(+0x7DC5) [0x7f8141d0fdc5]
 libc.so.6(clone+0x6D) [0x7f8141a3cbdd]
-----  END BACKTRACE  -----