2018-03-19T10:19:47.443+0300 I NETWORK  [listener] connection accepted from 10.0.0.109:33612 #380410 (720 connections now open)
2018-03-19T10:19:47.461+0300 I NETWORK  [listener] connection accepted from 10.0.0.66:54004 #380411 (721 connections now open)
2018-03-19T10:19:47.476+0300 I NETWORK  [conn380410] received client metadata from 10.0.0.109:33612 conn: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "16.04" } }
2018-03-19T10:19:47.477+0300 I ACCESS   [conn380410]  authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "sh0-rs3-0", $db: "$external" }
2018-03-19T10:19:47.477+0300 I SHARDING [conn380410] Refreshing chunks for collection deliveryclub.ancore_adwords_dc_type_daily_v1 based on version 0|0||000000000000000000000000
2018-03-19T10:19:47.485+0300 I SHARDING [ConfigServerCatalogCacheLoader-1465] Cache loader remotely refreshed for collection deliveryclub.ancore_adwords_dc_type_daily_v1 from collection version 0|0||000000000000000000000000 and found collection version 1|3||5aaf6492b34d08a0893112b5
2018-03-19T10:19:47.486+0300 I SHARDING [ConfigServerCatalogCacheLoader-1465] Cache loader found enqueued metadata from 1|0||5aaf6492b34d08a0893112b5 to 1|3||5aaf6492b34d08a0893112b5 and no persisted metadata, GTE cache version 0|0||000000000000000000000000
2018-03-19T10:19:47.486+0300 I SHARDING [ConfigServerCatalogCacheLoader-1465] Refresh for collection deliveryclub.ancore_adwords_dc_type_daily_v1 took 8 ms and found version 1|3||5aaf6492b34d08a0893112b5
2018-03-19T10:19:47.486+0300 I SHARDING [conn379795] Refreshing chunks for collection priceru.raw_adwords_adgroup_full based on version 79|16||59c8c896ca4864b1489f0eea
2018-03-19T10:19:47.486+0300 I SHARDING [conn380410] Marking collection deliveryclub.ancore_adwords_dc_type_daily_v1 as sharded with collection version: 1|3||5aaf6492b34d08a0893112b5, shard version: 0|0||5aaf6492b34d08a0893112b5
2018-03-19T10:19:47.486+0300 I SHARDING [migrateThread] Starting receiving end of migration of chunk { startDate: MinKey } -> { startDate: -4611686018427387900 } for collection deliveryclub.ancore_adwords_dc_type_daily_v1 from sh0-rs3/sh0-rs3-0.ohm.pw:27021,sh0-rs3-1.ohm.pw:27021,sh0-rs3-2.ohm.pw:27021 at epoch 5aaf6492b34d08a0893112b5 with session id sh0-rs3_rs0_5aaf64938825cf763e7d076d
2018-03-19T10:19:47.486+0300 I STORAGE  [ShardServerCatalogCacheLoader-5334] createCollection: config.cache.chunks.deliveryclub.ancore_adwords_dc_type_daily_v1 with generated UUID: 4f5a7628-18bb-4675-a097-310c7923870a
2018-03-19T10:19:47.493+0300 I SHARDING [conn380237] Refreshing chunks for collection priceru.raw_adwords_critea_full based on version 65|37||59c8c89cca4864b1489f0f42
2018-03-19T10:19:47.499+0300 I NETWORK  [conn380411] received client metadata from 10.0.0.66:54004 conn: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-13", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "16.04" } }
2018-03-19T10:19:47.501+0300 I ACCESS   [conn380411]  authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "sh0", $db: "$external" }
2018-03-19T10:19:47.502+0300 I INDEX    [ShardServerCatalogCacheLoader-5334] build index on: config.cache.chunks.deliveryclub.ancore_adwords_dc_type_daily_v1 properties: { v: 2, key: { lastmod: 1 }, name: "lastmod_1", ns: "config.cache.chunks.deliveryclub.ancore_adwords_dc_type_daily_v1" }
2018-03-19T10:19:47.502+0300 I INDEX    [ShardServerCatalogCacheLoader-5334]     building index using bulk method; build may temporarily use up to 8196 megabytes of RAM
2018-03-19T10:19:47.504+0300 I INDEX    [ShardServerCatalogCacheLoader-5334] build index done.  scanned 0 total records. 0 secs
2018-03-19T10:19:47.535+0300 I NETWORK  [listener] connection accepted from 10.0.0.66:54008 #380412 (722 connections now open)
2018-03-19T10:19:47.554+0300 I NETWORK  [migrateThread] Successfully connected to sh0-rs3/sh0-rs3-0.ohm.pw:27021,sh0-rs3-1.ohm.pw:27021,sh0-rs3-2.ohm.pw:27021 (1 connections now open to sh0-rs3/sh0-rs3-0.ohm.pw:27021,sh0-rs3-1.ohm.pw:27021,sh0-rs3-2.ohm.pw:27021 with a 0 second timeout)
2018-03-19T10:19:47.556+0300 I STORAGE  [migrateThread] createCollection: deliveryclub.ancore_adwords_dc_type_daily_v1 with provided UUID: 98b297f5-1023-45a7-96a2-d2953f40b317
2018-03-19T10:19:47.573+0300 I SHARDING [ConfigServerCatalogCacheLoader-1465] Cache loader remotely refreshed for collection priceru.raw_adwords_adgroup_full from collection version 82|1||59c8c896ca4864b1489f0eea and found collection version 82|19||59c8c896ca4864b1489f0eea
2018-03-19T10:19:47.574+0300 I SHARDING [ConfigServerCatalogCacheLoader-1465] Cache loader found enqueued metadata from 82|1||59c8c896ca4864b1489f0eea to 82|19||59c8c896ca4864b1489f0eea and persisted metadata from 79|16||59c8c896ca4864b1489f0eea to 82|1||59c8c896ca4864b1489f0eea, GTE cache version 79|16||59c8c896ca4864b1489f0eea
2018-03-19T10:19:47.575+0300 I SHARDING [ConfigServerCatalogCacheLoader-1465] Refresh for collection priceru.raw_adwords_adgroup_full took 89 ms and found version 82|19||59c8c896ca4864b1489f0eea
2018-03-19T10:19:47.575+0300 I SHARDING [conn379795] Marking collection priceru.raw_adwords_adgroup_full as sharded with collection version: 82|19||59c8c896ca4864b1489f0eea, shard version: 82|1||59c8c896ca4864b1489f0eea
2018-03-19T10:19:47.579+0300 I SHARDING [ConfigServerCatalogCacheLoader-1466] Cache loader remotely refreshed for collection priceru.raw_adwords_critea_full from collection version 65|40||59c8c89cca4864b1489f0f42 and found collection version 65|64||59c8c89cca4864b1489f0f42
2018-03-19T10:19:47.581+0300 I NETWORK  [conn380412] received client metadata from 10.0.0.66:54008 conn: { driver: { name: "MongoDB Internal Client", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "16.04" } }
2018-03-19T10:19:47.583+0300 I SHARDING [ConfigServerCatalogCacheLoader-1466] Cache loader found enqueued metadata from 65|40||59c8c89cca4864b1489f0f42 to 65|64||59c8c89cca4864b1489f0f42 and persisted metadata from 65|37||59c8c89cca4864b1489f0f42 to 65|40||59c8c89cca4864b1489f0f42, GTE cache version 65|37||59c8c89cca4864b1489f0f42
2018-03-19T10:19:47.583+0300 I ACCESS   [conn380412]  authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "sh0", $db: "$external" }
2018-03-19T10:19:47.583+0300 I SHARDING [ConfigServerCatalogCacheLoader-1466] Refresh for collection priceru.raw_adwords_critea_full took 90 ms and found version 65|64||59c8c89cca4864b1489f0f42
2018-03-19T10:19:47.584+0300 I SHARDING [conn380237] Marking collection priceru.raw_adwords_critea_full as sharded with collection version: 65|64||59c8c89cca4864b1489f0f42, shard version: 65|58||59c8c89cca4864b1489f0f42
2018-03-19T10:19:47.589+0300 I INDEX    [migrateThread] build index on: deliveryclub.ancore_adwords_dc_type_daily_v1 properties: { v: 2, key: { startDate: 1 }, name: "startDate_1", ns: "deliveryclub.ancore_adwords_dc_type_daily_v1", background: true }
2018-03-19T10:19:47.589+0300 I INDEX    [migrateThread]          building index using bulk method; build may temporarily use up to 2049 megabytes of RAM
2018-03-19T10:19:47.591+0300 I NETWORK  [listener] connection accepted from 10.0.0.61:50234 #380413 (723 connections now open)
2018-03-19T10:19:47.599+0300 I INDEX    [migrateThread] build index on: deliveryclub.ancore_adwords_dc_type_daily_v1 properties: { v: 2, key: { endDate: 1 }, name: "endDate_1", ns: "deliveryclub.ancore_adwords_dc_type_daily_v1", background: true }
2018-03-19T10:19:47.599+0300 I INDEX    [migrateThread]          building index using bulk method; build may temporarily use up to 2049 megabytes of RAM
2018-03-19T10:19:47.605+0300 I INDEX    [migrateThread] build index on: deliveryclub.ancore_adwords_dc_type_daily_v1 properties: { v: 2, key: { accountId: 1 }, name: "accountId_1", ns: "deliveryclub.ancore_adwords_dc_type_daily_v1", background: true }
2018-03-19T10:19:47.605+0300 I INDEX    [migrateThread]          building index using bulk method; build may temporarily use up to 2049 megabytes of RAM
2018-03-19T10:19:47.611+0300 I INDEX    [migrateThread] build index on: deliveryclub.ancore_adwords_dc_type_daily_v1 properties: { v: 2, key: { startDate: "hashed" }, name: "startDate_hashed", ns: "deliveryclub.ancore_adwords_dc_type_daily_v1", background: true }
2018-03-19T10:19:47.611+0300 I INDEX    [migrateThread]          building index using bulk method; build may temporarily use up to 2049 megabytes of RAM
2018-03-19T10:19:47.612+0300 I SHARDING [migrateThread] Scheduling deletion of any documents in deliveryclub.ancore_adwords_dc_type_daily_v1 range [{ startDate: MinKey }, { startDate: -4611686018427387900 }) before migrating in a chunk covering the range
2018-03-19T10:19:47.612+0300 I SHARDING [Collection Range Deleter] No documents remain to delete in deliveryclub.ancore_adwords_dc_type_daily_v1 range [{ startDate: MinKey }, { startDate: -4611686018427387900 })
2018-03-19T10:19:47.612+0300 I SHARDING [migrateThread] Finished deleting deliveryclub.ancore_adwords_dc_type_daily_v1 range [{ startDate: MinKey }, { startDate: -4611686018427387900 })
2018-03-19T10:19:47.613+0300 I ASIO     [NetworkInterfaceASIO-ShardRegistry-0] Connecting to sh0-rs3-0.ohm.pw:27021
2018-03-19T10:19:47.613+0300 I SHARDING [migrateThread] Waiting for replication to catch up before entering critical section
2018-03-19T10:19:47.628+0300 I NETWORK  [conn380413] received client metadata from 10.0.0.61:50234 conn: { driver: { name: "MongoDB Internal Client", version: "3.6.3" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "16.04" } }
2018-03-19T10:19:47.628+0300 I ACCESS   [conn380413]  authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "sh0", $db: "$external" }
2018-03-19T10:19:47.654+0300 I ASIO     [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to sh0-rs3-0.ohm.pw:27021, took 41ms (1 connections now open to sh0-rs3-0.ohm.pw:27021)
2018-03-19T10:19:47.684+0300 F -        [sessionCatalogMigration-sh0-rs3_rs0_5aaf64938825cf763e7d076d] Invariant failure oplogEntry.getWallClockTime() src/mongo/db/s/session_catalog_migration_destination.cpp 246
2018-03-19T10:19:47.684+0300 F -        [sessionCatalogMigration-sh0-rs3_rs0_5aaf64938825cf763e7d076d]

***aborting after invariant() failure


2018-03-19T10:19:47.790+0300 I NETWORK  [listener] connection accepted from 10.0.0.66:54014 #380414 (724 connections now open)
2018-03-19T10:19:47.825+0300 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to sh0-rs0-0.ohm.pw:27021
2018-03-19T10:19:47.830+0300 F -        [sessionCatalogMigration-sh0-rs3_rs0_5aaf64938825cf763e7d076d] Got signal: 6 (Aborted).

 0x55688835d071 0x55688835c289 0x55688835c76d 0x7fb70f704390 0x7fb70f35e428 0x7fb70f36002a 0x556886ad525c 0x55688717dca8 0x55688846c170 0x7fb70f6fa6ba 0x7fb70f43041d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55688614B000","o":"2212071","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55688614B000","o":"2211289"},{"b":"55688614B000","o":"221176D"},{"b":"7FB70F6F3000","o":"11390"},{"b":"7FB70F329000","o":"35428","s":"gsignal"},{"b":"7FB70F329000","o":"3702A","s":"abort"},{"b":"55688614B000","o":"98A25C","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"55688614B000","o":"1032CA8","s":"_ZN5mongo34SessionCatalogMigrationDestination31_retrieveSessionStateFromSourceEPNS_14ServiceContextE"},{"b":"55688614B000","o":"2321170"},{"b":"7FB70F6F3000","o":"76BA"},{"b":"7FB70F329000","o":"10741D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.3", "gitVersion" : "9586e557d54ef70f9ca4b43c26892cd55257e1a5", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-112-generic", "version" : "#135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "55688614B000", "elfType" : 3, "buildId" : "7A0DE17E89E4F8A85A88DD691D5A6B4053CB0C42" }, { "b" : "7FFFDF3D8000", "elfType" : 3, "buildId" : "FA97F4849697BBE252BA1F7FB2316979E93E61DE" }, { "b" : "7FB7108E8000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "6EF73266978476EF9F2FD2CF31E57F4597CB74F8" }, { "b" : "7FB71067F000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "DCF10134B91ED2139E3E8C72564668F5CDBA8522" }, { "b" : "7FB71023B000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "1649272BE0CA9FA22F082DC86372B6C9959779B0" }, { "b" : "7FB710033000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "89C34D7A182387D76D5CDA1F7718F5D58824DFB3" }, { "b" : "7FB70FE2F000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "8CC8D0D119B142D839800BFF71FB71E73AEA7BD4" }, { "b" : "7FB70FB26000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "DFB85DE42DAFFD09640C8FE377D572DE3E168920" }, { "b" : "7FB70F910000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FB70F6F3000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "CE17E023542265FC11D9BC8F534BB4F070493D30" }, { "b" : "7FB70F329000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B5381A457906D279073822A5CEB24C4BFEF94DDB" }, { "b" : "7FB710B03000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5D7B6259552275A3C17BD4C3FD05F5A6BF40CAA5" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55688835d071]
 mongod(+0x2211289) [0x55688835c289]
 mongod(+0x221176D) [0x55688835c76d]
 libpthread.so.0(+0x11390) [0x7fb70f704390]
 libc.so.6(gsignal+0x38) [0x7fb70f35e428]
 libc.so.6(abort+0x16A) [0x7fb70f36002a]
 mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x0) [0x556886ad525c]
 mongod(_ZN5mongo34SessionCatalogMigrationDestination31_retrieveSessionStateFromSourceEPNS_14ServiceContextE+0x22A8) [0x55688717dca8]
 mongod(+0x2321170) [0x55688846c170]
 libpthread.so.0(+0x76BA) [0x7fb70f6fa6ba]
 libc.so.6(clone+0x6D) [0x7fb70f43041d]
-----  END BACKTRACE  -----
2018-03-19T10:19:47.831+0300 I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to sh0-rs0-0.ohm.pw:27021 - HostUnreachable: Connection refused
2018-03-19T10:19:47.831+0300 I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to sh0-rs0-0.ohm.pw:27021 due to failed operation on a connection
2018-03-19T10:19:47.831+0300 I REPL_HB  [replexec-1460] Error in heartbeat (requestId: 232658414) to sh0-rs0-0.ohm.pw:27021, response status: HostUnreachable: Connection refused
2018-03-19T10:19:47.831+0300 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to sh0-rs0-0.ohm.pw:27021