Uploaded image for project: 'Drivers'
  1. Drivers
  2. DRIVERS-2012

Mark "max retries exceeded" as atlas failure

    • Type: Icon: Task Task
    • Resolution: Fixed
    • Priority: Icon: Unknown Unknown
    • None
    • Component/s: Atlas Testing
    • None

      This exception is an intermittent atlas problem and should be marked as such:

      1/12/17 05:14:49.612] INFO:astrolabe.runner:Cluster 96146eba85: current state: creating; wanted state: idle; waited for 643.2 sec
       [2021/12/17 05:14:50.649] INFO:astrolabe.runner:Cluster 96146eba85: current state: creating; wanted state: idle; waited for 644.3 sec
       [2021/12/17 05:14:51.691] INFO:astrolabe.runner:Cluster 96146eba85: current state: creating; wanted state: idle; waited for 645.3 sec
       [2021/12/17 05:14:52.729] INFO:astrolabe.runner:Cluster 96146eba85: current state: creating; wanted state: idle; waited for 646.3 sec
       [2021/12/17 05:14:53.774] INFO:astrolabe.runner:Cluster 96146eba85: current state: creating; wanted state: idle; waited for 647.4 sec
       [2021/12/17 05:14:54.840] INFO:astrolabe.runner:Cluster 96146eba85: current state: creating; wanted state: idle; waited for 648.5 sec
       [2021/12/17 05:14:55.881] INFO:astrolabe.runner:Cluster 96146eba85: current state: creating; wanted state: idle; waited for 649.5 sec
       [2021/12/17 05:14:56.921] INFO:astrolabe.runner:Cluster 96146eba85: current state: creating; wanted state: idle; waited for 650.5 sec
       [2021/12/17 05:14:57.967] INFO:astrolabe.runner:Cluster 96146eba85: current state: creating; wanted state: idle; waited for 651.6 sec
       [2021/12/17 05:14:59.015] INFO:astrolabe.runner:Cluster 96146eba85: current state: creating; wanted state: idle; waited for 652.6 sec
       [2021/12/17 05:15:02.889] Traceback (most recent call last):
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/urllib3/connection.py", line 175, in _new_conn
       [2021/12/17 05:15:02.889]     (self._dns_host, self.port), self.timeout, **extra_kw
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/urllib3/util/connection.py", line 96, in create_connection
       [2021/12/17 05:15:02.889]     raise err
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/urllib3/util/connection.py", line 86, in create_connection
       [2021/12/17 05:15:02.889]     sock.connect(sa)
       [2021/12/17 05:15:02.889] OSError: [Errno 101] Network is unreachable
       [2021/12/17 05:15:02.889] During handling of the above exception, another exception occurred:
       [2021/12/17 05:15:02.889] Traceback (most recent call last):
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
       [2021/12/17 05:15:02.889]     chunked=chunked,
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 382, in _make_request
       [2021/12/17 05:15:02.889]     self._validate_conn(conn)
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
       [2021/12/17 05:15:02.889]     conn.connect()
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/urllib3/connection.py", line 358, in connect
       [2021/12/17 05:15:02.889]     conn = self._new_conn()
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/urllib3/connection.py", line 187, in _new_conn
       [2021/12/17 05:15:02.889]     self, "Failed to establish a new connection: %s" % e
       [2021/12/17 05:15:02.889] urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fd827221eb8>: Failed to establish a new connection: [Errno 101] Network is unreachable
       [2021/12/17 05:15:02.889] During handling of the above exception, another exception occurred:
       [2021/12/17 05:15:02.889] Traceback (most recent call last):
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
       [2021/12/17 05:15:02.889]     timeout=timeout
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
       [2021/12/17 05:15:02.889]     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
       [2021/12/17 05:15:02.889]     raise MaxRetryError(_pool, url, error or ResponseError(cause))
       [2021/12/17 05:15:02.889] urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='cloud-qa.mongodb.com', port=443): Max retries exceeded with url: /api/atlas/v1.0/groups/605cd3edefdd5e52ab52bd92/clusters/96146eba85 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fd827221eb8>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
       [2021/12/17 05:15:02.889] During handling of the above exception, another exception occurred:
       [2021/12/17 05:15:02.889] Traceback (most recent call last):
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/atlasclient/client.py", line 213, in request
       [2021/12/17 05:15:02.889]     response = requests.request(method, url, **request_kwargs)
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/requests/api.py", line 61, in request
       [2021/12/17 05:15:02.889]     return session.request(method=method, url=url, **kwargs)
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
       [2021/12/17 05:15:02.889]     resp = self.send(prep, **send_kwargs)
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
       [2021/12/17 05:15:02.889]     r = adapter.send(request, **kwargs)
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
       [2021/12/17 05:15:02.889]     raise ConnectionError(e, request=request)
       [2021/12/17 05:15:02.889] requests.exceptions.ConnectionError: HTTPSConnectionPool(host='cloud-qa.mongodb.com', port=443): Max retries exceeded with url: /api/atlas/v1.0/groups/605cd3edefdd5e52ab52bd92/clusters/96146eba85 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fd827221eb8>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
       [2021/12/17 05:15:02.889] During handling of the above exception, another exception occurred:
       [2021/12/17 05:15:02.889] Traceback (most recent call last):
       [2021/12/17 05:15:02.889]   File "astrolabevenv/bin/astrolabe", line 33, in <module>
       [2021/12/17 05:15:02.889]     sys.exit(load_entry_point('astrolabe', 'console_scripts', 'astrolabe')())
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/click/core.py", line 829, in __call__
       [2021/12/17 05:15:02.889]     return self.main(*args, **kwargs)
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/click/core.py", line 782, in main
       [2021/12/17 05:15:02.889]     rv = self.invoke(ctx)
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
       [2021/12/17 05:15:02.889]     return _process_result(sub_ctx.command.invoke(sub_ctx))
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
       [2021/12/17 05:15:02.889]     return _process_result(sub_ctx.command.invoke(sub_ctx))
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
       [2021/12/17 05:15:02.889]     return ctx.invoke(self.callback, **ctx.params)
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/click/core.py", line 610, in invoke
       [2021/12/17 05:15:02.889]     return callback(*args, **kwargs)
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabevenv/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
       [2021/12/17 05:15:02.889]     return f(get_current_context(), *args, **kwargs)
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabe/cli.py", line 453, in run_single_test
       [2021/12/17 05:15:02.889]     failed = runner.run()
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabe/runner.py", line 408, in run
       [2021/12/17 05:15:02.889]     return self.do_run()
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabe/runner.py", line 428, in do_run
       [2021/12/17 05:15:02.889]     active_case.wait_for_idle()
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/astrolabe/runner.py", line 315, in wait_for_idle
       [2021/12/17 05:15:02.889]     cluster_info = self.cluster_url.get().data
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/atlasclient/client.py", line 56, in get
       [2021/12/17 05:15:02.889]     return self._client.request('GET', self._path, **params)
       [2021/12/17 05:15:02.889]   File "/data/mci/8fd35e75b2490c44d971227a8cca21df/astrolabe-src/atlasclient/client.py", line 218, in request
       [2021/12/17 05:15:02.889]     request_method=method
       [2021/12/17 05:15:02.889] atlasclient.exceptions.AtlasClientError: HTTPSConnectionPool(host='cloud-qa.mongodb.com', port=443): Max retries exceeded with url: /api/atlas/v1.0/groups/605cd3edefdd5e52ab52bd92/clusters/96146eba85 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fd827221eb8>: Failed to establish a new connection: [Errno 101] Network is unreachable',)) (GET https://cloud-qa.mongodb.com/api/atlas/v1.0/groups/605cd3edefdd5e52ab52bd92/clusters/96146eba85)
       [2021/12/17 05:15:02.890] Finished 'subprocess.exec' in "run test" in 11m2.487427182s
      

      HTTP error could also be a genuine astrolabe problem, therefore only the above message should be matched and only in that case should the build be marked purple.

            Assignee:
            oleg.pudeyev@mongodb.com Oleg Pudeyev (Inactive)
            Reporter:
            oleg.pudeyev@mongodb.com Oleg Pudeyev (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: