Uploaded image for project: 'Drivers'
  1. Drivers
  2. DRIVERS-1947

Kill workload executor when astrolabe exits with an error

XMLWordPrintableJSON

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Component/s: Atlas Testing
    • None

      Currently when astrolabe exits with an error it does not kill the workload executor, if one is running. This causes two types of issues:

      • On a local machine, the workload executor continues running in the background, potentially spamming terminal with warnings/errors if it cannot connect to cluster or perform an operation
      • In evergreen, the test run gets timed out because evergreen uses go and go waits for all processes in a spawned process tree to exit, not just the spawned process itself (https://github.com/golang/go/issues/20730)
      [2021/10/08 17:22:12.455] W, [2021-10-08T17:22:12.200850 #22238]  WARN -- : MONGODB | Error running awaited hello on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017: Mongo::Error::SocketError: Errno::ECONNREFUSED: Connection refused - connect(2) for 54.176.135.219:27017 (for 54.176.135.219:27017 (2e694dd263-shard-00-02.80104.mongodb-qa.net:27017, TLS)) (on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017)
[2021/10/08 17:22:12.455] W, [2021-10-08T17:22:12.205861 #22238]  WARN -- : MONGODB | Error running awaited hello on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017: Mongo::Error::SocketError: Errno::ECONNREFUSED: Connection refused - connect(2) for 54.176.135.219:27017 (for 54.176.135.219:27017 (2e694dd263-shard-00-02.80104.mongodb-qa.net:27017, TLS)) (on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017)
[2021/10/08 17:22:12.455] W, [2021-10-08T17:22:12.263242 #22238]  WARN -- : MONGODB | Error running awaited hello on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017: Mongo::Error::SocketError: Errno::ECONNREFUSED: Connection refused - connect(2) for 54.176.135.219:27017 (for 54.176.135.219:27017 (2e694dd263-shard-00-02.80104.mongodb-qa.net:27017, TLS)) (on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017)
[2021/10/08 17:22:12.455] W, [2021-10-08T17:22:12.268822 #22238]  WARN -- : MONGODB | Error running awaited hello on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017: Mongo::Error::SocketError: Errno::ECONNREFUSED: Connection refused - connect(2) for 54.176.135.219:27017 (for 54.176.135.219:27017 (2e694dd263-shard-00-02.80104.mongodb-qa.net:27017, TLS)) (on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017)
[2021/10/08 17:22:12.455] W, [2021-10-08T17:22:12.326601 #22238]  WARN -- : MONGODB | Error running awaited hello on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017: Mongo::Error::SocketError: Errno::ECONNREFUSED: Connection refused - connect(2) for 54.176.135.219:27017 (for 54.176.135.219:27017 (2e694dd263-shard-00-02.80104.mongodb-qa.net:27017, TLS)) (on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017)
[2021/10/08 17:22:12.455] W, [2021-10-08T17:22:12.330875 #22238]  WARN -- : MONGODB | Error running awaited hello on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017: Mongo::Error::SocketError: Errno::ECONNREFUSED: Connection refused - connect(2) for 54.176.135.219:27017 (for 54.176.135.219:27017 (2e694dd263-shard-00-02.80104.mongodb-qa.net:27017, TLS)) (on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017)
[2021/10/08 17:22:12.455] W, [2021-10-08T17:22:12.388785 #22238]  WARN -- : MONGODB | Error running awaited hello on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017: Mongo::Error::SocketError: Errno::ECONNREFUSED: Connection refused - connect(2) for 54.176.135.219:27017 (for 54.176.135.219:27017 (2e694dd263-shard-00-02.80104.mongodb-qa.net:27017, TLS)) (on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017)
[2021/10/08 17:22:12.455] W, [2021-10-08T17:22:12.393696 #22238]  WARN -- : MONGODB | Error running awaited hello on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017: Mongo::Error::SocketError: Errno::ECONNREFUSED: Connection refused - connect(2) for 54.176.135.219:27017 (for 54.176.135.219:27017 (2e694dd263-shard-00-02.80104.mongodb-qa.net:27017, TLS)) (on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017)
[2021/10/08 17:22:12.455] W, [2021-10-08T17:22:12.452531 #22238]  WARN -- : MONGODB | Error running awaited hello on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017: Mongo::Error::SocketError: Errno::ECONNREFUSED: Connection refused - connect(2) for 54.176.135.219:27017 (for 54.176.135.219:27017 (2e694dd263-shard-00-02.80104.mongodb-qa.net:27017, TLS)) (on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017)
[2021/10/08 17:22:12.455] W, [2021-10-08T17:22:12.455549 #22238]  WARN -- : MONGODB | Error running awaited hello on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017: Mongo::Error::SocketError: Errno::ECONNREFUSED: Connection refused - connect(2) for 54.176.135.219:27017 (for 54.176.135.219:27017 (2e694dd263-shard-00-02.80104.mongodb-qa.net:27017, TLS)) (on 2e694dd263-shard-00-02.80104.mongodb-qa.net:27017)
[2021/10/08 19:03:26.112] Command stopped early: context canceled
[2021/10/08 19:03:26.166] Running task-timeout commands.

      To avoid these issues astrolabe should attempt to terminate the workload executor if it launched one and astrolabe is exiting with an error.

            Assignee:
            oleg.pudeyev@mongodb.com Oleg Pudeyev (Inactive)
            Reporter:
            oleg.pudeyev@mongodb.com Oleg Pudeyev (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: