Uploaded image for project: 'C Driver'
  1. C Driver
  2. CDRIVER-4150

AWS ECS tasks failing setup

    • Type: Icon: Task Task
    • Resolution: Fixed
    • Priority: Icon: Unknown Unknown
    • 1.22.1
    • Affects Version/s: None
    • Component/s: None

      The test-aws-openssl-ecs-latest and test-aws-openssl-ecs-4.4 tasks are consistently failing to run the setup script aws_e2e_ecs.js.

      Here is an example failure with relevant logs:

      [2021/08/31 18:30:24.372] ===== Testing auth via ECS task metadata =====
      [2021/08/31 18:30:24.372] Makefile
      [2021/08/31 18:30:24.372] README.md
      [2021/08/31 18:30:24.372] action.yml
      [2021/08/31 18:30:24.372] evergreen_config_generator
      [2021/08/31 18:30:24.440] MongoDB shell version v5.1.0-alpha-691-gc96a409
      [2021/08/31 18:30:24.440] loading file: setup.js
      [2021/08/31 18:30:24.440] loading file: aws_e2e_ecs.js
      [2021/08/31 18:30:24.441] {"t":{"$date":"2021-08-31T18:30:24.441Z"},"s":"I",  "c":"-",        "id":22810,   "ctx":"js","msg":"shell: Started program","attr":{"pid":"8350","argv":["/bin/sh","-c","python3 -u  lib/container_tester.py -v remote_gc_services  --cluster arn:aws:ecs:us-east-1:557821124784:cluster/authtest-cluster"]}}
      [2021/08/31 18:30:25.393] sh8350| AWS_SHARED_CREDENTIALS_FILE: None
      [2021/08/31 18:30:25.393] sh8350| INFO:botocore.credentials:Found credentials in environment variables.
      [2021/08/31 18:30:25.804] sh8350| INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): ecs.us-east-1.amazonaws.com
      [2021/08/31 18:30:25.882] {"t":{"$date":"2021-08-31T18:30:25.882Z"},"s":"I",  "c":"-",        "id":22810,   "ctx":"js","msg":"shell: Started program","attr":{"pid":"8356","argv":["/bin/sh","-c","python3 -u  lib/container_tester.py -d -v run_e2e_test --cluster arn:aws:ecs:us-east-1:557821124784:cluster/authtest-cluster --task_definition arn:aws:ecs:us-east-1:557821124784:task-definition/app:2 --subnets subnet-6dd81326 --subnets subnet-50f3fe0a --security_group sg-09bacbe8dda5ccb7b --files /data/mci/33a57c2e3c4e7bb7bb053b284953a6bd/mongoc/mongodb/bin/mongod:/root/mongod /data/mci/33a57c2e3c4e7bb7bb053b284953a6bd/mongoc/mongodb/bin/mongo:/root/mongo  lib/ecs_hosted_test.js:/root/ecs_hosted_test.js /data/mci/33a57c2e3c4e7bb7bb053b284953a6bd/mongoc:/root --script lib/ecs_hosted_test.sh"]}}
      [2021/08/31 18:30:26.084] sh8356| AWS_SHARED_CREDENTIALS_FILE: None
      [2021/08/31 18:30:26.084] sh8356| DEBUG:botocore.credentials:Looking for credentials via: env
      [2021/08/31 18:30:26.085] sh8356| INFO:botocore.credentials:Found credentials in environment variables.
      [2021/08/31 18:30:26.085] sh8356| DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.6/dist-packages/botocore/data/endpoints.json
      [2021/08/31 18:30:26.090] sh8356| DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.6/dist-packages/botocore/data/ecs/2014-11-13/service-2.json
      [2021/08/31 18:30:26.092] sh8356| DEBUG:botocore.loaders:Loading JSON file: /usr/local/lib/python3.6/dist-packages/botocore/data/_retry.json
      [2021/08/31 18:30:26.094] sh8356| DEBUG:botocore.client:Registering retry handlers for service: ecs
      [2021/08/31 18:30:26.094] sh8356| DEBUG:botocore.hooks:Event creating-client-class.ecs: calling handler <function add_generate_presigned_url at 0x7f48f992e488>
      [2021/08/31 18:30:26.094] sh8356| DEBUG:botocore.args:The s3 config key is not a dictionary type, ignoring its value of: None
      [2021/08/31 18:30:26.097] sh8356| DEBUG:botocore.endpoint:Setting ecs timeout as (60, 60)
      [2021/08/31 18:30:26.097] sh8356| DEBUG:botocore.hooks:Event before-parameter-build.ecs.CreateService: calling handler <function generate_idempotent_uuid at 0x7f48f9865620>
      [2021/08/31 18:30:26.097] sh8356| Traceback (most recent call last):
      [2021/08/31 18:30:26.097] sh8356|   File "lib/container_tester.py", line 385, in <module>
      [2021/08/31 18:30:26.098] sh8356|     main()
      [2021/08/31 18:30:26.098] sh8356|   File "lib/container_tester.py", line 381, in main
      [2021/08/31 18:30:26.098] sh8356|     args.func(args)
      [2021/08/31 18:30:26.098] sh8356|   File "lib/container_tester.py", line 291, in _run_e2e_test_args
      [2021/08/31 18:30:26.098] sh8356|     _run_e2e_test(args.script, args.files, args.cluster, args.task_definition, args.subnets, args.security_group)
      [2021/08/31 18:30:26.098] sh8356|   File "lib/container_tester.py", line 303, in _run_e2e_test
      [2021/08/31 18:30:26.098] sh8356|     remote_create_container(cluster, task_definition, service_name, subnets, security_group)
      [2021/08/31 18:30:26.098] sh8356|   File "lib/container_tester.py", line 170, in remote_create_container
      [2021/08/31 18:30:26.098] sh8356|     'assignPublicIp': "ENABLED"
      [2021/08/31 18:30:26.098] sh8356|   File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 253, in _api_call
      [2021/08/31 18:30:26.185] sh8356|     return self._make_api_call(operation_name, kwargs)
      [2021/08/31 18:30:26.185] sh8356|   File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 531, in _make_api_call
      [2021/08/31 18:30:26.186] sh8356|     api_params, operation_model, context=request_context)
      [2021/08/31 18:30:26.186] sh8356|   File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 586, in _convert_to_request_dict
      [2021/08/31 18:30:26.186] sh8356|     api_params, operation_model)
      [2021/08/31 18:30:26.186] sh8356|   File "/usr/local/lib/python3.6/dist-packages/botocore/validate.py", line 291, in serialize_to_request
      [2021/08/31 18:30:26.186] sh8356|     raise ParamValidationError(report=report.generate_report())
      [2021/08/31 18:30:26.186] sh8356| botocore.exceptions.ParamValidationError: Parameter validation failed:
      [2021/08/31 18:30:26.186] sh8356| Unknown parameter in input: "launchType", must be one of: cluster, serviceName, taskDefinition, loadBalancers, desiredCount, clientToken, role, deploymentConfiguration, placementConstraints, placementStrategy
      [2021/08/31 18:30:26.186] sh8356| Unknown parameter in input: "networkConfiguration", must be one of: cluster, serviceName, taskDefinition, loadBalancers, desiredCount, clientToken, role, deploymentConfiguration, placementConstraints, placementStrategy
      [2021/08/31 18:30:26.214] uncaught exception: Error: [1] != [0] are not equal : Container Test failed :
      [2021/08/31 18:30:26.214] doassert@src/mongo/shell/assert.js:20:14
      [2021/08/31 18:30:26.214] assert.eq@src/mongo/shell/assert.js:179:9
      [2021/08/31 18:30:26.214] @aws_e2e_ecs.js:43:4
      [2021/08/31 18:30:26.214] @aws_e2e_ecs.js:6:2
      [2021/08/31 18:30:26.220] failed to load: aws_e2e_ecs.js
      [2021/08/31 18:30:26.220] exiting with code -3
      [2021/08/31 18:30:26.220] Command failed: command encountered problem: error waiting on process '025152fe-bd6e-446c-b9e9-767787eb67bc': exit status 253
      [2021/08/31 18:30:26.220] Task completed - FAILURE.
      

            Assignee:
            kevin.albertson@mongodb.com Kevin Albertson
            Reporter:
            kevin.albertson@mongodb.com Kevin Albertson
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: