Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-39564

burn_in_tests.py doesn't run tests in the configuration for *_gen Evergreen tasks

    • Fully Compatible
    • ALL
    • v4.0
    • Hide

      I applied the following patch to track down where in the burn_in_tests.py script things were going wrong.

      $ git checkout 8e5e745e98d33633e7d24a2629f22cdba79d9851
      $ python buildscripts/resmoke.py --findSuites jstests/auth/getMore.js
      [resmoke] 2019-02-12T23:51:37.998-0500 jstests/auth/getMore.js will be run by the following suite(s): ['auth', 'auth_audit']
      $ python buildscripts/burn_in_tests.py --branch=master --buildVariant=enterprise-rhel-62-64-bit --testListOutfile=jstests/new_tests.json --noExec --checkEvergreen
      Comparing current branch against 7a7baa2539ec169335086e45c7d0b85ba7cdb877
      @@@ changed_tests ['jstests/auth/getMore.js']
      @@@ tests_by_executor defaultdict(<type 'list'>, {'auth_audit': ['jstests/auth/getMore.js'], 'auth': ['jstests/auth/getMore.js']})
      @@@ skipping task compile
      ...
      @@@ skipping task auth_gen
      @@@ skipping task auth_audit_gen
      ...
      @@@ tests_by_task {}
      
      Unable to find source-code formatter for language: diff. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      diff --git a/buildscripts/burn_in_tests.py b/buildscripts/burn_in_tests.py
      index cae3ee9bac..0bb0aae000 100644
      --- a/buildscripts/burn_in_tests.py
      +++ b/buildscripts/burn_in_tests.py
      @@ -146,7 +146,7 @@ def find_changed_tests(branch_name, base_commit, max_revisions, buildvariant, ch
               # activated.
               revs_to_check = callo(["git", "rev-list", base_commit, "--max-count=200",
                                      "--skip=1"]).splitlines()
      -        last_activated = find_last_activated_task(revs_to_check, buildvariant, branch_name)
      +        last_activated = "7a7baa2539ec169335086e45c7d0b85ba7cdb877"
               if last_activated is None:
                   # When the current commit is the first time 'buildvariant' has run, there won't be a
                   # commit among 'revs_to_check' that's been activated in Evergreen. We handle this by
      @@ -282,6 +282,8 @@ def create_task_list(evergreen_conf, buildvariant, suites, exclude_tasks):
                   resmoke_args = task.combined_resmoke_args
                   if resmoke_args:
                       variant_task_args[task.name] = resmoke_args
      +            else:
      +                print '@@@ skipping task', task.name
      
           # Create the list of tasks to run for the specified suite.
           tasks_to_run = {}
      @@ -363,6 +365,7 @@ def main():
                                                  values.buildvariant, values.check_evergreen)
               exclude_suites, exclude_tasks, exclude_tests = find_exclude_tests(values.selector_file)
               changed_tests = filter_tests(changed_tests, exclude_tests)
      +        print '@@@ changed_tests', changed_tests
               # If there are no changed tests, exit cleanly.
               if not changed_tests:
                   print "No new or modified tests found."
      @@ -372,8 +375,10 @@ def main():
               suites = resmokelib.suitesconfig.get_suites(
                   suite_files=values.suite_files.split(","), test_files=changed_tests)
               tests_by_executor = create_executor_list(suites, exclude_suites)
      +        print '@@@ tests_by_executor', tests_by_executor
               tests_by_task = create_task_list(evergreen_conf, values.buildvariant, tests_by_executor,
                                                exclude_tasks)
      +        print '@@@ tests_by_task', tests_by_task
               if values.test_list_outfile is not None:
                   _write_report_file(tests_by_task, values.test_list_outfile)
      
      
      Show
      I applied the following patch to track down where in the burn_in_tests.py script things were going wrong. $ git checkout 8e5e745e98d33633e7d24a2629f22cdba79d9851 $ python buildscripts/resmoke.py --findSuites jstests/auth/getMore.js [resmoke] 2019-02-12T23:51:37.998-0500 jstests/auth/getMore.js will be run by the following suite(s): ['auth', 'auth_audit'] $ python buildscripts/burn_in_tests.py --branch=master --buildVariant=enterprise-rhel-62-64-bit --testListOutfile=jstests/new_tests.json --noExec --checkEvergreen Comparing current branch against 7a7baa2539ec169335086e45c7d0b85ba7cdb877 @@@ changed_tests ['jstests/auth/getMore.js'] @@@ tests_by_executor defaultdict(<type 'list'>, {'auth_audit': ['jstests/auth/getMore.js'], 'auth': ['jstests/auth/getMore.js']}) @@@ skipping task compile ... @@@ skipping task auth_gen @@@ skipping task auth_audit_gen ... @@@ tests_by_task {} Unable to find source-code formatter for language: diff. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml diff --git a/buildscripts/burn_in_tests.py b/buildscripts/burn_in_tests.py index cae3ee9bac..0bb0aae000 100644 --- a/buildscripts/burn_in_tests.py +++ b/buildscripts/burn_in_tests.py @@ -146,7 +146,7 @@ def find_changed_tests(branch_name, base_commit, max_revisions, buildvariant, ch # activated. revs_to_check = callo([ "git" , "rev-list" , base_commit, "--max-count=200" , "--skip=1" ]).splitlines() - last_activated = find_last_activated_task(revs_to_check, buildvariant, branch_name) + last_activated = "7a7baa2539ec169335086e45c7d0b85ba7cdb877" if last_activated is None: # When the current commit is the first time 'buildvariant' has run, there won't be a # commit among 'revs_to_check' that's been activated in Evergreen. We handle this by @@ -282,6 +282,8 @@ def create_task_list(evergreen_conf, buildvariant, suites, exclude_tasks): resmoke_args = task.combined_resmoke_args if resmoke_args: variant_task_args[task.name] = resmoke_args + else : + print '@@@ skipping task' , task.name # Create the list of tasks to run for the specified suite. tasks_to_run = {} @@ -363,6 +365,7 @@ def main(): values.buildvariant, values.check_evergreen) exclude_suites, exclude_tasks, exclude_tests = find_exclude_tests(values.selector_file) changed_tests = filter_tests(changed_tests, exclude_tests) + print '@@@ changed_tests' , changed_tests # If there are no changed tests, exit cleanly. if not changed_tests: print "No new or modified tests found." @@ -372,8 +375,10 @@ def main(): suites = resmokelib.suitesconfig.get_suites( suite_files=values.suite_files.split( "," ), test_files=changed_tests) tests_by_executor = create_executor_list(suites, exclude_suites) + print '@@@ tests_by_executor' , tests_by_executor tests_by_task = create_task_list(evergreen_conf, values.buildvariant, tests_by_executor, exclude_tasks) + print '@@@ tests_by_task' , tests_by_task if values.test_list_outfile is not None: _write_report_file(tests_by_task, values.test_list_outfile)
    • DAG 2019-02-25

      I noticed this upon reviewing my patch build for backporting SERVER-35551 to the 4.0 branch. I then took a look at the master branch and found this task had run tests via burn_in_tests.py but this other task didn't run tests via burn_in_tests.py despite the jstests/auth/getMore.js test being modified.

      The issue stems from how the task definitions for the *_gen Evergreen tasks have a different style compared to before and leads to task.combined_resmoke_args being None and for them to be skipped over in the tests_by_task mapping.

      - name: auth_gen
        commands: 
        - func: "generate resmoke tasks"
          vars: 
            task: auth
            resmoke_args: --storageEngine=wiredTiger
            fallback_num_sub_suites: 4
      
      - <<: *task_template
        name: auth
        commands: 
        - func: "do setup"
        - func: "run tests"
          vars: 
            resmoke_args: --suites=auth --storageEngine=wiredTiger
      

            Assignee:
            david.bradford@mongodb.com David Bradford (Inactive)
            Reporter:
            max.hirschhorn@mongodb.com Max Hirschhorn
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: