Summary

All drivers must audit and update their CI configs for usages of vars in function scopes by October 6 at which point the changes in EVG-20276 will take effect causing vars defined in function scopes to no longer be retained in subsequent function executions.

Motivation

Who is the affected end user?

Drivers with evergreen configs that use vars in function scopes.

How does this affect the end user?

Drivers that have been using the evergreen var persistence as a feature may see CI failures OR experience silent behavior changes (e.g., skipped tests, unintended propagation of default values in builds) if they do not fix their usage before the rollout of the fix.

How likely is it that this problem or use case will occur?

Persistence of vars across function scopes has been an unintended feature of evergreen since the beginning of time. This change potentially affects any CI task that defines more than one `func:` where at least one of the functions specifies a `vars:` attribute.

If the problem does occur, what are the consequences and how severe are they?

Anything from easy to fix build failures to misconfigured build artifacts deployed to production.

• The following are the possible usage scenarios of a given env variable defined in an earlier `func: vars:` by a subsequent `func:` execution
  1. The env variable is not meaningful for the subsequent execution
     • No impact from this change
  2. The env variable is meaningful for the subsequent execution but its value is set explicitly inside that execution, ignoring any previously set values
     • No impact from this change
  3. The env variable is meaningful for the subsequent execution and is meant to be set outside the current and earlier function execution (either via proper use of expansions or intentional fallback to defaults)
     • The change fixes what is currently a red build OR a silent failure/unintentional build behavior; possibly no effect if the intended value coincides with the earlier function value
  4. The env variable is meaningful for the subsequent execution and is meant to be set in the earlier function execution, erroring if not set
     • This change results in CI failure because the bug was being used as a feature, but is straightforward to fix (via proper use of expansions or modifying the script to set the env variable where it is needed)
  5. The env variable is meaningful for the subsequent execution and is meant to be set in the earlier function execution, but has fallback logic to defaults OR expansion values defined outside the current or previous function in the config 
     • This change results in silent build behavior alterations (unless the default values coincide with the intended values) because the bug was being used as a feature; alterations that can only be caught via an audit of the task config in combination with the corresponding executed code; the worst case scenario may include any range of potential effects such as:
       • Loss of test coverage (either by disabling tests entirely or by running them against a default config/settings instead of the intended config/settings)
       • Failure to clean up resources (by failing to correctly specify the cleanup parameters)
       • Deployment of production systems, including docker images or build artifacts intended for use by customers, with incorrect default settings or test values baked in (e.g., placeholder text, log verbosity settings, timeout or performance-related settings, undefined endpoints, etc)

Is this issue urgent?

Yes, must be completed by October 6th (unless EVG-20276 deployment timeline is pushed back).

Is this ticket required by a downstream team?

No.

Is this ticket only for tests?

This is a CI-only ticket.

Acceptance Criteria

• Drivers should perform the steps provided by the evergreen team to audit their configs and test the change ahead of its deployment. Specifically, check each task that defines more than one `func:`. If `vars:` are set in any `func:` other than the last one, check the subsequent `func:` definitions/scripts for reliance on `vars:` defined in the preceding functions.
• It is NOT sufficient to look for failures after enabling the feature flag for the fix as the variable could have a different value defined outside the function scope that causes a change in behavior and not an outright build failure.

The following is copied from the relevant comment on EVG-20276:

As promised, we’ve introduced a way to test unsetting function vars before rolling it out for everyone. To do so, please set unset_function_vars: true at the top of your project config (example). 

For those who wish to audit their project configs in advance, here are some tips: 

• After setting unset_function_vars to true, run all tasks in the project and look out for anything that passed on main but is now failing, or anything that seems to be behaving in an unexpected way. If the test does not behave as expected, it is likely indicative of the task relying on a function var outside of that function’s. As an example, this project config was run with and without setting unset_function_vars.When unset_functin_vars is set to true, the value of foo is undefined when referenced outside of the function scope. However, when unset_function_vars isn’t set, which represents the current behavior, the value of foo is defined outside of the function scope. 
• If you notice an instance of a function var being used outside of the function scope in your project, consider using an expansion instead to set the key value pair so that they are available either on a build variant or project wide level. If you need the value to be set during runtime, you can use expansions.update. 
• If a function var and an expansion share a key name, tests can fall back on the expansion value instead of encountering an undefined variable. 
• I’d  recommend looking for any function vars that share a key name with expansions, and making them unique before testing. 
• If there are any other fallbacks explicitly programmed in any scripts run by tasks for when a function var is not defined, remove them when testing. 

If anyone wishes to preserve the current behavior, either because it’s desired for a use case or because more time is needed to fix it, the old behavior can be preserved by simply calling expansions.update at the end of every function with the key value pairs of the vars you want to persist. 

For example, this project config has foo defined as a function var and then calls expansions.update to save the current function value ${foo} as an expansion. As you can see by the echos in this example, despite unset_function_vars being set to true, it behaved in a manner consistent with the current behavior and the function vars continued to be defined after the function exited. The output is the same as in this patch where unset_function_vars was not set. 

If you have any questions, please reach out to us in #evergreen-users.