-
Type: Task
-
Resolution: Fixed
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: None
-
Fully Compatible
-
STM 2022-02-21
-
2
Use the Evergreen API to get the runtime and list of test that had run in burn-in, if any test is taking more than 1 standard deviation more than its historical average over the past 20 runs (or days, whichever is easier), fail the task with a helpful error message
Determine if this approach is sufficient for reducing the runtime or if TIPS tooling is needed for things like change points.
Design sketch:
- After burn-in calls resmoke to run the tests, get the report object from resmoke and get the average timing of the tests
- Call the Evergreen API to get the test timing for the past 20 days or runs
- Raise exception if the average test time in burn in is greater than 1 stddev above the past average.