-
Type: Improvement
-
Resolution: Fixed
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: Testing Infrastructure
-
Fully Compatible
-
DAG 2021-05-03
-
2
In the current server build most generated tasks, if not all, depend on the compile task completing so that their artifacts are available. However, those tasks are considered started immediately after the generation phase. This is a little perplexing visually, where all of the test suites that contain generated tasks appear to be running immediately when a patch build starts. That is a somewhat minor UX concern. The more significant consequence is that the wall clock time for the task is rendered inaccurate, because the clock starts when compile starts, not when it ends.
See, for instance:
- https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_compile_ea816a71cf3f3c61e42c6a09195998eaf3bffe22_20_06_05_22_14_43
- https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_display_sharded_causally_consistent_jscore_txns_passthrough_ea816a71cf3f3c61e42c6a09195998eaf3bffe22_20_06_05_22_14_43
The sharded_causally_consistent_jscore_txns_passthrough suite is shown has having a wall clock time of 29 minutes. However, this is inaccurate, because the clock started at the same time as compile, at 06:41. Had the clock started when tests actually began to execute, the wall clock time would I think have been something more like 10 minutes.
This double counting of time makes it difficult to accurately A/B compare performance of builds because changes in the duration of the compile task are accounted for multiple times.
In the absence of EVG-2535 there doesn't seem to be a way to model the difference without collecting and parsing logs for every task. Addressing one, or ideally both, of these issues would provide much more accurate tools with which to understand build performance.