-
Type: Task
-
Resolution: Duplicate
-
Priority: Unknown
-
None
-
Affects Version/s: None
-
Component/s: Performance
-
None
We have some cross-language driver benchmarks run by Evergreen, but there is no automatic detection of regressions, and our release process does not include analyzing manually whether the results are fine or not.
Checking the results of benchmarks before releasing a new version, and making a decision whether it is acceptable to release based on the results may help us to avoid regressions like JAVA-4452. When done, adding more benchmarks may further increase our chances of spotting a regression before releasing a version.
The relevant Evergreen docs:
- perf.send project command
- performance plugins
- performance tooling (likely, the most useful link)
- a new performance discovery tool from performance tooling, that ironically discovers benchmark regressions (see the "Performance Test Results" tab) when comparing 4.5 against 4.4 But that is because the existing benchmarks are not able to feel the degradation1 in
JAVA-4452which was fixed in 4.5.
1 It does not seem like the existing driver benchmarks show the JAVA-4452 regression. There is no data before May 18, but based on the Jira tickets that caused JAVA-4452, the commits to master happened on Apr 14, May 1, Jun 8, Sep 27, and I don't see charts having a stair step shape near those dates.