Summary

Out current benchmark and alerting system is noisy and results in a lot of false positives. For example many of our benchmarks have a run to run variance of 10% as described in PYTHON-4498. This makes it hard to identify true performance changes, both positive and negative. It also makes it hard to validate small performance improvements.

We should dedicate some time to make benchmarks less noisy.

Motivation

Who is the affected end user?

The python team.

If the problem does occur, what are the consequences and how severe are they?

Time spent triaging alerts for false positives.

Is this ticket required by a downstream team?

No

Is this ticket only for tests?

Test only. But more stable benchmarks can make it easier to identify performance changes and improve pymongo.

Cast of Characters

Engineering Lead:
Document Author:
POCers:
Product Owner:
Program Manager:
Stakeholders:

Channels & Docs

Slack Channel

[Scope Document|some.url]

[Technical Design Document|some.url]