-
Type: Epic
-
Resolution: Unresolved
-
Priority: Unknown
-
Affects Version/s: None
-
Component/s: None
-
Python Drivers
-
To Do
-
Improve PyMongo performance
-
0
-
0
-
0
-
100
Context
When running a single-threaded benchmark ( bench-command.py ), PyMongo master is 15-20% slower when compared to PyMongo 3.6.1:
$ python bench-command.py 1 Python: 3.9.13, PyMongo: 4.8.0.dev0, MongoDB: 5.0.15, PID: 99424 Running 10000 operations across 1 thread(s) command: 1.33s find_one: 1.95s insert_one: 1.88s
vs 3.6.1:
$ python bench-command.py 1 Python: 3.9.13, PyMongo: 3.6.1, MongoDB: 5.0.15, PID: 98867 Running 10000 operations across 1 thread(s) command: 1.11s find_one: 1.66s insert_one: 1.66s
When running a 100-thread benchmark, PyMongo master is 40-50% slower when compared to PyMongo 3.6.1:
$ python bench-command.py 100 Python: 3.9.13, PyMongo: 4.8.0.dev0, MongoDB: 5.0.15 Running 10000 operations across 100 thread(s) command: 1.65s find_one: 1.76s insert_one: 1.53s
vs 3.6.1:
$ python bench-command.py 100 Python: 3.9.13, PyMongo: 3.6.1, MongoDB: 5.0.15 Running 10000 operations across 100 thread(s) command: 1.12s find_one: 1.20s insert_one: 1.10s
Here's another representation of the benchmark results, this time with a few more PyMongo versions:
1 thread:
PyMongo version | command | find_one | insert_one |
---|---|---|---|
3.5.1 | 1.02 | 1.56 | 1.47 |
3.6.1 | 1.11 | 1.66 | 1.66 |
3.7.2 | 1.15 | 1.68 | 1.70 |
3.13.0 | 1.28 | 1.89 | 1.83 |
4.0.2 | 1.25 | 1.85 | 1.79 |
4.7.3 | 1.27 | 1.90 | 1.91 |
master | 1.33 | 1.95 | 1.88 |
100 threads:
PyMongo version | command | find_one | insert_one |
---|---|---|---|
3.5.1 | 0.70 | 0.79 | 0.69 |
3.6.1 | 1.12 | 1.20 | 1.10 |
3.7.2 | 1.15 | 1.27 | 1.21 |
3.13.0 | 1.29 | 1.49 | 1.40 |
4.0.2 | 1.36 | 1.54 | 1.40 |
4.7.3 | 1.39 | 1.64 | 1.55 |
master | 1.65 | 1.76 | 1.53 |
Note these tests were run locally on macos where master was commit d6bf0e1e78c3287149ef56d9718bb6098aa6e41f (PYTHON-4264).
Definition of done
Investigate the performance regressions and bottlenecks in the the driver and identify potential optimizations.
We should be able to improve both single-threaded and multi-threaded performance. One goal is that the 100-thread benchmark should complete faster than the single threaded version just as it does in PyMongo 3.5.1.
Pitfalls
It could be impossible to match 3.5.1 performance because many features have been added since then which we can't remove (sessions, retryable reads/writes, CSOT, maxConnecting, logging, OP_MSG, etc...) but we should be able to get closer the performance of 3.7 (or ideally 3.5).
- related to
-
PYTHON-4347 Optimize MongoClient._get_topology
- Closed
-
PYTHON-4473 Optimize find/aggregate/command by avoiding duplicate calls to _get_topology
- Closed