Uploaded image for project: 'Python Driver'
  1. Python Driver
  2. PYTHON-4481

Improve PyMongo performance

    • Type: Icon: Epic Epic
    • Resolution: Unresolved
    • Priority: Icon: Unknown Unknown
    • 4.11
    • Affects Version/s: None
    • Component/s: None
    • Python Drivers
    • To Do
    • Improve PyMongo performance
    • 0
    • 0
    • 0
    • 100

      Context

      When running a single-threaded benchmark ( bench-command.py ), PyMongo master is 15-20% slower when compared to PyMongo 3.6.1:

      $ python bench-command.py 1
      Python: 3.9.13, PyMongo: 4.8.0.dev0, MongoDB: 5.0.15, PID: 99424
      Running 10000 operations across 1 thread(s)
      command: 1.33s
      find_one: 1.95s
      insert_one: 1.88s
      

      vs 3.6.1:

      $ python bench-command.py 1  
      Python: 3.9.13, PyMongo: 3.6.1, MongoDB: 5.0.15, PID: 98867
      Running 10000 operations across 1 thread(s)
      command: 1.11s
      find_one: 1.66s
      insert_one: 1.66s
      

      When running a 100-thread benchmark, PyMongo master is 40-50% slower when compared to PyMongo 3.6.1:

      $ python bench-command.py 100
      Python: 3.9.13, PyMongo: 4.8.0.dev0, MongoDB: 5.0.15
      Running 10000 operations across 100 thread(s)
      command: 1.65s
      find_one: 1.76s
      insert_one: 1.53s
      

      vs 3.6.1:

      $ python bench-command.py 100
      Python: 3.9.13, PyMongo: 3.6.1, MongoDB: 5.0.15
      Running 10000 operations across 100 thread(s)
      command: 1.12s
      find_one: 1.20s
      insert_one: 1.10s
      

      Here's another representation of the benchmark results, this time with a few more PyMongo versions:
      1 thread:

      PyMongo version command find_one insert_one
      3.5.1 1.02 1.56 1.47
      3.6.1 1.11 1.66 1.66
      3.7.2 1.15 1.68 1.70
      3.13.0 1.28 1.89 1.83
      4.0.2 1.25 1.85 1.79
      4.7.3 1.27 1.90 1.91
      master 1.33 1.95 1.88

      100 threads:

      PyMongo version command find_one insert_one
      3.5.1 0.70 0.79 0.69
      3.6.1 1.12 1.20 1.10
      3.7.2 1.15 1.27 1.21
      3.13.0 1.29 1.49 1.40
      4.0.2 1.36 1.54 1.40
      4.7.3 1.39 1.64 1.55
      master 1.65 1.76 1.53

      Note these tests were run locally on macos where master was commit d6bf0e1e78c3287149ef56d9718bb6098aa6e41f (PYTHON-4264).

      Definition of done

      Investigate the performance regressions and bottlenecks in the the driver and identify potential optimizations.

      We should be able to improve both single-threaded and multi-threaded performance. One goal is that the 100-thread benchmark should complete faster than the single threaded version just as it does in PyMongo 3.5.1.

      Pitfalls

      It could be impossible to match 3.5.1 performance because many features have been added since then which we can't remove (sessions, retryable reads/writes, CSOT, maxConnecting, logging, OP_MSG, etc...) but we should be able to get closer the performance of 3.7 (or ideally 3.5).

            Assignee:
            Unassigned Unassigned
            Reporter:
            shane.harvey@mongodb.com Shane Harvey
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: