Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20944

Contention on ThreadPoolTaskExecutor::_mutex

    • Type: Icon: Improvement Improvement
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 3.2.0-rc3
    • Affects Version/s: 3.1.9
    • Component/s: Internal Code
    • None
    • Fully Compatible
    • Sharding B (10/30/15), QuInt C (11/23/15)

      The new mongos query path schedules commands to run on the shards using the ThreadPoolTaskExecutor attached to Grid. There is a single instance of the ThreadPoolTaskExecutor per mongos process which synchronizes access to its internal queues through a single mutex. Performance testing has shown that, when there are a sufficient number of concurrent queries executing on mongos, the threads spend much of their time blocked waiting to acquire this lock. Hacking mongos to create a ThreadPoolTaskExecutor per connection thread shows a 3X increase in throughput for a find-by-_id benchRun() workload.

      For reference, I'm using the following script to generate load on mongos:

      #!/bin/bash
      
      THREADS=$1
      BENCH_SECONDS=$2
      RPC=opQueryOnly
      PORT=20004
      
      ./mongo --port $PORT --quiet --eval 'db.foo.insert({_id:1})'
      
      ./mongo --port $PORT --rpcProtocols $RPC --eval '
      ops = [{op:"find", ns:"test.foo", query: {_id: 1}}]
      results = benchRun({ops:ops, parallel:'$THREADS', seconds:'$BENCH_SECONDS', host:"localhost:'$PORT'"})
      print(Math.round(results["totalOps/s"]))
      '
      

        1. vtune_locks_and_waits.png
          227 kB
          David Storch

            Assignee:
            david.storch@mongodb.com David Storch
            Reporter:
            david.storch@mongodb.com David Storch
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: