Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19966

Excessive cursor caching in integration layer under WiredTiger

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 3.0.5
    • Component/s: WiredTiger
    • Fully Compatible
    • ALL

      With modest numbers of mongod client connections and collections a large number of WT cursors can be cached by the integration layer. Each cached WT cursor can consume significant memory (as much as the largest document it has handled), so this can result in very significant excess memory utilization and OOM. Repro:

      1000 connections:

      for t in $(seq 1000); do
          js test "load('test.js'); repro($t)" &
      done
      wait
      

      each connection updates 100 collections, each with 4 indexes:

      collections = 100
      
      function repro(t) {
          var round = 0;
          while (true) {
              round++;
              if (t==1)
                  print("=== Starting round: " + round)
              for (i=0; i<collections; i++) {
                  var c = db["col" + i]
                  if (round==1) {
                      c.ensureIndex({a:1})
                      c.ensureIndex({b:1})
                      c.ensureIndex({c:1})
                  }
                  c.insert({a:round, b:round, c:round})
              }
              if (t==1) {
                  cursors = db.serverStatus().wiredTiger.session['open cursor count']
                  print('cursors', cursors)
              }
          }
      }
      

      Result is more than 500k cached WT cursors:

      === Starting round: 1
      cursors 112025
      === Starting round: 2
      cursors 231014
      === Starting round: 3
      cursors 318091
      === Starting round: 4
      cursors 375483
      === Starting round: 5
      cursors 393430
      === Starting round: 6
      cursors 393760
      === Starting round: 7
      cursors 393950
      === Starting round: 8
      cursors 393950
      === Starting round: 9
      cursors 397336
      === Starting round: 10
      cursors 408085
      === Starting round: 11
      cursors 416445
      === Starting round: 12
      cursors 443923
      === Starting round: 13
      cursors 471652
      === Starting round: 14
      cursors 485585
      === Starting round: 15
      cursors 488255
      === Starting round: 16
      cursors 495570
      === Starting round: 17
      cursors 498425
      === Starting round: 18
      cursors 500035
      

        1. wt-cache-spacetime.png
          204 kB
          Michael Cahill

            Assignee:
            david.hows David Hows
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            4 Vote for this issue
            Watchers:
            23 Start watching this issue

              Created:
              Updated:
              Resolved: