Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-18829

Cache usage exceeds configured maximum during index builds under WiredTiger

    • Fully Compatible
    • ALL

      Issue Status as of Jul 14, 2015

      ISSUE SUMMARY
      MongoDB running with the WiredTiger storage engine may, during large index builds, exceed the amount of memory allocated to the database cache.

      This issue may prevent large index builds in some situations, such as during initial sync of new replica set members.

      USER IMPACT
      Excessive memory consumption may cause mongod to either abort with an out-of-memory condition, or be killed by the operating system's OOM killer, leading to a loss of availability of the affected node.

      WORKAROUNDS
      Lowering the amount of cache available to WiredTiger to the 1GB minimum may allow users affected by this issue to complete large index builds.

      AFFECTED VERSIONS
      MongoDB 3.0.0 through 3.0.4

      FIX VERSION
      The fix is included in the 3.0.5 production release.

      Original description

      This has been seen under somewhat different circumstances by a couple of customers.

      Initial sync of a large db (multi TB, couple billion documents). Foreground build of _id index starts at A, and cache usage rises steadily to 6x configured maximum before being terminated by OOM.

      Mongorestore of 100GB db. Multiple parallel background index builds begin at A; at B cache usage begins to grow until it reaches about 1.5x configured max and is terminated by OOM.

      No complete repro yet, although may have got a partial repro: during initial sync of a 500 GB db cache usage briefly rose to about 120% of configured max.

        1. incident.png
          incident.png
          183 kB
        2. oom.png
          oom.png
          73 kB
        3. partial-repro.png
          partial-repro.png
          153 kB
        4. partial-repro-ckpt.png
          partial-repro-ckpt.png
          122 kB
        5. partial-repro-stacks.png
          partial-repro-stacks.png
          178 kB
        6. ss-alex.html
          2.04 MB
        7. ss-alex.log
          6.06 MB

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            4 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: