Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17424

WiredTiger uses substantially more memory than accounted for by cache

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 3.0.3
    • Affects Version/s: None
    • Component/s: WiredTiger
    • Fully Compatible
    • ALL

      Running sysbench shows the following timeline:

      • At A the cache has reached its configured limit (actually 85%, I believe). At B the insert phase of the benchmark ends and the update phase begins.
      • Second row shows allocated bytes as reported by tcmalloc. Per documentation I believe this is the bytes as requested by the app so does not include any tcmalloc overhead. This reaches approximately 6 GB.
      • Third row shows bytes in cache as reported by WT, which reases 5 GB, about 20% less than the allocated bytes.
      • Last row shows difference between bytes in cache and allocated bytes. This grows in proportion to the number of bytes in the cache at a rate of about 20%, and stops growing at the point where bytes in cache stops growing. This suggests that WT allocates about 20% more bytes than are accounted for in the "bytes currently in cache" statistic when it is doing inserts.

      Call stack data was obtained by using perf to monitor calls to all tmalloc entry points. Here's a partially expanded reversed call tree for a shorter run up to 449 MB bytes allocated by WT. Timeline shows currently active memory (i.e. accounting for both allcoate and free) charged to each call site over the course of the run. Read max bytes active throughout the run for each call site from the "max.MB" column. Full call tree attached as well.

        1. sysbench.png
          sysbench.png
          105 kB
        2. ss.log
          20.26 MB
        3. simple.png
          simple.png
          64 kB
        4. repro.png
          repro.png
          70 kB
        5. perf-wt-reverse.png
          perf-wt-reverse.png
          290 kB
        6. perf-wt-reverse.html.gz
          2.72 MB
        7. noncache.png
          noncache.png
          87 kB
        8. cache.png
          cache.png
          379 kB
        9. cache_overhead=30.png
          cache_overhead=30.png
          74 kB

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            8 Vote for this issue
            Watchers:
            38 Start watching this issue

              Created:
              Updated:
              Resolved: