Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-33296

Excessive memory usage due to heap fragmentation

    • Product Performance
    • Linux
    • Hide

      david.daly - Handing over to you per e-mail discussion.

      Show
      david.daly - Handing over to you per e-mail discussion.
    • Dev Tools 2019-05-06, Dev Tools 2019-04-22

      The changes described in SERVER-20306 eliminated a common source of memory fragmentation, but it can still occur for other reasons. Here's an example from a node undergoing initial sync:

      Over time

      • allocated memory never exceeds 8 GB
      • but heap size and resident memory reach nearly 14 GB
      • this is due to an accumulation of pageheap_free_bytes

      A common cause of this is a shifting distribution of allocated memory sizes, which leaves free pages dedicated to one size of buffer unable to be used for new memory requests because they are for a different size buffer.

      Setting TCMALLOC_AGGRESSIVE_DECOMMIT can address this issue by causing tcmalloc to aggressively return the free pages to the o/s where they can then be re-used by tcmalloc to satisfy new memory requests. However can have an unacceptable negative performance impact. Is there a tweak to tcmalloc that can give us better behavior for workloads like this?

        1. image-2019-10-21-15-09-33-867.png
          image-2019-10-21-15-09-33-867.png
          225 kB
        2. fragmentation-3.6.2.png
          fragmentation-3.6.2.png
          107 kB
        3. fragmentation.png
          fragmentation.png
          104 kB
        4. 4.3.3-rr100.png
          4.3.3-rr100.png
          330 kB

            Assignee:
            backlog-server-perf [DO NOT USE] Backlog - Performance Team
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            8 Vote for this issue
            Watchers:
            63 Start watching this issue

              Created:
              Updated: