Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-36346

Large memory consumption per pending getmore

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 5.2.0
    • Affects Version/s: None
    • Component/s: None
    • Fully Compatible
    • Query 2020-03-09, Query 2020-03-23, Query 2020-04-06, Query 2020-04-20, Query 2020-05-04, Query 2020-05-18, Query 2020-06-01, QE 2021-09-20, QE 2021-10-04, QE 2021-10-18, QE 2021-11-01
    • 135

      For getmores we allocate a 32 MB buffer for the response up front, which remains allocated while the getmore is pending. (This is the 16 MB bson maximum document size, plus some amount for the response message overhead, rounded up to 32 MB by the buffer allocator). This can add up to a lot of memory overhead if there are a large number of pending getmores, for example if you have a lot of oplog tails or change streams.

      Can this be improved? For example maybe we don't need to allocate for the worst case up front? Even the simple expedient of teaching the buffer allocator that it doesn't always need to stick to powers of two would improve this by a factor of two.

        1. comparison.png
          comparison.png
          201 kB
        2. diagnostic.data.no-prealloc.zip
          156 kB
        3. diagnostic.data-prealloc.zip
          156 kB
        4. no-prealloc-memory-usage.png
          no-prealloc-memory-usage.png
          572 kB
        5. prealloc-memory-usage.png
          prealloc-memory-usage.png
          570 kB

            Assignee:
            mohammad.dashti@mongodb.com Mohammad Dashti (Inactive)
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            4 Vote for this issue
            Watchers:
            43 Start watching this issue

              Created:
              Updated:
              Resolved: