Uploaded image for project: 'Go Driver'
  1. Go Driver
  2. GODRIVER-2259

Excess memory consumption from Zstd compression

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 1.10.0
    • Affects Version/s: None
    • Component/s: None
    • None

      Original description:

      Summary

      We enabled zLib compression on ADL frontends and saw a large increase in memory consumed in the Go driver. A pprof heap profile taken while consumption was high is attached.

      ADL was not under particularly heavy load however memory consumption remained high. A Grafana chart of our memory consumption is also attached. The left side shows memory consumption with compression enabled; the right side shows memory consumption after disabling compression and restarting. The pprof snapshot was taken at one of the peaks on the left side.

      Please provide the version of the driver. If applicable, please provide the MongoDB server version and topology (standalone, replica set, or sharded cluster).

      Driver was v1.7.4. Server version and topology varies widely.

      How to Reproduce

      Unknown. Perhaps run a large number of find / match / count queries over time with zLib compression enabled.

      Additional Background

      The linked ticket, MHOUSE-3597, contains a discussion around enabling / disabling compression. There was also some benchmarking results for all compression included in ticket comments that you may find interesting, although it is unlikely to shed any light on this ticket.

      and were also attached with the original request.

      The original request was to investigate excess memory usage when using zlib compression. However, the attached pprof snapshot, profile006.png, indicated the memory usage of zstd. Our initial investigation and profiling on a minimal reproduction running zlib and zstd side by side also implied that zstd consumed memory more significantly as shown in diagrams zstd.png and zlib.png. Thus, this ticket was amended to investigate excess memory usage when using zstd compression.

        1. profile006.png
          profile006.png
          771 kB
        2. Screen Shot 2021-12-20 at 11.00.58 AM.png
          Screen Shot 2021-12-20 at 11.00.58 AM.png
          142 kB
        3. zlib.png
          zlib.png
          149 kB
        4. zstd_encodeall.png
          zstd_encodeall.png
          167 kB
        5. zstd_fixed.png
          zstd_fixed.png
          193 kB
        6. zstd.png
          zstd.png
          187 kB

            Assignee:
            qingyang.hu@mongodb.com Qingyang Hu
            Reporter:
            thomas.henretty@mongodb.com Thomas Henretty (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: