Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-85861

ExceededMemoryLimit thrown when closing windows with 4GB of state

    • Atlas Streams
    • Sprint 55

      this is the pipeline i used in the test:
      https://github.com/10gen/mongohouse/blob/master/testdata/bin/streams-smoke/utils/pipelines.go#L103
      this is what the input docs look like:
      https://github.com/10gen/mongohouse/blob/master/testdata/bin/streams-smoke/utils/pipelines.go#L194
      this is where the number of docs to insert are calculated, based on MBSTATEPERWINDOW of 4000:
      https://github.com/10gen/mongohouse/blob/master/testdata/bin/streams-smoke/smoke_test.go#L308

       

      When I tested this, the memory tracker reported the expected memory usage when the window was open (about 4GB), then hit “out of memory” on SP30 when closing the window. Note it’s a clean memory tracker error, not an OS OOM. So we might not have to fix anything in the short term.

      But it would be interesting to understand why the window close causes such a large spike in the memory consumption.

      ===

       

      See https://docs.google.com/document/d/1zPurErldtRGkl9COOM8jv_R_SAkfn4KW2Zd0temb71I/edit

      =====

      In this PR, I added a smoketest that creates a 6GB window.

      Things work fine when the window is open, and the SP reports the expected stateSize. When the window is closed, the SP runs into "Stream processing instance out of memory".

      We should understand where this memory spike comes from, it doesn't seem like it should be happening. We can likely re-create this using jstests against a local engine build. We can also repro this in the service environment using the smoke test I added.

      Another odd thing is the Grafana/Prom reported memory consumption and the state size seem to disagree when the "6GB" window is open. The SP state size reports 6GB, Grafana reports ~1GB.

      AtlasStreamProcessing> sp.local_immortal_smoke_test_c075d2ba_e26e6f6f.stats()
      {
        ok: 1,
        ns: '651f3b1e28508c679f051ee5.651efa6e04b6d81727d5fa9f.local_immortal_smoke_test_c075d2ba_e26e6f6f',
        stats:

      {    ....     stateSize: Long("6203768000"),     watermark: ISODate("2024-02-12T21:10:17.262Z"),     ok: 1   }

       

       

       

            Assignee:
            Unassigned Unassigned
            Reporter:
            matthew.normyle@mongodb.com Matthew Normyle
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: