Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-83185

Close Windows on completely idle streams

    • Type: Icon: New Feature New Feature
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 7.2.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • Atlas Streams
    • Fully Compatible
    • Sprint 36

      Today, completely idle streams do not advance the watermark, allowing windows to close. This behavior is preferred in systems that need deterministic results but can produce an edge case with streams that are at times, inactive and will not advance the event-watermarks used to close the windows. In these edge cases, it is possible for an event to be stuck in a window[15min] potentially for many minutes/hours/days before a new event is read from the source, which advances the watermark closing the window and producing the output. 

       

      To help solve these edge cases, another option should be configurable, which looks at the wall clock time only when the source inputs are entirely inactive. It should also consider the window time duration as well, as the window should not be closed if its possible events could still be consumed that would advance the watermark time. 

       

      ie - 15-minute window, with a sourceIdleness of 30 seconds, should only force the window to close on the windowEndTime + sourceidleness have been reached on the wall. 

       

      By default, this configuration would not be used as it creates un-deterministic results.  If the source cannot be connected to for some reason this property should have no effect .

            Assignee:
            matthew.normyle@mongodb.com Matthew Normyle
            Reporter:
            joe.niemiec@mongodb.com Joe Niemiec
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: