Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-99672

Validate if DLQ is passed at .createStreamProcessor if used in Operators

    • Type: Icon: Improvement Improvement
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Atlas Streams
    • Fully Compatible
    • Sprint 66

      Update 2/3/2025:

      • We have the code ready but need to find a safe way to ship this. We only do the validation when not restoring from a checkpoint (indicating a fresh start of the processor). But there are some users of onError=dlq and validationAction=dlq that are also using a sample_stream_solar source (where checkpoint is disabled). The current implementation could break them.
      • Ideas:
        • skip the validation for sample sources?
        • use stream processor feature flags (a bit cumbersome)

      There are many operators like $validate, and future ones like $externalFunctions, $https that have `onError : [dlq, ignore, fail]` type functionality, or `validationAction : [discard, dlq]`. 

       

      During createStreamProcessor the pipeline should be validated to check if a DLQ has been passed in the create command against if any of the pipeline is configured to use a DLQ. If an operator is configured to use a DLQ but a DLQ is not passed at create the processor should fail to create and a error message telling the user which operator is requiring a DLQ provided. 

            Assignee:
            calvin.nix@mongodb.com Calvin Nix
            Reporter:
            joe.niemiec@mongodb.com Joe Niemiec
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: