-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Atlas Streams
-
Fully Compatible
-
Sprint 66
Update 2/3/2025:
- We have the code ready but need to find a safe way to ship this. We only do the validation when not restoring from a checkpoint (indicating a fresh start of the processor). But there are some users of onError=dlq and validationAction=dlq that are also using a sample_stream_solar source (where checkpoint is disabled). The current implementation could break them.
- Ideas:
-
- skip the validation for sample sources?
- use stream processor feature flags (a bit cumbersome)
There are many operators like $validate, and future ones like $externalFunctions, $https that have `onError : [dlq, ignore, fail]` type functionality, or `validationAction : [discard, dlq]`.
During createStreamProcessor the pipeline should be validated to check if a DLQ has been passed in the create command against if any of the pipeline is configured to use a DLQ. If an operator is configured to use a DLQ but a DLQ is not passed at create the processor should fail to create and a error message telling the user which operator is requiring a DLQ provided.