-
Type: Task
-
Resolution: Fixed
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Atlas Streams
-
Fully Compatible
-
Sprint 41, Sprint 42, Sprint 43, Sprint 44
-
161
In the changestream case, we can continue to advance the resume token even when no documents have been read from the stream.
However in the kafka case, if we're not reading documents, there's no point in writing another checkpoint, the offsets have not advanced at all. So the main goal of this work is to avoid writing new checkpoints for Kafka $source when they are not useful (when we have not read any documents since the last checkpoint).
There are some other things to consider below, but we should probably leave these for a follow up ticket https://jira.mongodb.org/browse/SERVER-82440:
and/or sizable amount of input has been processed
and/or oldest seen input message is quite old
- has to be done before
-
SERVER-87071 Enable new checkpoint coordinator logic during shutdown too
- Needs Scheduling