Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-80178

Send a new checkpoint message only if some output docs have been emitted since the last checkpoint.

    • Atlas Streams
    • Fully Compatible
    • Sprint 41, Sprint 42, Sprint 43, Sprint 44
    • 161

      In the changestream case, we can continue to advance the resume token even when no documents have been read from the stream.

      However in the kafka case, if we're not reading documents, there's no point in writing another checkpoint, the offsets have not advanced at all. So the main goal of this work is to avoid writing new checkpoints for Kafka $source when they are not useful (when we have not read any documents since the last checkpoint). 

       

      There are some other things to consider below, but we should probably leave these for a follow up ticket https://jira.mongodb.org/browse/SERVER-82440:

      and/or sizable amount of input has been processed

      and/or oldest seen input message is quite old

            Assignee:
            mayuresh.kulkarni@mongodb.com Mayuresh Kulkarni
            Reporter:
            sandeep.dhoot@mongodb.com Sandeep Dhoot
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: