Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-416

MongoMicroBatchPartitionReader.next should stop calling tryNext if the resume token timestamp is equal to the end offset timestamp

XMLWordPrintableJSON

    • Type: Icon: Improvement Improvement
    • Resolution: Done
    • Priority: Icon: Minor - P4 Minor - P4
    • 10.3.0
    • Affects Version/s: None
    • Component/s: Performance, Reads
    • None
    • Not Needed
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?
    • None
    • None
    • None
    • None
    • None
    • None

      Currently, if getTimestamp(cursor.getResumeToken()).compareTo(partition.getEndOffsetTimestamp()) is 0, MongoMicroBatchPartitionReader.next calls tryNext, but it shouldn't do it because such a request is guaranteed to return no change stream events because of the Filters.lt("clusterTime", partition.getEndOffsetTimestamp()) filter. Thanks to that same filter, the current behavior does not affect correctness, and may only affect performance.

            Assignee:
            valentin.kovalenko@mongodb.com Valentin Kavalenka
            Reporter:
            valentin.kovalenko@mongodb.com Valentin Kavalenka
            None
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved:
              None
              None
              None
              None