Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-230

Change stream doesn't recover with errors.tolerance=all on Mongo 4

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Unknown Unknown
    • 1.6.0
    • Affects Version/s: 1.5.1, 1.6.0
    • Component/s: Source
    • Environment:
      Confluent 5.5.4 (kafka 2.4.x I believe)

      Currently if we fall off the end of the oplog for whatever reason we get the following error on mongo 4:

      INFO An exception occurred when trying to get the next item from the Change Stream: Query failed with error code 280 and error message 'Resume of change stream was not possible, as the resume point may no longer be in the oplog.' on server mongos-profile-1.sailthru.cloud:17020 (com.mongodb.kafka.connect.source.MongoSourceTask)

      With errors.tolerance=all the source should auto recover starting with a fresh "change stream".

      However this doesn't happen due to:

      1. The resumeTokenNotFound error detection not matching on the returned error.
      2. The error actually being thrown during the reading of the cursor, not the creation of the cursor(where the recovery logic currently lives)

      This minimal patch is enough to fix for us https://github.com/tim-patterson/mongo-kafka/commit/a281a6c42256efdea6613e5a2a91bca61cfa75c6

      (I'm currently chasing up on my companies open source policies so I'm not sure if I'm going to be in a position to submit a fix/PR)

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            tpatterson@sailthru.com Tim Patterson
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: