Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-247

Recreate change stream from the point of failure for event > 16 MB

    • Type: Icon: Improvement Improvement
    • Resolution: Won't Fix
    • Priority: Icon: Unknown Unknown
    • None
    • Affects Version/s: None
    • Component/s: Source

      When a change event in the change stream exceeds the 16MB limit, existing change stream is closed with an exception and new change stream is opened. In a system with a higher update load this will likely miss the change events in the time it takes to start a new change stream. I have 2 proposal for improvement.

      Solution #1

      The error message of the exception contains the resumeToken of the failed event. Use the "ChangeStream.startAfter(<resumeToken>)" to start the new stream just after the failed event, leading to zero loss of events.

      Example error message

      BSONObj size: 19001449 (0x121F069) is invalid. Size must be between 0 and 16793600(16MB) First element: _id: { _data: "826115AEE9000000012B022C0100296E5A1004D148D6B22E8F49B3A65DAE80A4683566463C5F6964003C316663726B36326F6D30303030303030000004" }
      

      Solution #2

      Increment the "clusterTime" (introduced in v4.0) available in the MongoCommandException, by 1 ordinal and use it with "ChangeStream.startAtOperationTime(<BsonTimestamp>)"

      For sharded cluster, it is possible that multiple events may have same cluster time and this approach can skip few good events with same timestamp as the bad one.

            Assignee:
            Unassigned Unassigned
            Reporter:
            dhruvangmakadia1@gmail.com Dhruvang Makadia
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: