Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-127

Kafka Source connector handling documents greater than 16MB BSON

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 1.1
    • Component/s: None
    • None
    • Environment:
      Kafka Connector 1.1
      Mongo DB:3.6

      Change stream response documents must adhere to the 16MB BSON : https://docs.mongodb.com/manual/administration/change-streams-production-recommendations/

      Currently Kafka connector:

      1) Doesn't create a error or exception but has an info message:

       INFO An exception occurred when trying to get the next item from the changestream. (com.mongodb.kafka.connect.source.MongoSourceTask) INFO An exception occurred when trying to get the next item from the changestream. (com.mongodb.kafka.connect.source.MongoSourceTask)kfc-mongodb-7bbc79cb64-hdddg kfc-mongodb 2020-07-09T16:29:47.376320977Z com.mongodb.MongoCommandException: Command failed with error 10334 (Location10334): 'BSONObj size: 19376544 (0x127A9A0) is invalid. Size must be between 0 and 16793600(16MB) First element: _id: { _data: BinData(0, 825F0745F700000005461E5F69640031035342000004BC005A100458C65EB8491243558BB64BDAC2914E5204), _typeBits: BinData(0, 02) }' on server db-core-1.ebs.us-east-1.stage-us.int.evbg.io:29001. The full response is {"operationTime": {"$timestamp": {"t": 1594312187, "i": 87, "ok": 0.0, "errmsg": "BSONObj size: 19376544 (0x127A9A0) is invalid. Size must be between 0 and 16793600(16MB) First element: _id: { _data: BinData(0, 825F0745F700000005461E5F69640031035342000004BC005A100458C65EB8491243558BB64BDAC2914E5204), _typeBits: BinData(0, 02) }", "code": 10334, "codeName": "Location10334", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1594312187, "i": 87}}, "signature": {"hash":

      {"$binary": "WPjr3+K8zh6NDOpkTmK6rNvmM4M=", "$type": "00"}

      , "keyId": {"$numberLong": "6828579125364523010"}}}}}}

      2. Task status is still running but it fails to process change stream

      kfc-mongodb-7bbc79cb64-hdddg kfc-mongodb 2020-07-09T15:59:57.191098599Z [2020-07-09 15:59:57,190] INFO Failed to resume change stream: BSONObj size: 19224166 (0x1255666) is invalid. Size must be between 0 and 16793600(16MB) First element: _id: { _data: BinData(0, 825F073D080000000E461E5F69640031035342000004BC005A100458C65EB8491243558BB64BDAC2914E5204), _typeBits: BinData(0, 02) } 10334 (com.mongodb.kafka.connect.source.MongoSourceTask)

       

      Regarding error handling, this seems related to https://jira.mongodb.org/browse/KAFKA-89

      Is there a way to have connector resilient of this issues with improved error handling and task status. Thanks in advance.

        1. mongo_1.1.txt
          12 kB
        2. mongo_1.2.txt
          4 kB
        3. MongoTest.java
          3 kB

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            sabari.mgn@gmail.com Sabari Gandhi
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: