Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-177

MongoDB Sink Connector deadletterqueuetopic issue

    • Type: Icon: Bug Bug
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 1.3.0
    • Component/s: Documentation, Sink
    • None
    • Environment:
      Dev

      Our development team has found a defect on the Kafka Connect MongoDB Sink (https://docs.mongodb.com/kafka-connector/master/kafka-sink). It seems that support for the deadletterqueue has been removed/disappeared and cant get the feature to working in the latest version as per the MongoDB documentation. (we also confirmed the code is missing from the mainline in GIT)

      According this document, the sink connector is supposed to support deadletterqueue topic (https://docs.mongodb.com/kafka-connector/master/kafka-sink-properties#dead-letter-queue-configuration-settings

      errors.deadletterqueue.topic.name=example.deadletterqueue

      However, when I publish a message that violates the MongoDB unique index, I do not see the message in dead letter queue topic. I tried both console consumer and in Spring Boot application, no luck.

      And then, I went to the GitHub (master) to check the source code. It seems the dead letter queue topic is not part of MongoSinkTopicConfig.java (https://github.com/mongodb/mongo-kafka/blob/master/src/main/java/com/mongodb/kafka/connect/sink/MongoSinkTopicConfig.java), i.e. errors.deadletterqueue.topic.name is not part of the configuration. On the other hand, that properties is available in MongoSourceConfig.java (https://github.com/mongodb/mongo-kafka/blob/master/src/main/java/com/mongodb/kafka/connect/source/MongoSourceConfig.java).

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            adam.cowin@td.com Adam Cowin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: