Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-384

Mongo Source Connector Enhancement Request

    • Type: Icon: Question Question
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Source
    • None

      This is on lines of Jira ticket i have submitted for kafka sink connector ( 

      KAFKA-380 Mongo Sink Connector Enhancement Request - MongoDB Jira), this is a separate request for a similar enhancement for Kafka source connector

       

      Mongo Source Connector Enhancement Request:-

       

                      When the target Mongo DB is unavailable, the mongo sink connector with “error.tolerance = all” writes all the data to dead letter topic.

                      The purpose of setting “error.tolerance = all” is to ensure continuous processing of data while at the same time messages leading to data exceptions are moved to dead letter topic.

                                      Ex:- duplicate / malformed messages.

       

      This enhancement request is expected to resolve the current limitation by providing finer granularity of exception handling and to differentiate between retriable (aka transient) and non-retriable (aka non-transient) errors, and process them accordingly.

      Examples for the transient errors are mongo DB unavailable, network/connection issue & typical timeout errors.

      Examples for the non-transient errors are data violation errors thrown by Mongo DB, like duplicate key/primary key violations etc.,

       

      While encountering transient errors, the connector has to provide multiple retry attempts with time interval between each attempt.

      While encountering non-transient errors, the connector should continue to put the message into dead letter topic, and start to process the next message.

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            sanchayan.sen@natwest.com Sanchayan Sen
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: