Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-185

Support dynamic topic naming strategies

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 1.4.0
    • Affects Version/s: None
    • Component/s: Source
    • None
    • Needed
    • Hide

      Added Dynamic Topic Mapping for the Source

      Added a new interface `TopicMapper` with a `getTopic` method. Implementations can take the Change Stream Document to determine the correct `topic` to publish the data to.

      The Source connector includes a single implementation: `DefaultTopicMapper`. The `DefaultTopicMapper` uses the 'ns' field to determine the topic to publish to.

      The following configuration options can help configure topic mapping:

      • `topic.prefix`
        Adds a prefix to the topic eg: `<prefix>.<databaseName>.<collectionName>`
        Default: ""
      • `topic.suffix`
        Adds a suffix to the topic eg: `<databaseName>.<collectionName>.<suffix>`
        Default: ""
      • `topic.namespace.map`
        A json map to maps change stream document namespaces to topics:
        For example: ` {\"db\": \"dbTopic\", \"db.coll\": \"dbCollTopic\"}

        ` will map all change stream documents from the `db` database to `dbTopic.<collectionName>` apart from any documents from the `db.coll` namespace which map to the `dbCollTopic` topic.
        Default: ""

      • `topic.mapper` - use an alternative mapping class, users can implement their own `TopicMapper` class to handle the mapping logic.
        Default: com.mongodb.kafka.connect.source.topic.mapping.DefaultTopicMapper
      Show
      Added Dynamic Topic Mapping for the Source Added a new interface `TopicMapper` with a `getTopic` method. Implementations can take the Change Stream Document to determine the correct `topic` to publish the data to. The Source connector includes a single implementation: `DefaultTopicMapper`. The `DefaultTopicMapper` uses the 'ns' field to determine the topic to publish to. The following configuration options can help configure topic mapping: `topic.prefix` Adds a prefix to the topic eg: `<prefix>.<databaseName>.<collectionName>` Default: "" `topic.suffix` Adds a suffix to the topic eg: `<databaseName>.<collectionName>.<suffix>` Default: "" `topic.namespace.map` A json map to maps change stream document namespaces to topics: For example: ` {\"db\": \"dbTopic\", \"db.coll\": \"dbCollTopic\"} ` will map all change stream documents from the `db` database to `dbTopic.<collectionName>` apart from any documents from the `db.coll` namespace which map to the `dbCollTopic` topic. Default: "" `topic.mapper` - use an alternative mapping class, users can implement their own `TopicMapper` class to handle the mapping logic. Default: com.mongodb.kafka.connect.source.topic.mapping.DefaultTopicMapper

      Similar to KAFKA-159 but takes the MongoNamespace and maps it to a topic.

      Only a default mapper will be added and will include prefix and suffix settings. However, it will allow users to be flexible in

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            ross@mongodb.com Ross Lawley
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: