Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-92

Kafka connector - Number of sink tasks

XMLWordPrintableJSON

    • Type: Icon: Epic Epic
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 1.2.0
    • Affects Version/s: 1.0
    • Component/s: Sink
    • None
    • Environment:
      AWS EC2, Sharded cluster
    • Done
    • Kafka connector - Number of sink tasks
    • Hide

      Engineer(s): Ross

      This epic is more like a one off ticket. We converted this work to an epic so it showed up on our board for quarterly planning.
      The actual work required for this ticket turned out to be easier than expected and is now in code review.

      Show
      Engineer(s): Ross This epic is more like a one off ticket. We converted this work to an epic so it showed up on our board for quarterly planning. The actual work required for this ticket turned out to be easier than expected and is now in code review.

      Epic Summary

      Summary

      The Kafka sink connector only ever supports a single task. Users should be able to use the tasks.max setting to increase parallelism with the connector:

      tasks.max - The maximum number of tasks that should be created for this connector. The connector may create fewer tasks if it cannot achieve this level of parallelism. 

      Was:

      I am testing this MongoDB sink connector to migrate large datasets (multi-TB) from one mongodb cluster to mongodb another. A challenge I am facing is throughput on the sink side. Irrespective of task.max parameter, only one sink task is created. Then I found below 2 tickets related to limiting the number of sink tasks to only 1. In this case, how do I improve throughput on the sink side? Just curious to know why number of tasks should limit to one? Were there any plans to improve this? 
      https://jira.mongodb.org/browse/KAFKA-62
      https://jira.mongodb.org/browse/KAFKA-46

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            yaramati@adobe.com Rajaramesh Yaramati
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: