Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-93792

Streams: Expose kafka per-partition watermark in verbose stats()

    • Atlas Streams
    • Fully Compatible
    • Sprint 56

      We currently only expose the combined watermark in the kafka source in the stats(). We want to expose per-partition watermark information in verbose stats as well.

      In the KafkaConsumerOperator, there is a watermark for each input partition.

      We can expose the partition watermark as a "watermark" field within the KafkaConsumerPartitionState struct:

      https://github.com/10gen/mongo/blob/master/src/mongo/db/modules/enterprise/src/streams/commands/stream_ops.idl#L134

      We can test this new functionality in the below places:

      • kafka_consumer_operator_test (run with ninja +kafka_consumer_operator_test -j400). This tests using mock Kafka classes.
      • kafka.js test (run with s-mongo-build && test-streams-kafka). This tests against a real kafka broker running in a local docker container.

       

            Assignee:
            calvin.nix@mongodb.com Calvin Nix
            Reporter:
            aadesh.patel@mongodb.com Aadesh Patel (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 1 week
                1w
                Remaining:
                Remaining Estimate - 1 week
                1w
                Logged:
                Time Spent - Not Specified
                Not Specified