Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-343

Improve schema inference for documents nested in arrays

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Unknown Unknown
    • 1.9.0
    • Affects Version/s: None
    • Component/s: Source
    • None
    • Needed

      Schema inference for documents nested in arrays falls back to "string" when any difference is detected in the schemas for the nested documents. This is necessary because Kafka schemas can not handle arrays with elements of different type. But we can improve the schema inference to detect some cases where the schemas for the nested documents are actually compatible:

      1. Where the field is present in one document but missing in another
      2. Where the field is present in one document but null in another
      3. Where the field types conflict (in this case we can push the conflict down to the schema for the field)
      4. Where the field is an array with elements of some type in one document but an empty array in another

            Assignee:
            jeff.yemin@mongodb.com Jeffrey Yemin
            Reporter:
            jeff.yemin@mongodb.com Jeffrey Yemin
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: