Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-232

Add date field conversion post processor

XMLWordPrintableJSON

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Unknown Unknown
    • 1.6.0
    • Affects Version/s: None
    • Component/s: Sink
    • None
    • Needed
    • Hide
      Config Description Field type Default Value Validation
      timeseries.timefield.auto.convert Converts the timeseries field to a BSON datetime type. If the value is a number value it will use it as the milliseconds from epoch. If the value is a String it will use `timeseries.timefield.auto.convert.date.format` configuration to parse the date. boolean false
      timeseries.timefield.auto.convert.date.format The DateTimeFormatter pattern to use when converting String dates.Defaults to supporting UTC ISO date times. string yyyy-MM-dd['T'][ ]HH:mm:ss[.][SSSSSS][SSS]['Z'] A valid DateTimeFormatter format
      Show
      Config Description Field type Default Value Validation timeseries.timefield.auto.convert Converts the timeseries field to a BSON datetime type. If the value is a number value it will use it as the milliseconds from epoch. If the value is a String it will use `timeseries.timefield.auto.convert.date.format` configuration to parse the date. boolean false timeseries.timefield.auto.convert.date.format The DateTimeFormatter pattern to use when converting String dates.Defaults to supporting UTC ISO date times. string yyyy-MM-dd ['T'] [ ]HH:mm:ss [.] [SSSSSS] [SSS] ['Z'] A valid DateTimeFormatter format

      The conversion SMT has some limitations, so for ease of use lets add a post process that handles json dates and longs / ints and converts them to BsonDateTime.

      Created an automatic post processor that will process documents when timeseries.timefield.auto.convert=true.
      For Strings yyyy-MM-dd['T'][ ]HH:mm:ss[.][SSSSSS][SSS]['Z'] is the default format used. So supports all of the following formatted strings: "2021-07-13T12:00:00.000001Z", "2021-07-13 12:00:00.000001", "2021-07-13T12:00:00.001Z", "2021-07-13 12:00:00.001", "2021-07-13T12:00:00Z", "2021-07-13 12:00:00"

      Failing to convert the field does not error in the processor and the timefield is left as is.

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            ross@mongodb.com Ross Lawley
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: