Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-328

Incorrect precision and scale is inferrred for Array[Decimal128] field in spark read.

      Precision and scale for an Array[Decimal128] field is inferred based on the value at the 0th index of the array. This results in 'null' values for all Decimal128 subfields having precision greater than the first element of the array.

      Example

      Source data - Mongo collection holding single record

      1. _id = "xxx" (String)
      2. arr = [1.4, 12.45] (Decimal128)

      Schema inferred for above collection by MongoSpark read is

      – _id: string (nullable = true)
      – arr: array (nullable = true)
        – element: decimal(2,1) (containsNull = true)

      Here the second value (12.45) becomes null as it has higher precision.

       

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            santhoshsuresh95@gmail.com Santhosh Suresh
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: