Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-235

Try to save a list with null values return "Cannot cast null into a StringType"

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 2.4.1, 2.3.3, 2.2.7, 2.1.6
    • Affects Version/s: 2.4.0
    • Component/s: Writes
    • None

      I need to store documents in mongodb that contains list with null values like

      [
       {"id": 1, "type": "good record", "attributes": {"params": "anyparam", "liste": ["a", "b", "c"]}},
       {"id": 2, "type": "bad record", "attributes": {"params": "anyparam", "liste": ["d", null, null, "e"]}}
       ]

      when trying to save them I have the following exception :

      Caused by: com.mongodb.spark.exceptions.MongoTypeConversionException: Cannot cast null into a StringType
       at com.mongodb.spark.sql.MapFunctions$$anonfun$com$mongodb$spark$sql$MapFunctions$$wrappedDataTypeToBsonValueMapper$1.apply(MapFunctions.scala:87)
       at com.mongodb.spark.sql.MapFunctions$$anonfun$com$mongodb$spark$sql$MapFunctions$$wrappedDataTypeToBsonValueMapper$1.apply(MapFunctions.scala:83)
       at com.mongodb.spark.sql.MapFunctions$$anonfun$12$$anonfun$apply$9.apply(MapFunctions.scala:158)
       at com.mongodb.spark.sql.MapFunctions$$anonfun$12$$anonfun$apply$9.apply(MapFunctions.scala:158)

       

      I start with a  .map() applying json.dumps to convert data to valid json.  At this step, values in my RDD are correct. But mongodb need a dataframe to write to db.

      So moving data to a dataframe give me a schema that is compliant and null json values are converted back to python None values. 

      But at last, when attempting to write data, the driver seems to misinterpret this dataset.

       

        1. output.txt
          32 kB
          Alexis Vannier
        2. test_mongodb.py
          1 kB
          Alexis Vannier

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            alexisvannier Alexis Vannier
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: