Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-347

Can't write MapType to mongodb

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Unknown Unknown
    • 10.0.2
    • Affects Version/s: None
    • Component/s: None
    • None

      h3 What did I use

      • Databricks Runtime Version 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12)
        • org.mongodb.spark:mongo-spark-connector:10.0.1
      • MongoDB 5.0

      h3 What did I do

      data = [
        ["tmp_user_1", [{"attr_1": "aaa"}, {"attr_2": "bbb"}]],
        ["tmp_user_2", []],
        ["tmp_user_3", None],
      ] 
      columns = ["username", "attributes"] 
      
      df = spark.createDataFrame(data, columns)
      
      (
        df
        .write
        .format("mongodb")
        .option("database", database)
        .option("collection", collection)
        .option("connection.uri", connection_uri)
        .mode("overwrite")
        .save()
      )
      

      h3 What did I get

      the data can't be written to mongodb

      Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 25.0 failed 1 times, most recent failure: Lost task 2.0 in stage 25.0 (TID 90) (ip-10-172-164-192.us-west-2.compute.internal executor driver): com.mongodb.spark.sql.connector.exceptions.DataException: Cannot cast [tmp_user_1,WrappedArray(Map(attr_1 -> aaa), Map(attr_2 -> bbb))] into a BsonValue. StructType(StructField(username,StringType,true),StructField(attributes,ArrayType(MapType(StringType,StringType,true),true),true)) has no matching BsonValue. Error: Cannot cast [Map(attr_1 -> aaa), Map(attr_2 -> bbb)] into a BsonValue. ArrayType(MapType(StringType,StringType,true),true) has no matching BsonValue. Error: Cannot cast Map(attr_1 -> aaa) into a BsonValue. MapType(StringType,StringType,true) has no matching BsonValue. Error: scala.collection.immutable.Map$Map1 cannot be cast to java.util.Map

      What do I expect

      The df can be written to mongodb

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            me@kytse.com Kit Yam Tse
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: