-
Type: Bug
-
Resolution: Fixed
-
Priority: Unknown
-
Affects Version/s: None
-
Component/s: None
-
None
h3 What did I use
- Databricks Runtime Version 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12)
- org.mongodb.spark:mongo-spark-connector:10.0.1
- MongoDB 5.0
h3 What did I do
data = [ ["tmp_user_1", [{"attr_1": "aaa"}, {"attr_2": "bbb"}]], ["tmp_user_2", []], ["tmp_user_3", None], ] columns = ["username", "attributes"] df = spark.createDataFrame(data, columns) ( df .write .format("mongodb") .option("database", database) .option("collection", collection) .option("connection.uri", connection_uri) .mode("overwrite") .save() )
h3 What did I get
the data can't be written to mongodb
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 25.0 failed 1 times, most recent failure: Lost task 2.0 in stage 25.0 (TID 90) (ip-10-172-164-192.us-west-2.compute.internal executor driver): com.mongodb.spark.sql.connector.exceptions.DataException: Cannot cast [tmp_user_1,WrappedArray(Map(attr_1 -> aaa), Map(attr_2 -> bbb))] into a BsonValue. StructType(StructField(username,StringType,true),StructField(attributes,ArrayType(MapType(StringType,StringType,true),true),true)) has no matching BsonValue. Error: Cannot cast [Map(attr_1 -> aaa), Map(attr_2 -> bbb)] into a BsonValue. ArrayType(MapType(StringType,StringType,true),true) has no matching BsonValue. Error: Cannot cast Map(attr_1 -> aaa) into a BsonValue. MapType(StringType,StringType,true) has no matching BsonValue. Error: scala.collection.immutable.Map$Map1 cannot be cast to java.util.Map
What do I expect
The df can be written to mongodb