-
Type: Task
-
Resolution: Done
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: None
-
None
Sorry - me again. Things going very well - I just pulled data from an aggregate pipeline into a MongoRDD and I can see the results with .toDF.show() and looks as expected. Also I can printSchema and it looks correct.But when I do this
aggBillingStatusDf.registerTempTable("temp_billingupdate") it looks successful but when I look for the table like this sqlContext.sql("select * from temp_aggregatedBillingStats").show() it's nowhere to be found. If I showTables , sqlContext.sql("show tables").show(), I still don't see it.
Being clever, and because I really need the data I tried to take it there like this:
aggBillingStatusDf.write.mode(SaveMode.Overwrite).saveAsTable("localBillingTest")
and got this error
java.lang.RuntimeException: Tables created with SQLContext must be TEMPORARY. Use a HiveContext instead.
but of course my sqlContext IS a HiveContext:
scala> sqlContext
res44: org.apache.spark.sql.SQLContext = org.apache.spark.sql.hive.HiveContext@64ef0650
Any ideas?
- is duplicated by
-
SPARK-45 Reuse sqlContext when creating new dataframes
- Closed