Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-58

DataFrame creation needs explicit schema support

XMLWordPrintableJSON

    • Type: Icon: Improvement Improvement
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 0.3
    • Affects Version/s: None
    • Component/s: None
    • None

      Due to restrictions in Scala 2.10 that limit Case Classes to 22 parameters, I tried to create an explicit schema definition and pass it to the MongoRDD's toDF() method and received this error:

      scala> val newTestType = StructType(
     |   StructField("sentClaims", DoubleType, true)::
     |   StructField("totalOrders", DoubleType, true)::
     |   Nil
     | )
newTestType: org.apache.spark.sql.types.StructType = StructType(StructField(sentClaims,DoubleType,true), StructField(totalOrders,DoubleType,true))
scala> val simpleProgrammaticMongoDF = sqlContext.loadFromMongoDB().toDF[newTestType]()
<console>:58: error: not found: type newTestType
         val simpleProgrammaticMongoDF = sqlContext.loadFromMongoDB().toDF[newTestType]()

      Please provide a toDF(schema: StructType) method that will allow me to build and use schemas as needed.

            Assignee:
            Unassigned Unassigned
            Reporter:
            rmwilliams Richard Williams
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: