Changes to Spark and Datasets with user defined functions mean that supporting the undefined UDF helper will no longer work as the generated pipeline $ne:null will also exclude Undefined values. Also, any fields with nullable=false declared in the schema need to be explicitly queried out in the aggregation pipeline.

Refs: SPARK-76