"C:\Program Files\Java\jdk1.8.0_92\bin\java" -Didea.launcher.port=7533 "-Didea.launcher.bin.path=C:\Program Files (x86)\JetBrains\IntelliJ IDEA Community Edition 2016.2.2\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program Files\Java\jdk1.8.0_92\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\deploy.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\ext\cldrdata.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\ext\dnsns.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\ext\jaccess.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\ext\jfxrt.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\ext\localedata.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\ext\nashorn.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\ext\sunec.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\ext\sunjce_provider.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\ext\sunmscapi.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\ext\sunpkcs11.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\ext\zipfs.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\javaws.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\jce.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\jfxswt.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\management-agent.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\plugin.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\resources.jar;C:\Program Files\Java\jdk1.8.0_92\jre\lib\rt.jar;D:\projects\transitgraph\target\scala-2.11\classes;C:\Users\REDACTED\.ivy2\cache\org.mongodb\mongo-java-driver\jars\mongo-java-driver-3.2.2.jar;C:\Users\REDACTED\.ivy2\cache\org.slf4j\slf4j-api\jars\slf4j-api-1.7.7.jar;C:\Users\REDACTED\.ivy2\cache\org.scala-lang\scala-reflect\jars\scala-reflect-2.11.0.jar;C:\Users\REDACTED\.ivy2\cache\org.scala-lang\scala-library\jars\scala-library-2.11.7.jar;C:\Users\REDACTED\.ivy2\cache\org.mongodb.spark\mongo-spark-connector_2.11\jars\mongo-spark-connector_2.11-2.0.0-rc1.jar;C:\Users\REDACTED\.ivy2\cache\graphframes\graphframes\jars\graphframes-0.2.0-spark2.0-s_2.11.jar;C:\Users\REDACTED\.ivy2\cache\com.typesafe.scala-logging\scala-logging-slf4j_2.11\jars\scala-logging-slf4j_2.11-2.1.2.jar;C:\Users\REDACTED\.ivy2\cache\com.typesafe.scala-logging\scala-logging-api_2.11\jars\scala-logging-api_2.11-2.1.2.jar;C:\spark\jars\xz-1.0.jar;C:\spark\jars\jta-1.1.jar;C:\spark\jars\jpam-1.1.jar;C:\spark\jars\guice-3.0.jar;C:\spark\jars\ivy-2.4.0.jar;C:\spark\jars\lz4-1.3.0.jar;C:\spark\jars\oro-2.0.8.jar;C:\spark\jars\ST4-4.0.4.jar;C:\spark\jars\avro-1.7.7.jar;C:\spark\jars\core-1.1.2.jar;C:\spark\jars\gson-2.2.4.jar;C:\spark\jars\jline-2.12.jar;C:\spark\jars\mail-1.4.7.jar;C:\spark\jars\mx4j-3.0.2.jar;C:\spark\jars\snappy-0.2.jar;C:\spark\jars\antlr-2.7.7.jar;C:\spark\jars\jsp-api-2.1.jar;C:\spark\jars\opencsv-2.3.jar;C:\spark\jars\py4j-0.10.1.jar;C:\spark\jars\xmlenc-0.52.jar;C:\spark\jars\base64-2.3.8.jar;C:\spark\jars\guava-14.0.1.jar;C:\spark\jars\janino-2.7.8.jar;C:\spark\jars\jets3t-0.9.3.jar;C:\spark\jars\jetty-6.1.26.jar;C:\spark\jars\jsr305-1.3.9.jar;C:\spark\jars\log4j-1.2.17.jar;C:\spark\jars\minlog-1.3.0.jar;C:\spark\jars\pyrolite-4.9.jar;C:\spark\jars\stream-2.7.0.jar;C:\spark\jars\jdo-api-3.0.1.jar;C:\spark\jars\json-20090211.jar;C:\spark\jars\objenesis-2.1.jar;C:\spark\jars\paranamer-2.3.jar;C:\spark\jars\scalap-2.11.8.jar;C:\spark\jars\avro-ipc-1.7.7.jar;C:\spark\jars\commons-io-2.4.jar;C:\spark\jars\httpcore-4.4.4.jar;C:\spark\jars\JavaEWAH-0.3.2.jar;C:\spark\jars\javax.inject-1.jar;C:\spark\jars\jaxb-api-2.2.2.jar;C:\spark\jars\libfb303-0.9.2.jar;C:\spark\jars\stax-api-1.0-2.jar;C:\spark\jars\stax-api-1.0.1.jar;C:\spark\jars\aopalliance-1.0.jar;C:\spark\jars\commons-cli-1.2.jar;C:\spark\jars\commons-net-2.2.jar;C:\spark\jars\derby-10.11.1.1.jar;C:\spark\jars\joda-time-2.9.3.jar;C:\spark\jars\jodd-core-3.5.2.jar;C:\spark\jars\libthrift-0.9.2.jar;C:\spark\jars\super-csv-2.2.0.jar;C:\spark\jars\zookeeper-3.4.6.jar;C:\spark\jars\activation-1.1.1.jar;C:\spark\jars\chill-java-0.8.0.jar;C:\spark\jars\chill_2.11-0.8.0.jar;C:\spark\jars\commons-dbcp-1.4.jar;C:\spark\jars\commons-lang-2.6.jar;C:\spark\jars\httpclient-4.5.2.jar;C:\spark\jars\javolution-5.5.1.jar;C:\spark\jars\slf4j-api-1.7.16.jar;C:\spark\jars\spire_2.11-0.7.4.jar;C:\spark\jars\xercesImpl-2.9.1.jar;C:\spark\jars\antlr-runtime-3.4.jar;C:\spark\jars\guice-servlet-3.0.jar;C:\spark\jars\hadoop-auth-2.7.2.jar;C:\spark\jars\hadoop-hdfs-2.7.2.jar;C:\spark\jars\hk2-api-2.4.0-b34.jar;C:\spark\jars\jackson-xc-1.9.13.jar;C:\spark\jars\jetty-util-6.1.26.jar;C:\spark\jars\jtransforms-2.4.0.jar;C:\spark\jars\kryo-shaded-3.0.3.jar;C:\spark\jars\metrics-jvm-3.1.2.jar;C:\spark\jars\netty-3.8.0.Final.jar;C:\spark\jars\pmml-model-1.2.15.jar;C:\spark\jars\api-util-1.0.0-M20.jar;C:\spark\jars\breeze_2.11-0.11.2.jar;C:\spark\jars\commons-codec-1.10.jar;C:\spark\jars\commons-pool-1.5.4.jar;C:\spark\jars\compress-lzf-1.0.3.jar;C:\spark\jars\jackson-core-2.6.5.jar;C:\spark\jars\leveldbjni-all-1.8.jar;C:\spark\jars\metrics-core-3.1.2.jar;C:\spark\jars\metrics-json-3.1.2.jar;C:\spark\jars\pmml-schema-1.2.15.jar;C:\spark\jars\bcprov-jdk15on-1.51.jar;C:\spark\jars\commons-lang3-3.3.2.jar;C:\spark\jars\commons-math3-3.4.1.jar;C:\spark\jars\hadoop-client-2.7.2.jar;C:\spark\jars\hadoop-common-2.7.2.jar;C:\spark\jars\hk2-utils-2.4.0-b34.jar;C:\spark\jars\java-xmlbuilder-1.0.jar;C:\spark\jars\javassist-3.18.1-GA.jar;C:\spark\jars\jersey-guava-2.22.2.jar;C:\spark\jars\jul-to-slf4j-1.7.16.jar;C:\spark\jars\protobuf-java-2.5.0.jar;C:\spark\jars\snappy-java-1.1.2.4.jar;C:\spark\jars\antlr4-runtime-4.5.3.jar;C:\spark\jars\bonecp-0.8.0.RELEASE.jar;C:\spark\jars\commons-digester-1.8.jar;C:\spark\jars\curator-client-2.6.0.jar;C:\spark\jars\jackson-jaxrs-1.9.13.jar;C:\spark\jars\jersey-client-2.22.2.jar;C:\spark\jars\jersey-common-2.22.2.jar;C:\spark\jars\jersey-server-2.22.2.jar;C:\spark\jars\parquet-column-1.7.0.jar;C:\spark\jars\parquet-common-1.7.0.jar;C:\spark\jars\parquet-hadoop-1.7.0.jar;C:\spark\jars\RoaringBitmap-0.5.11.jar;C:\spark\jars\scala-library-2.11.8.jar;C:\spark\jars\scala-reflect-2.11.8.jar;C:\spark\jars\scala-xml_2.11-1.0.2.jar;C:\spark\jars\slf4j-log4j12-1.7.16.jar;C:\spark\jars\spark-sql_2.11-2.0.0.jar;C:\spark\jars\stringtemplate-3.2.1.jar;C:\spark\jars\commons-logging-1.1.3.jar;C:\spark\jars\curator-recipes-2.6.0.jar;C:\spark\jars\hadoop-yarn-api-2.7.2.jar;C:\spark\jars\hive-cli-1.2.1.spark2.jar;C:\spark\jars\hk2-locator-2.4.0-b34.jar;C:\spark\jars\javax.ws.rs-api-2.0.1.jar;C:\spark\jars\jcl-over-slf4j-1.7.16.jar;C:\spark\jars\parquet-jackson-1.7.0.jar;C:\spark\jars\scala-compiler-2.11.8.jar;C:\spark\jars\spark-core_2.11-2.0.0.jar;C:\spark\jars\spark-hive_2.11-2.0.0.jar;C:\spark\jars\spark-repl_2.11-2.0.0.jar;C:\spark\jars\spark-tags_2.11-2.0.0.jar;C:\spark\jars\spark-yarn_2.11-2.0.0.jar;C:\spark\jars\xbean-asm5-shaded-4.4.jar;C:\spark\jars\api-asn1-api-1.0.0-M20.jar;C:\spark\jars\commons-compiler-2.7.6.jar;C:\spark\jars\commons-compress-1.4.1.jar;C:\spark\jars\commons-httpclient-3.1.jar;C:\spark\jars\hive-exec-1.2.1.spark2.jar;C:\spark\jars\hive-jdbc-1.2.1.spark2.jar;C:\spark\jars\jackson-databind-2.6.5.jar;C:\spark\jars\javax.inject-2.4.0-b34.jar;C:\spark\jars\json4s-ast_2.11-3.2.11.jar;C:\spark\jars\metrics-graphite-3.1.2.jar;C:\spark\jars\netty-all-4.0.29.Final.jar;C:\spark\jars\parquet-encoding-1.7.0.jar;C:\spark\jars\spark-mllib_2.11-2.0.0.jar;C:\spark\jars\apacheds-i18n-2.0.0-M15.jar;C:\spark\jars\arpack_combined_all-0.1.jar;C:\spark\jars\commons-beanutils-1.7.0.jar;C:\spark\jars\curator-framework-2.6.0.jar;C:\spark\jars\datanucleus-core-3.2.10.jar;C:\spark\jars\datanucleus-rdbms-3.2.9.jar;C:\spark\jars\jackson-core-asl-1.9.13.jar;C:\spark\jars\javax.servlet-api-3.1.0.jar;C:\spark\jars\json4s-core_2.11-3.2.11.jar;C:\spark\jars\parquet-generator-1.7.0.jar;C:\spark\jars\spark-graphx_2.11-2.0.0.jar;C:\spark\jars\spark-sketch_2.11-2.0.0.jar;C:\spark\jars\spark-unsafe_2.11-2.0.0.jar;C:\spark\jars\spire-macros_2.11-0.7.4.jar;C:\spark\jars\univocity-parsers-2.1.1.jar;C:\spark\jars\hadoop-annotations-2.7.2.jar;C:\spark\jars\hadoop-yarn-client-2.7.2.jar;C:\spark\jars\hadoop-yarn-common-2.7.2.jar;C:\spark\jars\javax.annotation-api-1.2.jar;C:\spark\jars\jersey-media-jaxb-2.22.2.jar;C:\spark\jars\avro-mapred-1.7.7-hadoop2.jar;C:\spark\jars\breeze-macros_2.11-0.11.2.jar;C:\spark\jars\commons-collections-3.2.2.jar;C:\spark\jars\commons-configuration-1.6.jar;C:\spark\jars\datanucleus-api-jdo-3.2.6.jar;C:\spark\jars\hive-beeline-1.2.1.spark2.jar;C:\spark\jars\jackson-annotations-2.6.5.jar;C:\spark\jars\jackson-mapper-asl-1.9.13.jar;C:\spark\jars\spark-catalyst_2.11-2.0.0.jar;C:\spark\jars\spark-launcher_2.11-2.0.0.jar;C:\spark\jars\apache-log4j-extras-1.2.17.jar;C:\spark\jars\eigenbase-properties-1.1.5.jar;C:\spark\jars\json4s-jackson_2.11-3.2.11.jar;C:\spark\jars\spark-streaming_2.11-2.0.0.jar;C:\spark\jars\validation-api-1.1.0.Final.jar;C:\spark\jars\hive-metastore-1.2.1.spark2.jar;C:\spark\jars\osgi-resource-locator-1.0.1.jar;C:\spark\jars\parquet-hadoop-bundle-1.6.0.jar;C:\spark\jars\commons-beanutils-core-1.8.0.jar;C:\spark\jars\htrace-core-3.1.0-incubating.jar;C:\spark\jars\mesos-0.21.1-shaded-protobuf.jar;C:\spark\jars\spark-mllib-local_2.11-2.0.0.jar;C:\spark\jars\calcite-core-1.2.0-incubating.jar;C:\spark\jars\jackson-module-paranamer-2.6.5.jar;C:\spark\jars\calcite-linq4j-1.2.0-incubating.jar;C:\spark\jars\hadoop-yarn-server-common-2.7.2.jar;C:\spark\jars\jackson-module-scala_2.11-2.6.5.jar;C:\spark\jars\jersey-container-servlet-2.22.2.jar;C:\spark\jars\parquet-format-2.3.0-incubating.jar;C:\spark\jars\spark-network-common_2.11-2.0.0.jar;C:\spark\jars\aopalliance-repackaged-2.4.0-b34.jar;C:\spark\jars\calcite-avatica-1.2.0-incubating.jar;C:\spark\jars\spark-network-shuffle_2.11-2.0.0.jar;C:\spark\jars\apacheds-kerberos-codec-2.0.0-M15.jar;C:\spark\jars\hadoop-mapreduce-client-app-2.7.2.jar;C:\spark\jars\hadoop-mapreduce-client-core-2.7.2.jar;C:\spark\jars\hadoop-yarn-server-web-proxy-2.7.2.jar;C:\spark\jars\spark-hive-thriftserver_2.11-2.0.0.jar;C:\spark\jars\scala-parser-combinators_2.11-1.0.4.jar;C:\spark\jars\hadoop-mapreduce-client-common-2.7.2.jar;C:\spark\jars\jersey-container-servlet-core-2.22.2.jar;C:\spark\jars\hadoop-mapreduce-client-shuffle-2.7.2.jar;C:\spark\jars\hadoop-mapreduce-client-jobclient-2.7.2.jar;C:\Program Files (x86)\JetBrains\IntelliJ IDEA Community Edition 2016.2.2\lib\idea_rt.jar" com.intellij.rt.execution.application.AppMain to.rwt.transitgraph.MultipleCollections
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/10/24 20:45:10 INFO SparkContext: Running Spark version 2.0.0
16/10/24 20:45:11 INFO SecurityManager: Changing view acls to: REDACTED
16/10/24 20:45:11 INFO SecurityManager: Changing modify acls to: REDACTED
16/10/24 20:45:11 INFO SecurityManager: Changing view acls groups to: 
16/10/24 20:45:11 INFO SecurityManager: Changing modify acls groups to: 
16/10/24 20:45:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(REDACTED); groups with view permissions: Set(); users  with modify permissions: Set(REDACTED); groups with modify permissions: Set()
16/10/24 20:45:11 INFO Utils: Successfully started service 'sparkDriver' on port 50674.
16/10/24 20:45:11 INFO SparkEnv: Registering MapOutputTracker
16/10/24 20:45:11 INFO SparkEnv: Registering BlockManagerMaster
16/10/24 20:45:11 INFO DiskBlockManager: Created local directory at C:\Users\REDACTED\AppData\Local\Temp\blockmgr-c6cfda3e-4032-4345-94cf-54e6fb3d25c9
16/10/24 20:45:11 INFO MemoryStore: MemoryStore started with capacity 1996.2 MB
16/10/24 20:45:11 INFO SparkEnv: Registering OutputCommitCoordinator
16/10/24 20:45:11 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/10/24 20:45:11 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.2.1:4040
16/10/24 20:45:11 INFO Executor: Starting executor ID driver on host localhost
16/10/24 20:45:11 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 50716.
16/10/24 20:45:11 INFO NettyBlockTransferService: Server created on 192.168.2.1:50716
16/10/24 20:45:11 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.2.1, 50716)
16/10/24 20:45:11 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.2.1:50716 with 1996.2 MB RAM, BlockManagerId(driver, 192.168.2.1, 50716)
16/10/24 20:45:11 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.2.1, 50716)
16/10/24 20:45:12 WARN SparkContext: Use an existing SparkContext, some configuration may not take effect.
16/10/24 20:45:12 INFO SharedState: Warehouse path is 'file:///tmp/spark-warehouse'.
16/10/24 20:45:12 WARN SparkSession$Builder: Use an existing SparkSession, some configuration may not take effect.
16/10/24 20:45:12 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 216.0 B, free 1996.2 MB)
16/10/24 20:45:12 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 404.0 B, free 1996.2 MB)
16/10/24 20:45:12 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.2.1:50716 (size: 404.0 B, free: 1996.2 MB)
16/10/24 20:45:12 INFO SparkContext: Created broadcast 0 from broadcast at MongoSpark.scala:499
16/10/24 20:45:12 INFO cluster: Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
16/10/24 20:45:12 INFO cluster: Cluster description not yet available. Waiting for 30000 ms before timing out
16/10/24 20:45:12 INFO connection: Opened connection [connectionId{localValue:1, serverValue:245}] to localhost:27017
16/10/24 20:45:12 INFO cluster: Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 3, 10]}, minWireVersion=0, maxWireVersion=5, maxDocumentSize=16777216, roundTripTimeNanos=547823, setName='REDACTED', canonicalAddress=REDACTED:27017, hosts=[REDACTED:27017], passives=[], arbiters=[], primary='REDACTED:27017', tagSet=TagSet{[]}, electionId=7fffffff000000000000002b, setVersion=1}
16/10/24 20:45:12 INFO MongoClientCache: Creating MongoClient: [localhost:27017]
16/10/24 20:45:13 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 216.0 B, free 1996.2 MB)
16/10/24 20:45:13 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 404.0 B, free 1996.2 MB)
16/10/24 20:45:13 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.2.1:50716 (size: 404.0 B, free: 1996.2 MB)
16/10/24 20:45:13 INFO SparkContext: Created broadcast 1 from broadcast at MongoSpark.scala:499
16/10/24 20:45:13 INFO connection: Opened connection [connectionId{localValue:2, serverValue:246}] to localhost:27017
16/10/24 20:45:13 INFO SparkContext: Starting job: treeAggregate at MongoInferSchema.scala:76
16/10/24 20:45:13 INFO DAGScheduler: Got job 0 (treeAggregate at MongoInferSchema.scala:76) with 1 output partitions
16/10/24 20:45:13 INFO DAGScheduler: Final stage: ResultStage 0 (treeAggregate at MongoInferSchema.scala:76)
16/10/24 20:45:13 INFO DAGScheduler: Parents of final stage: List()
16/10/24 20:45:13 INFO DAGScheduler: Missing parents: List()
16/10/24 20:45:13 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[6] at treeAggregate at MongoInferSchema.scala:76), which has no missing parents
16/10/24 20:45:13 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.7 KB, free 1996.2 MB)
16/10/24 20:45:13 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.6 KB, free 1996.2 MB)
16/10/24 20:45:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.2.1:50716 (size: 2.6 KB, free: 1996.2 MB)
16/10/24 20:45:13 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1012
16/10/24 20:45:13 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[6] at treeAggregate at MongoInferSchema.scala:76)
16/10/24 20:45:13 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
16/10/24 20:45:13 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0, NODE_LOCAL, 5377 bytes)
16/10/24 20:45:13 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/10/24 20:45:14 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1737 bytes result sent to driver
16/10/24 20:45:14 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 220 ms on localhost (1/1)
16/10/24 20:45:14 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
16/10/24 20:45:14 INFO DAGScheduler: ResultStage 0 (treeAggregate at MongoInferSchema.scala:76) finished in 0.234 s
16/10/24 20:45:14 INFO DAGScheduler: Job 0 finished: treeAggregate at MongoInferSchema.scala:76, took 0.301014 s
16/10/24 20:45:14 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 216.0 B, free 1996.2 MB)
16/10/24 20:45:14 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 404.0 B, free 1996.2 MB)
16/10/24 20:45:14 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.2.1:50716 (size: 404.0 B, free: 1996.2 MB)
16/10/24 20:45:14 INFO SparkContext: Created broadcast 3 from broadcast at MongoSpark.scala:499
16/10/24 20:45:14 INFO BlockManagerInfo: Removed broadcast_1_piece0 on 192.168.2.1:50716 in memory (size: 404.0 B, free: 1996.2 MB)
16/10/24 20:45:14 INFO BlockManagerInfo: Removed broadcast_2_piece0 on 192.168.2.1:50716 in memory (size: 2.6 KB, free: 1996.2 MB)


this is the route schema, after aggregating.

root
 |-- _id: struct (nullable = true)
 |    |-- oid: string (nullable = true)
 |-- agency: struct (nullable = true)
 |    |-- oid: string (nullable = true)
 |-- name: string (nullable = true)



16/10/24 20:45:15 INFO BlockManagerInfo: Removed broadcast_0_piece0 on 192.168.2.1:50716 in memory (size: 404.0 B, free: 1996.2 MB)
16/10/24 20:45:15 INFO ContextCleaner: Cleaned accumulator 44
16/10/24 20:45:15 INFO CodeGenerator: Code generated in 261.886281 ms
16/10/24 20:45:15 INFO CodeGenerator: Code generated in 8.863954 ms
16/10/24 20:45:15 INFO SparkContext: Starting job: count at MultipleCollections.scala:29
16/10/24 20:45:15 INFO DAGScheduler: Registering RDD 13 (count at MultipleCollections.scala:29)
16/10/24 20:45:15 INFO DAGScheduler: Got job 1 (count at MultipleCollections.scala:29) with 1 output partitions
16/10/24 20:45:15 INFO DAGScheduler: Final stage: ResultStage 2 (count at MultipleCollections.scala:29)
16/10/24 20:45:15 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1)
16/10/24 20:45:15 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 1)
16/10/24 20:45:15 INFO DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[13] at count at MultipleCollections.scala:29), which has no missing parents
16/10/24 20:45:15 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 11.1 KB, free 1996.2 MB)
16/10/24 20:45:15 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 5.8 KB, free 1996.2 MB)
16/10/24 20:45:15 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.2.1:50716 (size: 5.8 KB, free: 1996.2 MB)
16/10/24 20:45:15 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1012
16/10/24 20:45:15 INFO DAGScheduler: Submitting 3 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[13] at count at MultipleCollections.scala:29)
16/10/24 20:45:15 INFO TaskSchedulerImpl: Adding task set 1.0 with 3 tasks
16/10/24 20:45:15 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, partition 0, NODE_LOCAL, 5402 bytes)
16/10/24 20:45:15 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 2, localhost, partition 1, NODE_LOCAL, 5414 bytes)
16/10/24 20:45:15 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 3, localhost, partition 2, NODE_LOCAL, 5402 bytes)
16/10/24 20:45:15 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
16/10/24 20:45:15 INFO Executor: Running task 1.0 in stage 1.0 (TID 2)
16/10/24 20:45:15 INFO Executor: Running task 2.0 in stage 1.0 (TID 3)
16/10/24 20:45:15 INFO connection: Opened connection [connectionId{localValue:3, serverValue:247}] to localhost:27017
16/10/24 20:45:15 INFO connection: Opened connection [connectionId{localValue:4, serverValue:248}] to localhost:27017
16/10/24 20:45:15 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1725 bytes result sent to driver
16/10/24 20:45:15 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 77 ms on localhost (1/3)
16/10/24 20:45:15 INFO Executor: Finished task 2.0 in stage 1.0 (TID 3). 1638 bytes result sent to driver
16/10/24 20:45:15 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 3) in 85 ms on localhost (2/3)
16/10/24 20:45:15 INFO Executor: Finished task 1.0 in stage 1.0 (TID 2). 1725 bytes result sent to driver
16/10/24 20:45:15 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 2) in 92 ms on localhost (3/3)
16/10/24 20:45:15 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
16/10/24 20:45:15 INFO DAGScheduler: ShuffleMapStage 1 (count at MultipleCollections.scala:29) finished in 0.093 s
16/10/24 20:45:15 INFO DAGScheduler: looking for newly runnable stages
16/10/24 20:45:15 INFO DAGScheduler: running: Set()
16/10/24 20:45:15 INFO DAGScheduler: waiting: Set(ResultStage 2)
16/10/24 20:45:15 INFO DAGScheduler: failed: Set()
16/10/24 20:45:15 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[16] at count at MultipleCollections.scala:29), which has no missing parents
16/10/24 20:45:15 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 7.0 KB, free 1996.2 MB)
16/10/24 20:45:15 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 3.7 KB, free 1996.2 MB)
16/10/24 20:45:15 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on 192.168.2.1:50716 (size: 3.7 KB, free: 1996.2 MB)
16/10/24 20:45:15 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1012
16/10/24 20:45:15 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[16] at count at MultipleCollections.scala:29)
16/10/24 20:45:15 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
16/10/24 20:45:15 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 4, localhost, partition 0, ANY, 5275 bytes)
16/10/24 20:45:15 INFO Executor: Running task 0.0 in stage 2.0 (TID 4)
16/10/24 20:45:15 INFO ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 3 blocks
16/10/24 20:45:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 4 ms
16/10/24 20:45:15 INFO Executor: Finished task 0.0 in stage 2.0 (TID 4). 1873 bytes result sent to driver
16/10/24 20:45:15 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 4) in 28 ms on localhost (1/1)
16/10/24 20:45:15 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 
16/10/24 20:45:15 INFO DAGScheduler: ResultStage 2 (count at MultipleCollections.scala:29) finished in 0.029 s
16/10/24 20:45:15 INFO DAGScheduler: Job 1 finished: count at MultipleCollections.scala:29, took 0.149476 s
16/10/24 20:45:15 INFO CodeGenerator: Code generated in 7.666099 ms



1113




16/10/24 20:45:15 WARN SparkSession$Builder: Use an existing SparkSession, some configuration may not take effect.
16/10/24 20:45:15 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 224.0 B, free 1996.2 MB)
16/10/24 20:45:15 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 406.0 B, free 1996.2 MB)
16/10/24 20:45:15 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on 192.168.2.1:50716 (size: 406.0 B, free: 1996.2 MB)
16/10/24 20:45:15 INFO SparkContext: Created broadcast 6 from broadcast at MongoSpark.scala:499
16/10/24 20:45:15 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 224.0 B, free 1996.2 MB)
16/10/24 20:45:15 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 406.0 B, free 1996.2 MB)
16/10/24 20:45:15 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on 192.168.2.1:50716 (size: 406.0 B, free: 1996.2 MB)
16/10/24 20:45:15 INFO SparkContext: Created broadcast 7 from broadcast at MongoSpark.scala:499
16/10/24 20:45:15 INFO cluster: Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
16/10/24 20:45:15 INFO cluster: Cluster description not yet available. Waiting for 30000 ms before timing out
16/10/24 20:45:15 INFO connection: Opened connection [connectionId{localValue:5, serverValue:249}] to localhost:27017
16/10/24 20:45:15 INFO cluster: Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 3, 10]}, minWireVersion=0, maxWireVersion=5, maxDocumentSize=16777216, roundTripTimeNanos=262151, setName='REDACTED', canonicalAddress=REDACTED:27017, hosts=[REDACTED:27017], passives=[], arbiters=[], primary='REDACTED:27017', tagSet=TagSet{[]}, electionId=7fffffff000000000000002b, setVersion=1}
16/10/24 20:45:15 INFO MongoClientCache: Creating MongoClient: [localhost:27017]
16/10/24 20:45:15 INFO connection: Opened connection [connectionId{localValue:6, serverValue:250}] to localhost:27017
16/10/24 20:45:15 INFO SparkContext: Starting job: treeAggregate at MongoInferSchema.scala:76
16/10/24 20:45:15 INFO DAGScheduler: Got job 2 (treeAggregate at MongoInferSchema.scala:76) with 1 output partitions
16/10/24 20:45:15 INFO DAGScheduler: Final stage: ResultStage 3 (treeAggregate at MongoInferSchema.scala:76)
16/10/24 20:45:15 INFO DAGScheduler: Parents of final stage: List()
16/10/24 20:45:15 INFO DAGScheduler: Missing parents: List()
16/10/24 20:45:15 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[22] at treeAggregate at MongoInferSchema.scala:76), which has no missing parents
16/10/24 20:45:15 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 4.6 KB, free 1996.2 MB)
16/10/24 20:45:15 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 2.6 KB, free 1996.2 MB)
16/10/24 20:45:15 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on 192.168.2.1:50716 (size: 2.6 KB, free: 1996.2 MB)
16/10/24 20:45:15 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1012
16/10/24 20:45:15 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[22] at treeAggregate at MongoInferSchema.scala:76)
16/10/24 20:45:15 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
16/10/24 20:45:15 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 5, localhost, partition 0, NODE_LOCAL, 5378 bytes)
16/10/24 20:45:15 INFO Executor: Running task 0.0 in stage 3.0 (TID 5)
16/10/24 20:45:17 INFO BlockManagerInfo: Removed broadcast_5_piece0 on 192.168.2.1:50716 in memory (size: 3.7 KB, free: 1996.2 MB)
16/10/24 20:45:17 INFO BlockManagerInfo: Removed broadcast_4_piece0 on 192.168.2.1:50716 in memory (size: 5.8 KB, free: 1996.2 MB)
16/10/24 20:45:19 INFO ContextCleaner: Cleaned shuffle 0
16/10/24 20:45:19 INFO ContextCleaner: Cleaned accumulator 56
16/10/24 20:45:19 INFO ContextCleaner: Cleaned accumulator 55
16/10/24 20:45:19 INFO ContextCleaner: Cleaned accumulator 54
16/10/24 20:45:19 INFO ContextCleaner: Cleaned accumulator 53
16/10/24 20:45:19 INFO ContextCleaner: Cleaned accumulator 52
16/10/24 20:45:19 INFO ContextCleaner: Cleaned accumulator 51
16/10/24 20:45:19 INFO ContextCleaner: Cleaned accumulator 50
16/10/24 20:45:19 INFO ContextCleaner: Cleaned accumulator 49
16/10/24 20:45:19 INFO ContextCleaner: Cleaned accumulator 48
16/10/24 20:45:19 INFO ContextCleaner: Cleaned accumulator 47
16/10/24 20:45:19 INFO ContextCleaner: Cleaned accumulator 46
16/10/24 20:45:19 INFO ContextCleaner: Cleaned accumulator 45
16/10/24 20:45:20 INFO MongoClientCache: Closing MongoClient: [localhost:27017]
16/10/24 20:45:20 INFO connection: Closed connection [connectionId{localValue:4, serverValue:248}] to localhost:27017 because the pool has been closed.
16/10/24 20:45:20 INFO connection: Closed connection [connectionId{localValue:3, serverValue:247}] to localhost:27017 because the pool has been closed.
16/10/24 20:45:20 INFO connection: Closed connection [connectionId{localValue:2, serverValue:246}] to localhost:27017 because the pool has been closed.
16/10/24 20:45:24 INFO Executor: Finished task 0.0 in stage 3.0 (TID 5). 7513 bytes result sent to driver
16/10/24 20:45:24 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 5) in 9236 ms on localhost (1/1)
16/10/24 20:45:24 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 
16/10/24 20:45:24 INFO DAGScheduler: ResultStage 3 (treeAggregate at MongoInferSchema.scala:76) finished in 9.236 s
16/10/24 20:45:24 INFO DAGScheduler: Job 2 finished: treeAggregate at MongoInferSchema.scala:76, took 9.242302 s
16/10/24 20:45:24 WARN MongoInferSchema: Field 'from' contains conflicting types converting to StringType
16/10/24 20:45:24 WARN MongoInferSchema: Field 'schoolholiday' contains conflicting types converting to StringType
16/10/24 20:45:24 WARN MongoInferSchema: Field 'schoolterm' contains conflicting types converting to StringType
16/10/24 20:45:24 WARN MongoInferSchema: Field 'to' contains conflicting types converting to StringType
16/10/24 20:45:24 WARN MongoInferSchema: Field 'wheelchair' contains conflicting types converting to StringType
16/10/24 20:45:24 WARN MongoInferSchema: Field 'from' contains conflicting types converting to StringType
16/10/24 20:45:24 WARN MongoInferSchema: Field 'to' contains conflicting types converting to StringType
16/10/24 20:45:24 WARN MongoInferSchema: Field 'modified' contains conflicting types converting to StringType
16/10/24 20:45:24 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 216.0 B, free 1996.2 MB)
16/10/24 20:45:24 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 404.0 B, free 1996.2 MB)
16/10/24 20:45:24 INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on 192.168.2.1:50716 (size: 404.0 B, free: 1996.2 MB)
16/10/24 20:45:24 INFO SparkContext: Created broadcast 9 from broadcast at MongoSpark.scala:499



root
 |-- __v: integer (nullable = true)
 |-- _id: struct (nullable = true)
 |    |-- oid: string (nullable = true)
 |-- agency: struct (nullable = true)
[REDACTED]



16/10/24 20:45:25 INFO cluster: Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
16/10/24 20:45:25 INFO cluster: Cluster description not yet available. Waiting for 30000 ms before timing out
16/10/24 20:45:25 INFO connection: Opened connection [connectionId{localValue:7, serverValue:251}] to localhost:27017
16/10/24 20:45:25 INFO cluster: Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 3, 10]}, minWireVersion=0, maxWireVersion=5, maxDocumentSize=16777216, roundTripTimeNanos=552527, setName='REDACTED', canonicalAddress=REDACTED:27017, hosts=[REDACTED:27017], passives=[], arbiters=[], primary='REDACTED:27017', tagSet=TagSet{[]}, electionId=7fffffff000000000000002b, setVersion=1}
16/10/24 20:45:25 INFO MongoClientCache: Creating MongoClient: [localhost:27017]
16/10/24 20:45:25 INFO connection: Opened connection [connectionId{localValue:8, serverValue:252}] to localhost:27017
16/10/24 20:45:25 INFO SparkContext: Starting job: count at MultipleCollections.scala:36
16/10/24 20:45:25 INFO DAGScheduler: Registering RDD 29 (count at MultipleCollections.scala:36)
16/10/24 20:45:25 INFO DAGScheduler: Got job 3 (count at MultipleCollections.scala:36) with 1 output partitions
16/10/24 20:45:25 INFO DAGScheduler: Final stage: ResultStage 5 (count at MultipleCollections.scala:36)
16/10/24 20:45:25 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 4)
16/10/24 20:45:25 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 4)
16/10/24 20:45:25 INFO DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[29] at count at MultipleCollections.scala:36), which has no missing parents
16/10/24 20:45:25 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 16.4 KB, free 1996.2 MB)
16/10/24 20:45:25 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 7.4 KB, free 1996.2 MB)
16/10/24 20:45:25 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on 192.168.2.1:50716 (size: 7.4 KB, free: 1996.2 MB)
16/10/24 20:45:25 INFO SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1012
16/10/24 20:45:25 INFO DAGScheduler: Submitting 3 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[29] at count at MultipleCollections.scala:36)
16/10/24 20:45:25 INFO TaskSchedulerImpl: Adding task set 4.0 with 3 tasks
16/10/24 20:45:25 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 6, localhost, partition 0, NODE_LOCAL, 5402 bytes)
16/10/24 20:45:25 INFO TaskSetManager: Starting task 1.0 in stage 4.0 (TID 7, localhost, partition 1, NODE_LOCAL, 5414 bytes)
16/10/24 20:45:25 INFO TaskSetManager: Starting task 2.0 in stage 4.0 (TID 8, localhost, partition 2, NODE_LOCAL, 5402 bytes)
16/10/24 20:45:25 INFO Executor: Running task 0.0 in stage 4.0 (TID 6)
16/10/24 20:45:25 INFO Executor: Running task 1.0 in stage 4.0 (TID 7)
16/10/24 20:45:25 INFO Executor: Running task 2.0 in stage 4.0 (TID 8)
16/10/24 20:45:25 INFO connection: Opened connection [connectionId{localValue:10, serverValue:254}] to localhost:27017
16/10/24 20:45:25 INFO connection: Opened connection [connectionId{localValue:9, serverValue:253}] to localhost:27017
16/10/24 20:45:26 INFO ContextCleaner: Cleaned accumulator 233
16/10/24 20:45:26 INFO BlockManagerInfo: Removed broadcast_8_piece0 on 192.168.2.1:50716 in memory (size: 2.6 KB, free: 1996.2 MB)
16/10/24 20:45:26 INFO BlockManagerInfo: Removed broadcast_7_piece0 on 192.168.2.1:50716 in memory (size: 406.0 B, free: 1996.2 MB)
16/10/24 20:45:26 INFO BlockManagerInfo: Removed broadcast_6_piece0 on 192.168.2.1:50716 in memory (size: 406.0 B, free: 1996.2 MB)
16/10/24 20:45:27 INFO Executor: Finished task 2.0 in stage 4.0 (TID 8). 1711 bytes result sent to driver
16/10/24 20:45:27 INFO TaskSetManager: Finished task 2.0 in stage 4.0 (TID 8) in 2074 ms on localhost (1/3)
16/10/24 20:45:27 INFO Executor: Finished task 0.0 in stage 4.0 (TID 6). 1711 bytes result sent to driver
16/10/24 20:45:27 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 6) in 2458 ms on localhost (2/3)
16/10/24 20:45:30 INFO MongoClientCache: Closing MongoClient: [localhost:27017]
16/10/24 20:45:30 INFO connection: Closed connection [connectionId{localValue:6, serverValue:250}] to localhost:27017 because the pool has been closed.
16/10/24 20:45:33 INFO Executor: Finished task 1.0 in stage 4.0 (TID 7). 1711 bytes result sent to driver
16/10/24 20:45:33 INFO TaskSetManager: Finished task 1.0 in stage 4.0 (TID 7) in 8238 ms on localhost (3/3)
16/10/24 20:45:33 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 
16/10/24 20:45:33 INFO DAGScheduler: ShuffleMapStage 4 (count at MultipleCollections.scala:36) finished in 8.240 s
16/10/24 20:45:33 INFO DAGScheduler: looking for newly runnable stages
16/10/24 20:45:33 INFO DAGScheduler: running: Set()
16/10/24 20:45:33 INFO DAGScheduler: waiting: Set(ResultStage 5)
16/10/24 20:45:33 INFO DAGScheduler: failed: Set()
16/10/24 20:45:33 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[32] at count at MultipleCollections.scala:36), which has no missing parents
16/10/24 20:45:33 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 7.0 KB, free 1996.2 MB)
16/10/24 20:45:33 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 3.7 KB, free 1996.2 MB)
16/10/24 20:45:33 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on 192.168.2.1:50716 (size: 3.7 KB, free: 1996.2 MB)
16/10/24 20:45:33 INFO SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1012
16/10/24 20:45:33 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[32] at count at MultipleCollections.scala:36)
16/10/24 20:45:33 INFO TaskSchedulerImpl: Adding task set 5.0 with 1 tasks
16/10/24 20:45:33 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 9, localhost, partition 0, ANY, 5275 bytes)
16/10/24 20:45:33 INFO Executor: Running task 0.0 in stage 5.0 (TID 9)
16/10/24 20:45:33 INFO ShuffleBlockFetcherIterator: Getting 3 non-empty blocks out of 3 blocks
16/10/24 20:45:33 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms

this is a count of agencies, should return a smaller number than this, and the schema above shows that routes are still being read in

1113


16/10/24 20:45:33 INFO Executor: Finished task 0.0 in stage 5.0 (TID 9). 1873 bytes result sent to driver
16/10/24 20:45:33 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 9) in 5 ms on localhost (1/1)
16/10/24 20:45:33 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool 
16/10/24 20:45:33 INFO DAGScheduler: ResultStage 5 (count at MultipleCollections.scala:36) finished in 0.006 s
16/10/24 20:45:33 INFO DAGScheduler: Job 3 finished: count at MultipleCollections.scala:36, took 8.257240 s
16/10/24 20:45:33 INFO MongoClientCache: Closing MongoClient: [localhost:27017]
16/10/24 20:45:33 INFO connection: Closed connection [connectionId{localValue:9, serverValue:253}] to localhost:27017 because the pool has been closed.
16/10/24 20:45:33 INFO SparkContext: Invoking stop() from shutdown hook
16/10/24 20:45:33 INFO connection: Closed connection [connectionId{localValue:8, serverValue:252}] to localhost:27017 because the pool has been closed.
16/10/24 20:45:33 INFO connection: Closed connection [connectionId{localValue:10, serverValue:254}] to localhost:27017 because the pool has been closed.
16/10/24 20:45:33 INFO SparkUI: Stopped Spark web UI at http://192.168.2.1:4040
16/10/24 20:45:33 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/10/24 20:45:33 INFO MemoryStore: MemoryStore cleared
16/10/24 20:45:33 INFO BlockManager: BlockManager stopped
16/10/24 20:45:33 INFO BlockManagerMaster: BlockManagerMaster stopped
16/10/24 20:45:33 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/10/24 20:45:33 INFO SparkContext: Successfully stopped SparkContext
16/10/24 20:45:33 INFO ShutdownHookManager: Shutdown hook called
16/10/24 20:45:33 INFO ShutdownHookManager: Deleting directory C:\Users\REDACTED\AppData\Local\Temp\spark-e8b96239-e2e0-4133-90a0-04f52db67c6b

Process finished with exit code 0