The following sections describe changes in Spark support in CDH 6 that might require special handling during upgrades, or code changes within existing applications.

• All Spark applications built against Spark 1.6 in CDH 5 must be rebuilt against Spark 2.x in CDH 6.

• Spark 2 in CDH 6 works with Java 8, not Java 7. If this change produces any Java code incompatibilities, update your Java code and rebuild the application.

• Spark 2 in CDH 6 works with Scala 2.11, not Scala 2.10. If this change produces any Scala code incompatibilities, update your Scala code and rebuild the application.

• HiveContext and SQLContext have been removed, although those variables still work for backward compatibility. Use the SparkSession object to replace both of these handles.

• DataFrames have been removed from the Scala API. DataFrame is now a special case of Dataset.

• Spark 2.0 and higher do not use an assembly JAR for standalone applications.