Cloudera Enterprise 6.0 Beta | Other versions

Frequently Asked Questions about Apache Spark in CDH

  Note:

This documentation refers to the Cloudera Distribution of Apache Spark 2.2 release 1. This component is generally available and is supported on CDH 5.7 through CDH 5.12.

A Hive compatibility issue in Cloudera Distribution of Apache Spark 2.0 release 1 affects CDH 5.10.1 and higher, CDH 5.9.2 and higher, CDH 5.8.5 and higher, and CDH 5.7.6 and higher. If you are using one of these CDH versions, you must upgrade to the Spark 2.0 release 2 or higher parcel, to avoid Spark 2 job failures when using Hive functionality.

This Frequently Asked Questions (FAQ) page covers general information about the Cloudera Distribution of Apache Spark 2, coexistence with Spark 1, and other questions that are relevant for early adopters of the latest Spark 2 features.

Continue reading:

Running Spark 1 and Spark 2 Side-by-Side

The Spark 2 service does not conflict with Spark 1 if it is installed. The history server uses a different port. Spark 2 shares the Spark 1 shuffle service if already available, or installs the shuffle service if not.

Why doesn't feature or library XYZ work?

A number of features, components, libraries, and integration points from Spark 1.6 are not supported with the Cloudera Distribution of Apache Spark 2. See Spark 2 Known Issues for details.

Page generated March 7, 2018.