Cloudera Enterprise 6.0 Beta | Other versions

Installing or Upgrading Cloudera Distribution of Apache Spark 2

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

Cloudera Distribution of Apache Spark 2 is distributed as two files: a CSD file and a parcel, both of which need to installed on the cluster.

Installing Cloudera Distribution of Apache Spark 2

You can install Spark2 using the following instructions:

If your Cloudera Manager Server does have Internet access, you can use the CSD and proceed with the following instructions:

  1. Check that all the software prerequisites are satisfied. If not, you might need to upgrade or install other software components first. See Product Compatibility Matrix for Apache Spark for details.
  2. Download the Spark2 CSD.
  3. Install the Spark2 CSD into Cloudera Manager, as described in .
  4. In the Cloudera Manager Admin Console, add the Spark2 parcel repository to the .
      Note: If your Cloudera Manager Server does not have Internet access, you can use the Spark 2 parcel files, put them into a , and then configure the Cloudera Manager Server to target this newly-created repository.
  5. Download the Spark2 parcel, distribute the parcel to the hosts in your cluster, and activate the parcel. See .
  6. Add the Spark 2 service to your cluster. When configuring the assignment of role instances to hosts, add a gateway role to every host. The History Server port in CDH 6 is 18088, the same as in CDH 5, instead of 18089 as was the case with the separate Spark 2 parcel.
  7. Return to the Home page by clicking the Cloudera Manager logo.
  8. Click to invoke the cluster restart wizard.
  9. Click Restart Stale Services.
  10. Click Restart Now.
  11. Click Finish.

Upgrading to Cloudera Distribution of Apache Spark 2.2

If you are already using Cloudera Distribution of Apache Spark 2.0 or 2.1, here are the steps to upgrade to Cloudera Distribution of Apache Spark 2.2, while keeping any non-default configurations for Spark 2 that have already been applied:

  • Remove the CSD JAR for Cloudera Distribution of Apache Spark 2.0 or 2.1 from /opt/cloudera/csd. Refer to Version and Packaging Information for the names of the JAR files corresponding to each version.

  • Add the CSD JAR for Cloudera Distribution of Apache Spark 2.2 to /opt/cloudera/csd. Set correct permissions and ownership.

  • Restart the cloudera-scm-server service.

  • In Cloudera Manager, deactivate the parcel corresponding to Cloudera Distribution of Apache Spark 2.0 or 2.1.

  • In Cloudera Manager, activate the parcel corresponding to Cloudera Distribution of Apache Spark 2.2.

  • Restart services and deploy the client configurations.

  Note:

All CDH clusters managed by a single Cloudera Manager instance must use exactly the same Spark 2 version. Make sure to install or upgrade the CSDs and parcels across all machines of all clusters at the same time.

Cloudera Distribution of Apache Spark 2.2 requires JDK 8.

Page generated March 7, 2018.