Cloudera Enterprise 6.0 Beta | Other versions

Upgrading to CDH 6.x Using Packages

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

This topic describes how to upgrade CDH from any version of CDH 5.x to CDH 6.x using Cloudera Manager and packages.The minor version of Cloudera Manager you use to perform the upgrade must be equal to or greater than the CDH minor version. To upgrade Cloudera Manager, see Overview of Upgrading Cloudera Manager.

If the CDH 5 cluster you are upgrading was installed using packages, you can upgrade it using parcels, and the upgraded version of CDH will then use parcels for future upgrades or changes. You can also migrate your cluster from using packages to using parcels before starting the upgrade.

Upgrading using packages is for advanced users. Some parts of this procedure require you to run commands from the command line on all cluster hosts. This may require significant time. Additionally, if other software packages are installed on cluster hosts, the upgrade steps described in this topic could update dependencies that affect those packages.

To upgrade CDH using packages:

Step 1: Collect Upgrade Information

Before starting an upgrade, collect the following information:
  1. Host credentials. You must have SSH access and be able to log in using a root account or an account that has password-less sudo permission.
  2. The version of Cloudera Manager used in your cluster. Go to Support > About.
  3. The version of the JDK deployed in the cluster. Go to Support > About.
  4. The version of CDH. The CDH version number displays next to the cluster name on the Home page.
  5. Whether the cluster was installed using parcels or packages. This information displays next to the CDH version on the Home page of Cloudera Manager.
  6. The services enabled in your cluster. Go to Clusters > Cluster name.
  7. Operating system type and version. Go to Hosts and click on a hostname in the list. The operating system type and version displays in the Distribution row in the Details section.
  8. Database information for the databases used by Sqoop, Oozie, Hue, Hive Metastore, and Sentry Server (information is only required if theses services are enabled in the cluster).
    Gather the following information:
    • Type of database (PostgreSQL, Embedded PostgreSQL, MySQL, MariaDB, or Oracle)
    • Hostnames of the databases
    • Credentials for the databases
    To locate database information:
    • Sqoop, Oozie, and Hue – Go to Cluster Name > Configuration > Database Settings.
    • Hive Metastore – Go to the Hive service, select Configuration, and select the Hive Metastore Database category.
    • Sentry – Go to the Sentry service, select Configuration, and select the Sentry Server Database category.

Step 2: Complete Pre-Upgrade Steps

Step 3: Upgrade Unmanaged Components

Step 4: Stop Cluster Services

  1. On the Home > Status tab, click to the right of the cluster name and select Stop.
  2. Click Stop in the confirmation screen. The Command Details window shows the progress of stopping services.

    When All services successfully stopped appears, the task is complete and you can close the Command Details window.

Step 5: Back Up Databases

  Note: Backing up databases requires that you stop some services, which may make them unavailable during backup.
Back up the databases for any of the following services that are deployed in your cluster:
Table 1. Service Databases to Back Up
Service Where to find database information
Sqoop Go to Clusters > Cluster Name > Sqoop service > Configuration and select the Database category.
Hue Go to Clusters > Cluster Name > Hue service > Configuration and select the Database category.
Oozie Go to Clusters > Cluster Name > Oozie service > Configuration and select the Database category.
Cloudera Navigator Audit Server Go to Clusters > Cloudera Management Service > Configuration and select the Database category.
Cloudera Navigator Metadata Server Go to Clusters > Cloudera Management Service > Configuration and select the Database category.
Activity Monitor Go to Clusters > Cloudera Management Service > Configuration and select the Database category.
Reports Manager Go to Clusters > Cloudera Management Service > Configuration and select the Database category.
Sentry Server Go to Clusters > Cluster Name > Sentry service > Configuration and select the Sentry Server Database category.
Hive Metastore Go to Clusters > Cluster Name > Hive service > Configuration and select the Hive Metastore Database category.
To back up the databases:
  1. If not already stopped, stop the service:
    1. On the Home > Status tab, click to the right of the service name and select Stop.
    2. Click Stop in the next screen to confirm. When you see a Finished status, the service has stopped.
  2. Back up the database. See Backing Up Databases for detailed instructions for each supported type of database.
  3. Restart the service:
    1. On the Home > Status tab, click to the right of the service name and select Start.
    2. Click Start that appears in the next screen to confirm. When you see a Finished status, the service has started.

Step 6: Establish Access to the Software

In order to upgrade CDH, Cloudera Manager needs access to a repository that contains the updated packages. By default, Cloudera Manager selects the most recent version of CDH for the upgrade. If you want to upgrade to a different version, or Cloudera Manager does not have access to the Internet, you must set up a local repository. See Creating and Using a Package Repository for Cloudera Manager.

Step 7: Upgrade Managed Components

Use one of the following strategies to upgrade CDH 5:

Use the Cloudera 1-Click Package

Using the Cloudera "1-click Install" package is the simplest way to upgrade only the Cloudera packages.
  1. Check whether you have the CDH 5 "1-click" repository installed by running the following command on each cluster host:
    RHEL/CentOS-compatible and SLES
    rpm -q CDH 5-repository

    If you are upgrading from CDH 5 Beta 1 or higher, and you used the "1-click" package for the previous CDH 5 release, you should see:

    CDH5-repository-1-0

    In this case, skip to Step install_upgrade_to_cdh5x_packages.html#concept_umd_wfj_rx_unique_2__install-packages_unique_2, Install the CDH packages. If instead you see:

    package CDH 5-repository is not installed

    proceed with Step install_upgrade_to_cdh5x_packages.html#concept_umd_wfj_rx_unique_2__install-1-click_unique_2, Install the 1-click package.

    Ubuntu and Debian
    dpkg -l | grep CDH 5-repository

    If the repository is installed, skip to Step install_upgrade_to_cdh5x_packages.html#concept_umd_wfj_rx_unique_2__install-packages_unique_2, Install the CDH packages; otherwise proceed with Step install_upgrade_to_cdh5x_packages.html#concept_umd_wfj_rx_unique_2__install-1-click_unique_2, Install the 1-click package.

  2. Install the CDH5 "1-click" package. If the CDH 5 "1-click" repository is not already installed on each host in the cluster, follow the instructions below for that host's operating system.
    RHEL compatible
    1. Download and install the "1-click Install" package on each cluster host.
      1. Download the CDH 5 "1-click Install" package (or RPM).

        Click the appropriate RPM and Save File to a directory with write access (for example, your home directory).

        OS Version Link to CDH 5 RPM
        RHEL/CentOS/Oracle 6 RHEL/CentOS/Oracle 6 link
        RHEL/CentOS/Oracle 7 RHEL/CentOS/Oracle 7 link
      2. Install the RPM for all RHEL versions: 
        $ sudo yum --nogpgcheck localinstall cloudera-cdh-5-0.x86_64.rpm
    2. Optionally add a repository key. Using a repository key helps to verify that you are using a signed version of the package. If you add a repository key, you can omit the --nogpgcheck option when running yum commands. Run the following command to add the repository key:
      Red Hat/CentOS/Oracle 5
      $ sudo rpm --import https://archive.cloudera.com/cdh5/redhat/5/x86_64/cdh/RPM-GPG-KEY-cloudera
      Red Hat/CentOS/Oracle 6
      $ sudo rpm --import https://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera
    SLES
    1. Download and install the "1-click Install" package on each cluster host:
      1. Download the CDH 5 "1-click Install" package.

        Download the RPM file, choose Save File, and save it to a directory to which you have write access (for example, your home directory).

      2. Install the RPM: 
        $ sudo rpm -i cloudera-cdh-5-0.x86_64.rpm
      3. Update your system package index by running the following: 
        $ sudo zypper refresh
    2. Optionally add a repository key. Using a repository key helps to verify that you are using a signed version of the package. 
      $ sudo rpm --import https://archive.cloudera.com/cdh5/sles/11/x86_64/cdh/RPM-GPG-KEY-cloudera
    Ubuntu and Debian
    1. Download and install the "1-click Install" package on each cluster host:
      1. Download the CDH 5 "1-click Install" package:
        OS Version Package Link
        Jessie Jessie package
        Wheezy Wheezy package
        Precise Precise package
        Trusty Trusty package
      2. Install the package by doing one of the following:
        • Choose Open with in the download window to use the package manager.
        • Choose Save File, save the package to a directory to which you have write access (for example, your home directory), and install it from the command line. For example: 
          sudo dpkg -i cdh5-repository_1.0_all.deb
    2. (Optionally) add a repository key (Using a repository key helps to verify that you are using a signed version of the package.):
      • Ubuntu Trusty 
        $ curl -s https://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh/archive.key | sudo apt-key add -
      • Ubuntu Precise 
        $ curl -s https://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh/archive.key | sudo apt-key add -
      • Debian Wheezy 
        $ curl -s https://archive.cloudera.com/cdh5/debian/wheezy/amd64/cdh/archive.key | sudo apt-key add -
  3. Install the CDH packages by running the following command on all cluster hosts:
      Note:
    • Installing these packages also installs all the other CDH packages required for a full CDH 5 installation.
    • Kudu is only supported on a subset of the operating systems supported by CDH. These are: RHEL/CentOS 6, RHEL/CentOS 7, SLES 12 SP1, Ubuntu 14.04, 16.04, Debian 8.2, 8.4. Remove the kudu package from the command if you are installing on any other operating systems.
    RHEL compatible
    $ sudo yum clean all
$ sudo yum install avro-tools crunch flume-ng hadoop-hdfs-fuse hadoop-httpfs hadoop-kms hbase hbase-solr hive-hbase hive-webhcat hue-beeswax hue-hbase hue-impala hue-pig hue-plugins hue-rdbms hue-search hue-spark hue-sqoop hue-zookeeper impala impala-shell kudu kite llama oozie parquet pig pig-udf-datafu search sentry solr solr-mapreduce spark-python sqoop sqoop2 zookeeper
    SLES
    $ sudo zypper clean --all
$ sudo zypper install avro-tools crunch flume-ng hadoop-hdfs-fuse hadoop-httpfs hadoop-kms hbase hbase-solr hive-hbase hive-webhcat hue-beeswax hue-hbase hue-impala hue-pig hue-plugins hue-rdbms hue-search hue-spark hue-sqoop hue-zookeeper impala impala-shell kudu kite llama oozie parquet pig pig-udf-datafu search sentry solr solr-mapreduce spark-python sqoop sqoop2 zookeeper
    Ubuntu and Debian
    $ sudo apt-get update
$ sudo apt-get install avro-tools crunch flume-ng hadoop-hdfs-fuse hadoop-httpfs hadoop-kms hbase hbase-solr hive-hbase hive-webhcat hue-beeswax hue-hbase hue-impala hue-pig hue-plugins hue-rdbms hue-search hue-spark hue-sqoop hue-zookeeper impala impala-shell kudu kite llama oozie parquet pig pig-udf-datafu search sentry solr solr-mapreduce spark-python sqoop sqoop2 zookeeper

Use Operating System Package Management Tools

  • Use your operating system's package management tools to update all packages to the latest version using standard repositories. This approach minimizes the amount of configuration required and uses the simplest commands. This can take a considerable amount of time if you have not upgraded the system recently. To update all packages on your system, run the following command on each cluster host:
    RHEL
    $ sudo yum update
    SLES
    $ sudo zypper up
    Ubuntu or Debian
    $ sudo apt-get upgrade

Use a Specific Set of Packages

To upgrade managed components to a specific version of CDH, specify the packages you want to use for the upgrade. Follow the procedure at Upgrade Managed Components Using a Specific Set of Packages and then continue with the procedures in this topic.

Step 8: Update Symlinks for the Newly Installed Components

Restart the Cloudera Manager Agents to force an update of the symlinks to point to the newly installed components on each host. Run the following command on all cluster hosts: 
$ sudo service cloudera-scm-agent restart

Step 9: Run the Upgrade Wizard

  1. If you cluster has Kudu installed, stop the Kudu services before performing the steps in this section.
  2. Log in to the Cloudera Manager Admin console.
  3. From the Home > Status tab, click next to the cluster name and select Upgrade Cluster. The Upgrade Wizard starts.
  4. In the Choose Method field, select the Use Packages option.
  5. In the Choose CDH Version (Packages) field, specify the CDH version of the packages you have installed on your cluster. Click Continue.
  6. Read the notices for steps you must complete before upgrading, click the Yes, I ... checkboxes after completing the steps, and click Continue.
  7. Cloudera Manager checks that cluster hosts have the correct software installed. If the packages have not been installed, a warning displays to that effect. Install the missing packages and click Retry. When there are no errors, click Continue.
  8. The Host Inspector runs. Correct any errors displayed and click Continue.

    The Choose Upgrade Procedure screen displays the available types of upgrades:

    • Full Cluster Restart - Cloudera Manager performs all service upgrades and restarts the cluster.
    • Manual upgrade Cloudera Manager configures the cluster to the specified CDH version but performs no upgrades or service restarts. Manually upgrading is difficult and for advanced users only. To perform a manual upgrade:
      1. Select the Let me upgrade the cluster checkbox.
      2. Click Continue.
      3. See Performing Upgrade Wizard Actions Manually for the required steps.
  9. Select Full Cluster Restart.
  10. Click Continue. The Upgrade Cluster Command screen displays the result of the commands run by the wizard as it shuts down all services, upgrades services, deploys client configuration files, and restarts services. If any of the steps fails or if you click the Abort button, the Retry button at the top right is enabled.

    Click Retry to retry the step and continue the wizard, or click the Cloudera Manager logo to return to the Home > Status tab and manually perform the failed step and all following steps. See Performing Upgrade Wizard Actions Manually.

  11. Click Continue. The wizard reports the result of the upgrade.
  12. Click Finish to return to the Home page.

Step 10: Recover from Failed Steps or Perform a Manual Upgrade

The actions performed by the upgrade wizard are listed in Performing Upgrade Wizard Actions Manually. If any of the steps in the Upgrade Cluster Command screen fail, complete the steps as described in that section before proceeding.

Step 11: Finalize the HDFS Metadata Upgrade

[Not required for CDH maintenance release upgrades.]

The steps in this section are required for all upgrade from CDH 5.x to CDH 6.x.

To determine if you can finalize, run important workloads and ensure that they are successful. Once you have finalized the upgrade, you cannot roll back to a previous version of HDFS without using backups. Verifying that you are ready to finalize the upgrade can take a long time.

Make sure you have enough free disk space, keeping in mind that the following behavior continues until the upgrade is finalized:
  • Deleting files does not free up disk space.
  • Using the balancer causes all moved replicas to be duplicated.
  • All on-disk data representing the NameNodes metadata is retained, which could more than double the amount of space required on the NameNode and JournalNode disks.
To finalize the metadata upgrade:
  1. Go to the HDFS service.
  2. Click the Instances tab.
  3. Select the NameNode instance. If you have enabled high availability for HDFS, select NameNode (Active).
  4. Select Actions > Finalize Metadata Upgrade and click Finalize Metadata Upgrade to confirm.

Step 12: Exit Maintenance Mode

If you entered maintenance mode during this upgrade, exit maintenance mode.

Page generated March 7, 2018.