Using loader-backup to Back Up Job Data

Scenario

After a job is created using the Loader WebUI or loader-tool, use loader-backup to back up data.

  • Only Loader jobs of data export support data backup.
  • This tool is an internal Loader interface and is invoked by the upper-layer component HBase. Only the data backup from HDFS to SFTP is supported.

Prerequisites

The Loader client has been installed and configured.

Procedure

  1. Log in to the node where the client is installed as the user who installs the client.
  2. Run the following command to go to the directory where the backup.properties file is located. For example, if the Loader client installation directory is /opt/hadoopclient/Loader, run the following command:

    cd /opt/hadoopclient/Loader/loader-tools-1.99.3/loader-backup/conf

  3. Run the following command to modify the configuration parameters of backup.properties. For details about the parameters, see Table 1.

    vi backup.properties

    server.url = 10.0.0.1:21351,10.0.0.2:12000
authentication.type = kerberos
authentication.user =
authentication.password=
job.jobId = 1
use.keytab = true
client.principal = loader/hadoop
client.keytab = /opt/client/conf/loader.keytab
    Table 1 Configuration parameters

    Configuration parameters

    Description

    Example Value

    server.url

    Floating IP address and port (21351) for Loader.

    For compatibility, multiple IP addresses and ports can be configured and need to be separated by commas (,). The first IP address and port must be those of Loader (21351). The others can be configured based on service requirements.

    10.0.0.1:21351,10.0.0.2:12000

    authentication.type

    Login authentication mode.

    • kerberos indicates that the security mode is used and Kerberos authentication is performed. Kerberos authentication provides two authentication modes: the password mode and the keytab file mode.
    • simple indicates that the normal mode is used and Kerberos authentication is not performed.

    kerberos

    authentication.user

    User for login when the normal mode or password authentication is used.

    In the keytab login mode, this parameter does not need to be set.

    bar

    authentication.password

    User password for login when the password authentication mode is used.

    In the normal mode or keytab login mode, this parameter does not need to be set.

    The password needs to be encrypted. The encryption method is described as follows:

    1. Go to the directory where encrypt_tool is located. For example, if the Loader client installation directory is /opt/hadoopclient/Loader, run the following command:

      cd /opt/hadoopclient/Loader/loader-tools-1.99.3

    2. Run the following command to encrypt the non-encrypted password:

      ./encrypt_tool Unencrypted password

      The obtained encrypted password is used as the value of authentication.password.

      NOTE:

      If a non-encrypted password contains special characters, the special characters must be escaped. For example, the dollar sign ($) is a special character and can be escaped using single quotation marks ('), for example, '1q2w#e$r'. If a non-encrypted password contains single quotation marks, use double quotation marks to escape the single quotation marks. If a non-encrypted password contains double quotation marks, use backslashes (\) to escape the double quotation marks. For details, see the shell escape character rules.

    43B80E33A96DF3D203ABBDFD1050C041

    job.jobId

    ID of the job whose data is to be backed up.

    Job IDs can be viewed under created jobs on the Loader web UI.

    1

    use.keytab

    Whether to use the keytab mode to log in.

    • true indicates using the keytab file to log in.
    • false indicates using the password to log in.

    true

    client.principal

    User principal for accessing the Loader service when the keytab authentication mode is used.

    In the normal mode or password login mode, this parameter does not need to be set.

    loader/hadoop

    client.keytab

    Directory where the used keytab file is located when the keytab authentication mode is used.

    In the normal mode or password login mode, this parameter does not need to be set.

    /opt/client/conf/loader.keytab

  4. Run the following command to go to the directory where the backup script run.sh is located. For example, if the Loader client installation directory is /opt/hadoopclient/Loader, run the following command:

    cd /opt/hadoopclient/Loader/loader-tools-1.99.3/loader-backup

  5. Run the following command to run the backup script run.sh to back up Loader job data. The system backs up data to a directory at the same layer of the job output directory.

    ./run.sh Backup data input directory

    For example, the backup data input directory is /user/hbase/, and the job output directory is/opt/client/sftp/sftp1. sftp1 acts as a placeholder. Run the following command to back up data to the /opt/client/sftp/hbase directory:

    ./run.sh /user/hbase/

Parent topic: Client Tool Description