Using Environment Variables in the properties.properties File

Scenario

This section describes how to use environment variables in the properties.properties configuration file.

Prerequisites

The Flume service is running properly and the Flume client has been installed.

Procedure

  1. Log in to the node where the Flume client is installed as user root.
  2. Switch to the following directory:

    cd Flume client installation directory/fusioninsight-flume-Flume component version/conf

  3. Add environment variables to the flume-env.sh file in the directory.

    • Format:
      export Variable name=Variable value
    • Example:
      JAVA_OPTS="-Xms2G -Xmx4G -XX:CMSFullGCsBeforeCompaction=1 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -DpropertiesImplementation=org.apache.flume.node.EnvVarResolverProperties"
export TAILDIR_PATH=/tmp/flumetest/201907/20190703/1/.*log.*

  4. Restart the Flume instance process.

    1. Log in to FusionInsight Manager.
    2. Choose Cluster > Services > Flume. On the page that is displayed, click the Instance tab, select all Flume instances, and choose More > Restart Instance. In the displayed Verify Identity dialog box, enter the password, and click OK.

    Do not restart the Flume service on FusionInsight Manager after flume-env.sh takes effect on the server. Otherwise, the user-defined environment variables will lost. You only need to restart the corresponding instances on FusionInsight Manager.

  5. In the Flume client installation directory/fusioninsight-flume-Flume component version number/conf/properties.properties configuration file, reference variables in the ${Variable name} format. The following is an example:

    client.sources.s1.type = TAILDIR
client.sources.s1.filegroups = f1
client.sources.s1.filegroups.f1 = ${TAILDIR_PATH}
client.sources.s1.positionFile = /tmp/flumetest/201907/20190703/1/taildir_position.json
client.sources.s1.channels = c1
    • Ensure that flume-env.sh takes effect before you go to 5 to configure the properties.properties file.
    • If you configure file on the local host, upload the file on FusionInsight Manager by performing the following steps. The user-defined environment variables may be lost if the operations are not performed in the correct sequence.
      1. Log in to FusionInsight Manager.
      2. Choose Cluster > Services > Flume. On the page that is displayed, click the Configurations tab, select the Flume instance, and click Upload File next to flume.config.file to upload the properties.properties file.

Parent topic: Using Flume