Local data is continuously uploaded to DIS.
Data can be stored in DIS, OBS, and DLI. For details about how to configure a storage location, see Creating a Dump Task.
+
Data can be stored in DIS, OBS, and DLI. For how to configure a storage location, see Creating a Dump Task.
The maximum number of days for DIS to preserve data cannot exceed Data Retention (days).
diff --git a/docs/dis/umn/dis_faq_0001.html b/docs/dis/umn/dis_faq_0001.html
new file mode 100644
index 00000000..70d8f99b
--- /dev/null
+++ b/docs/dis/umn/dis_faq_0001.html
@@ -0,0 +1,14 @@
+
+
+
What Is DIS?
+
Data Ingestion Service (DIS) addresses the challenge of transmitting data from outside the cloud to inside the cloud. DIS builds data intake streams for custom applications capable of processing or analyzing streaming data. DIS continuously captures, transmits, and stores terabytes of data from hundreds of thousands of sources every hour, such as logs, Internet of Things (IoT) data, social media feeds, website clickstreams, and location-tracking events.
+
Data Flows
- DIS collects data from multiple data sources in real time.
- DIS transforms data continuously to OBS for computing, analysis, and storage.
+
+
Figure 1 Data flows
+
+
+
diff --git a/docs/dis/umn/dis_faq_0002.html b/docs/dis/umn/dis_faq_0002.html
new file mode 100644
index 00000000..2143db53
--- /dev/null
+++ b/docs/dis/umn/dis_faq_0002.html
@@ -0,0 +1,13 @@
+
+
+
What Is a Partition?
+
Partitions are the base throughput unit of a DIS stream. When creating a DIS stream, you are expected to specify the number of partitions needed within your stream.
+
By default, each tenant has 1 to 50 partitions. Tenants can configure the number of partitions as required.
+
To increase the quota, submit a service ticket. The upper limit is calculated based on the actual load of the cluster.
+
+
+
diff --git a/docs/dis/umn/dis_faq_0003.html b/docs/dis/umn/dis_faq_0003.html
new file mode 100644
index 00000000..028545e4
--- /dev/null
+++ b/docs/dis/umn/dis_faq_0003.html
@@ -0,0 +1,15 @@
+
+
+
How Do I Send and Retrieve Data Using DIS?
+
- Create a DIS stream. Obtain your Access Key ID/Secret Access Key (AK/SK) pair from the Identity and Access Management (IAM) service.
- Download and decompress the DIS SDK software package.
Verify integrity of the DIS SDK software package by following the procedure in How Do I Check Software Package Integrity?
+
+ - Create a producer project and configure the user AK/SK, endpoint, project ID, region, stream name, and the number of partitions.
- Run the producer application to send data.
- Create a consumer project and configure the user AK/SK, endpoint, project, region, stream name, partition ID, and startingSequenceNumber.
- Run the consumer application to retrieve data.
+
+
+
diff --git a/docs/dis/umn/dis_faq_0004.html b/docs/dis/umn/dis_faq_0004.html
new file mode 100644
index 00000000..0c959167
--- /dev/null
+++ b/docs/dis/umn/dis_faq_0004.html
@@ -0,0 +1,12 @@
+
+
+
What Can I Do with DIS?
+
You can use DIS for rapid data intake from producers and continuous data processing. The following are typical scenarios for using DIS:
+
- Accelerated log or data transmission: Users do not need to wait for batch data processing. Data is put into a DIS stream immediately after it is generated by a data producer, preventing data loss in case of faults in the data producer. For example, system and application logs can be continuously put into a stream and processed within seconds.
- Real-time metrics and reporting: You can retrieve data from DIS streams for simple data analysis and reporting in real time. For example, your DIS applications can work on metrics and reporting for system and application logs as streaming data is being pushed to DIS streams using application programming interfaces (APIs), rather than wait to receive batches of data.
- Real-time data analysis: DIS combines the power of parallel processing with the value of real-time data. For example, you can transform data into valuable information and business intelligence by simply putting data into a DIS stream. This can be done in minutes instead of hours or days.
- Complex stream processing: You can create Directed Acyclic Graphs (DAGs) of DIS applications and streams. This typically involves putting data from one or multiple DIS applications into another stream for downstream processing by a different DIS application.
+
+
+
diff --git a/docs/dis/umn/dis_faq_0005.html b/docs/dis/umn/dis_faq_0005.html
new file mode 100644
index 00000000..5bf33518
--- /dev/null
+++ b/docs/dis/umn/dis_faq_0005.html
@@ -0,0 +1,11 @@
+
+
+
What Advantages Does DIS Have?
+
- Scalable: A DIS stream can seamlessly scale its data throughput from megabytes to terabytes per hour, and from thousands to millions of PUT records per second.
- Easy to use: You can create a DIS stream within seconds. It is easy to put data into a stream, and build a data processing application.
- Low cost: DIS has no upfront cost and you only pay for the resources you use.
- Parallel processing: DIS allows you to have multiple DIS applications processing the same stream concurrently. For example, you can have one application running real-time analytics and another sending data to OBS from the same stream.
- DIS preserves data for N*24 hours, reducing the probability of data loss in case of application failures, individual machine failures, or facility failures. The value of N is an integer from 1 to 7.
+
+
+
diff --git a/docs/dis/umn/dis_faq_0008.html b/docs/dis/umn/dis_faq_0008.html
new file mode 100644
index 00000000..5c3d7b72
--- /dev/null
+++ b/docs/dis/umn/dis_faq_0008.html
@@ -0,0 +1,15 @@
+
+
+
Which Modules Do DIS Have?
+
- Service control
- Creates, deletes, and configures DIS streams; synchronizes user information to the data plane
- Allocates resources on the data plane and automatically deploys DIS
+ - Data processing
- Receives user requests; receives and stores authenticated data
- Receives data read requests and returns the requested data to authorized users
- Removes old data from DIS streams according to data aging policies
- Stores user data into Object Storage Service (OBS) according to user options
+ - Service maintenance
- Installs and upgrades DIS
- Performs configuration, preventive maintenance, monitoring, and log collection and analysis for DIS
- Processes service orders
+ - User SDK
- Provides Java APIs for users to push and pull data
- Encrypts data
+
+
+
+
diff --git a/docs/dis/umn/dis_faq_0009.html b/docs/dis/umn/dis_faq_0009.html
new file mode 100644
index 00000000..45b3700d
--- /dev/null
+++ b/docs/dis/umn/dis_faq_0009.html
@@ -0,0 +1,22 @@
+
+
+
How Do I Check Software Package Integrity?
+
This section describes how to verify integrity of the DIS SDK software package on a Linux system by using a verification file.
+
Prerequisites
- The PuTTY tool is available.
- The WinSCP tool is available.
+
+
Procedure
- Upload the DIS SDK software package sdk-dis-x.x.x1.2.3.zip to any directory on the Linux system by using WinSCP.
In sdk-dis-x.x.x.zip, x.x.x indicates the version number of the DIS SDK software package.
+
+ - Log in to the Linux system by using PuTTY. In the directory in which sdk-dis-x.x.x1.2.3.zip is stored, run the following command to obtain the verification code of the DIS SDK software package
sha256sum sdk-dis-x.x.x.zip
+Example verification code:
+# sha256sum dis-sdk-x.x.x.zip
+8be2c937e8d78b1a9b99777cee4e7131f8bf231de3f839cf214e7c5b5ba3c088 cloud-sdk-dis-x.x.x.zip
+ - Open the DIS SDK verification file sdk-dis-x.x.x1.2.3.zip.sha256sum and compare it with the verification code obtained in Step 2.
- If they are consistent, the DIS-SDK compression package has not been tampered with.
- If they are inconsistent, the DIS SDK software package is tampered with and you need to obtain it again.
+
+
+
+
+
diff --git a/docs/dis/umn/dis_faq_0011.html b/docs/dis/umn/dis_faq_0011.html
new file mode 100644
index 00000000..aaeaf957
--- /dev/null
+++ b/docs/dis/umn/dis_faq_0011.html
@@ -0,0 +1,28 @@
+
+
+
General Questions
+
+
+
diff --git a/docs/dis/umn/dis_faq_0012.html b/docs/dis/umn/dis_faq_0012.html
new file mode 100644
index 00000000..bcb8aa05
--- /dev/null
+++ b/docs/dis/umn/dis_faq_0012.html
@@ -0,0 +1,19 @@
+
+
+
Dump Questions
+
+
+
diff --git a/docs/dis/umn/dis_faq_0013.html b/docs/dis/umn/dis_faq_0013.html
new file mode 100644
index 00000000..e2800425
--- /dev/null
+++ b/docs/dis/umn/dis_faq_0013.html
@@ -0,0 +1,19 @@
+
+
+
DIS Agent Questions
+
+
+
diff --git a/docs/dis/umn/dis_faq_0014.html b/docs/dis/umn/dis_faq_0014.html
new file mode 100644
index 00000000..5325c7ed
--- /dev/null
+++ b/docs/dis/umn/dis_faq_0014.html
@@ -0,0 +1,27 @@
+
+
+
How Can I Configure Agent to Listen To Multiple Directories or Files?
+
DIS Agent can listen to multiple directories or files. For example, to collect logs of the /home/folder1/file1 and /home/folder2/file2 files, configure multiple DIS streams.
+
---
+region: REGION
+ak: YOUR_AK
+sk: YOUR_SK
+projectId: YOUR_PROJECTID
+endpoint: ENDPOINT
+flows:
+ - DISStream: YOUR_STREAM
+ filePattern: /home/folder1/file1
+ initialPosition: START_OF_FILE
+ maxBufferAgeMillis: 5000
+
+ - DISStream: YOUR_STREAM
+ filePattern: /home/folder2/file2
+ initialPosition: START_OF_FILE
+ maxBufferAgeMillis: 5000
+
+
+
diff --git a/docs/dis/umn/dis_faq_0015.html b/docs/dis/umn/dis_faq_0015.html
new file mode 100644
index 00000000..91d8f05c
--- /dev/null
+++ b/docs/dis/umn/dis_faq_0015.html
@@ -0,0 +1,23 @@
+
+
+
How Can I Configure Recursive Listening for Directories on DIS Agent?
+
On DIS Agent, set directoryRecursionEnabled to true. For example, the following configuration can match /home/one.log, /home/child/two.log, and /home/child/child/three.log:
+
---
+region: REGION
+ak: YOUR_AK
+sk: YOUR_SK
+projectId: YOUR_PROJECTID
+endpoint: ENDPOINT
+flows:
+ - DISStream: YOUR_STREAM
+ filePattern: /home/*.log
+ directoryRecursionEnabled: true
+ initialPosition: START_OF_FILE
+ maxBufferAgeMillis: 5000
+
+
+
diff --git a/docs/dis/umn/dis_faq_0017.html b/docs/dis/umn/dis_faq_0017.html
new file mode 100644
index 00000000..59dab6c0
--- /dev/null
+++ b/docs/dis/umn/dis_faq_0017.html
@@ -0,0 +1,23 @@
+
+
+
How Does a Schema Support Default Fields or NULL Fields?
+
A source data schema is a user's JSON data sample used to describe the JSON data format. DIS can generate the Avro schema based on the JSON data sample. By default, the default field or NULL is not supported, as shown in Figure 1.
+
Figure 1 Example of not supporting the default field
+
The type of the key1 field is String (type: string in Avro Schema). If the key1 field in the source data is not transferred or the transferred value is null, the dump task reports an error.
+
To enable the schema generated based on the JSON data sample supports the default value or null, select the Convert and Move and click Convert Source Data Sample, as shown in Figure 2.
+
Figure 2 Example of supporting the default field
+
In this case, the type of the key1 field is Union (type: [null, string] in Avro Schema). If the key1 field in the source data is not transferred or the transferred value is null, null is automatically filled as the default value. The dump task can properly convert the format.
+
+
+
+
+
\ No newline at end of file
diff --git a/docs/dis/umn/dis_faq_0019.html b/docs/dis/umn/dis_faq_0019.html
new file mode 100644
index 00000000..7358378b
--- /dev/null
+++ b/docs/dis/umn/dis_faq_0019.html
@@ -0,0 +1,11 @@
+
+
+
How Do I Distinguish Different Types of Data When Accessing Data in Streams?
+
- By stream: Different types of data use different streams and can be distinguished by stream.
- By partition: Different types of data use different partitions on one stream. Each partition is assigned one partition key. Different types of data can be distinguished by partition key of each partition.
+
+
+
diff --git a/docs/dis/umn/dis_faq_0021.html b/docs/dis/umn/dis_faq_0021.html
new file mode 100644
index 00000000..730ba8c7
--- /dev/null
+++ b/docs/dis/umn/dis_faq_0021.html
@@ -0,0 +1,14 @@
+
+
+
How Do I Use Agent to Encrypt AK/SK?
+
Secret Access Key (SK) is sensitive information. To encrypt the SK, perform the following steps:
+
- Go to the bin/ directory.
cd /opt/dis-agent-X.X.X/bin
+ - Run the encryption script, enter the password, and press Enter.
bash dis-encrypt.sh
+ - View the encryption result. The character string following "Encrypt result:" displayed on the console is the encryption result. Use this method to encrypt the MySQL password and SK, respectively and record the ciphertext in the configuration file.
+
+
+
diff --git a/docs/dis/umn/dis_faq_0025.html b/docs/dis/umn/dis_faq_0025.html
new file mode 100644
index 00000000..4ca1fa3c
--- /dev/null
+++ b/docs/dis/umn/dis_faq_0025.html
@@ -0,0 +1,23 @@
+
+
+
How Can I Configure Recursive Listening for Directories on DIS Agent?
+
On DIS Agent, set directoryRecursionEnabled to true. For example, the following configuration can match /home/one.log, /home/child/two.log, and /home/child/child/three.log:
+
---
+region: REGION
+ak: YOUR_AK
+sk: YOUR_SK
+projectId: YOUR_PROJECTID
+endpoint: ENDPOINT
+flows:
+ - DISStream: YOUR_STREAM
+ filePattern: /home/*.log
+ directoryRecursionEnabled: true
+ initialPosition: START_OF_FILE
+ maxBufferAgeMillis: 5000
+
+
+
diff --git a/docs/dis/umn/en-us_image_0000001199305476.png b/docs/dis/umn/en-us_image_0000001199305476.png
new file mode 100644
index 00000000..3e6cfaaf
Binary files /dev/null and b/docs/dis/umn/en-us_image_0000001199305476.png differ
diff --git a/docs/dis/umn/en-us_image_0000001274606310.png b/docs/dis/umn/en-us_image_0000001274606310.png
new file mode 100644
index 00000000..d220666d
Binary files /dev/null and b/docs/dis/umn/en-us_image_0000001274606310.png differ
diff --git a/docs/dis/umn/en-us_image_0000001274783064.png b/docs/dis/umn/en-us_image_0000001274783064.png
new file mode 100644
index 00000000..a5c51c4a
Binary files /dev/null and b/docs/dis/umn/en-us_image_0000001274783064.png differ
diff --git a/docs/dis/umn/en-us_image_0000001325286261.png b/docs/dis/umn/en-us_image_0000001325286261.png
new file mode 100644
index 00000000..a5c51c4a
Binary files /dev/null and b/docs/dis/umn/en-us_image_0000001325286261.png differ
diff --git a/docs/dis/umn/en-us_image_0000001325622881.png b/docs/dis/umn/en-us_image_0000001325622881.png
new file mode 100644
index 00000000..d220666d
Binary files /dev/null and b/docs/dis/umn/en-us_image_0000001325622881.png differ
diff --git a/docs/dis/umn/en-us_image_0109727712.png b/docs/dis/umn/en-us_image_0109727712.png
new file mode 100644
index 00000000..e08d33a6
Binary files /dev/null and b/docs/dis/umn/en-us_image_0109727712.png differ
diff --git a/docs/dis/umn/en-us_image_0130822560.png b/docs/dis/umn/en-us_image_0130822560.png
new file mode 100644
index 00000000..95de1237
Binary files /dev/null and b/docs/dis/umn/en-us_image_0130822560.png differ
diff --git a/docs/dis/umn/en-us_image_0130854869.jpg b/docs/dis/umn/en-us_image_0130854869.jpg
new file mode 100644
index 00000000..e4657d7c
Binary files /dev/null and b/docs/dis/umn/en-us_image_0130854869.jpg differ
diff --git a/docs/dis/umn/en-us_image_0200877126.png b/docs/dis/umn/en-us_image_0200877126.png
new file mode 100644
index 00000000..ba0a2e87
Binary files /dev/null and b/docs/dis/umn/en-us_image_0200877126.png differ
diff --git a/docs/dis/umn/en-us_image_0200879188.png b/docs/dis/umn/en-us_image_0200879188.png
new file mode 100644
index 00000000..a1bbe3a3
Binary files /dev/null and b/docs/dis/umn/en-us_image_0200879188.png differ
diff --git a/docs/dis/umn/en-us_topic_0124374673.html b/docs/dis/umn/en-us_topic_0124374673.html
index badd291b..a73fc039 100644
--- a/docs/dis/umn/en-us_topic_0124374673.html
+++ b/docs/dis/umn/en-us_topic_0124374673.html
@@ -1,5 +1,15 @@
FAQs
-
This chapter describes some FAQs.
+
+
indicates that notification is enabled and 
indicates that notification is disabled. By default, notification is disabled. After notification is disabled, the system stops sending notifications of subscribed events but does not delete the subscription.
indicates that subscription to alarms of a specified stream is enabled. 
indicates that subscription to alarms of a specified channel is disabled. By default, the function is disabled.
. The filtered event query results are displayed.
on the right of Event and select an event name, for example, Dump task restored to filter the corresponding events.