2023-09-07
diff --git a/docs/dli/umn/dli_01_0003.html b/docs/dli/umn/dli_01_0003.html index fe5fa05c..35c6b2bf 100644 --- a/docs/dli/umn/dli_01_0003.html +++ b/docs/dli/umn/dli_01_0003.html @@ -8,9 +8,9 @@
In cross-source development scenarios, there is a risk of password leakage if datasource authentication information is directly configured. You are advised to use the datasource authentication provided by DLI. For details, see Overview.
Constraints
- Datasource connections cannot be created for the default queue.
- Flink jobs can directly access DIS, OBS, and SMN data sources without using datasource connections.
- VPC Administrator permissions are required for enhanced connections to use VPCs, subnets, routes, VPC peering connections.
- If you use an enhanced datasource connection, the CIDR block of the elastic resource pool or queue cannot overlap with that of the data source.
- Only queues bound with datasource connections can access datasource tables.
- Datasource tables do not support the preview function.
- When checking the connectivity of datasource connections, the constraints on IP addresses are as follows:
- The IP address must be valid, which consists of four decimal numbers separated by periods (.). The value ranges from 0 to 255.
- During the test, you can add a port after the IP address and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535.
For example, 192.168.xx.xx or 192.168.xx.xx:8181.
+diff --git a/docs/dli/umn/dli_01_0005.html b/docs/dli/umn/dli_01_0005.html index b077a7b4..6f1a0004 100644 --- a/docs/dli/umn/dli_01_0005.html +++ b/docs/dli/umn/dli_01_0005.html @@ -52,9 +52,9 @@Constraints
- Datasource connections cannot be created for the default queue.
- Flink jobs can directly access DIS, OBS, and SMN data sources without using datasource connections.
- VPC Administrator permissions are required for enhanced connections to use VPCs, subnets, routes, VPC peering connections.
- If you use an enhanced datasource connection, the CIDR block of the elastic resource pool or queue cannot overlap with that of the data source.
- Only queues bound with datasource connections can access datasource tables.
- Datasource tables do not support the preview function.
- When checking the connectivity of datasource connections, the notes and constraints on IP addresses are:
- The IP address must be valid, which consists of four decimal numbers separated by periods (.). The value ranges from 0 to 255.
- During the test, you can add a port after the IP address and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535.
- When checking the connectivity of datasource connections, the constraints on domain names are as follows:
- The domain name can contain 1 to 255 characters. Only letters, digits, underscores (_), and hyphens (-) are allowed.
- The top-level domain name must contain at least two letters, for example, .com, .net, and .cn.
- During the test, you can add a port after the domain name and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535. +
- When checking the connectivity of datasource connections, the notes and constraints on domain names are:
- The domain name can contain 1 to 255 characters. Only letters, digits, underscores (_), and hyphens (-) are allowed.
- The top-level domain name must contain at least two letters, for example, .com, .net, and .cn.
- During the test, you can add a port after the domain name and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535.
Creating a Table
Before creating a table, ensure that a database has been created.
-- You can create a table on either the Databases and Tables page or the SQL Editor page.
Datasource connection tables, such as View tables, HBase (MRS) tables, OpenTSDB (MRS) tables, GaussDB(DWS) tables, RDS tables, and CSS tables, cannot be created. You can use SQL to create views and datasource connection tables. For details, see sections Creating a View and Creating a Datasource Connection Table in the Data Lake Insight SQL Syntax Reference.
+- You can create a table on either the Databases and Tables page or the SQL Editor page.-
Datasource connection tables, such as View tables, HBase (MRS) tables, OpenTSDB (MRS) tables, GaussDB(DWS) tables, RDS tables, and CSS tables, cannot be created. You can use SQL to create views and datasource connection tables. For details, see sections Creating a View and Creating a Datasource Connection Table in the Data Lake Insight SQL Syntax Reference.
- To create a table on the Data Management page:
- On the left of the management console, choose Data Management > Databases and Tables.
- On the Databases and Tables page, select the database for which you want to create a table. In the Operation column, click More > Create Table to create a table in the current database.
- To create a table on the Data Management page:
- On the left of the management console, choose Data Management > Databases and Tables.
- On the Databases and Tables page, select the database for which you want to create a table. In the Operation column, click More > Create Table to create a table in the current database.
- To create a table on the SQL Editor page:
- On the left of the management console, click SQL Editor.
- In the navigation pane of the displayed SQL Editor page, click Databases. You can create a table in either of the following ways:
- Click a database name. In the Tables area, click
on the right to create a table in the current database. - Click
on the right of the database and choose Create Table from the shortcut menu to create a table in the current database.
- Click a database name. In the Tables area, click
Type
Data type of a column. This parameter corresponds to Column Name.
-- string: The data is of the string type.
- int: Each integer is stored on four bytes.
- date: The value ranges from 0000-01-01 to 9999-12-31.
- double: Each number is stored on eight bytes.
- boolean: Each value is stored on one byte.
- decimal: The valid bits are positive integers between 1 to 38, including 1 and 38. The decimal digits are integers less than 10.
- smallint/short: The number is stored on two bytes.
- bigint/long: The number is stored on eight bytes.
- timestamp: The data indicates a date and time. The value can be accurate to six decimal points.
- float: Each number is stored on four bytes.
- tinyint: Each number is stored on one byte. Only OBS tables support this data type.
- string: The data is of the string type.
- int: Each integer is stored on four bytes.
- date: The value ranges from 0000-01-01 to 9999-12-31.
- double: Each number is stored on eight bytes.
- boolean: Each value is stored on one byte.
- decimal: The valid bits are positive integers between 1 to 38, including 1 and 38. The decimal digits are integers less than 10.
- smallint/short: The number is stored on two bytes.
- bigint/long: The number is stored on eight bytes.
- timestamp: The data indicates a date and time. The value can be accurate to six decimal points.
- float: Each number is stored on four bytes.
- tinyint: Each number is stored on one byte. Only OBS tables support this data type.
string
Data Format
DLI supports the following data formats:
-- Parquet: DLI can read non-compressed data or data that is compressed using Snappy and gzip.
- CSV: DLI can read non-compressed data or data that is compressed using gzip.
- ORC: DLI can read non-compressed data or data that is compressed using Snappy.
- JSON: DLI can read non-compressed data or data that is compressed using gzip.
- Avro: DLI can read uncompressed Avro data.
- Parquet: DLI can read non-compressed data or data that is compressed using Snappy and gzip.
- CSV: DLI can read non-compressed data or data that is compressed using gzip.
- ORC: DLI can read non-compressed data or data that is compressed using Snappy.
- JSON: DLI can read non-compressed data or data that is compressed using gzip.
- Avro: DLI can read uncompressed Avro data.
CSV
User-defined Quotation Character
This parameter is valid only when Data Format is set to CSV and you select User-defined Quotation Character.
+This parameter is valid only when Data Format is set to CSV and you select User-defined Quotation Character.
The following quotation characters are supported:
- Single quotation mark (')
- Double quotation marks (")
- Others: Enter a user-defined quotation character.
- Click OK.
After a table is created, you can view and select the table for use on the Data Management page or SQL Editor page.
+- Click OK.
After a table is created, you can view and select the table for use on the Data Management page or SQL Editor page.
- (Optional) After a DLI table is created, you can decide whether to directly import data to the table.
diff --git a/docs/dli/umn/dli_01_0006.html b/docs/dli/umn/dli_01_0006.html index b1e3aecf..0973a4bb 100644 --- a/docs/dli/umn/dli_01_0006.html +++ b/docs/dli/umn/dli_01_0006.html @@ -5,9 +5,9 @@For example, to connect DLI to the MRS, RDS, CSS, Kafka, or GaussDB(DWS) data source, you need to enable the network between DLI and the VPC of the data source.
Create an enhanced datasource connection on the console.
-diff --git a/docs/dli/umn/dli_01_0012.html b/docs/dli/umn/dli_01_0012.html index b6ccd14f..88f5d69f 100644 --- a/docs/dli/umn/dli_01_0012.html +++ b/docs/dli/umn/dli_01_0012.html @@ -16,7 +16,7 @@Constraints
- Datasource connections cannot be created for the default queue.
- Flink jobs can directly access DIS, OBS, and SMN data sources without using datasource connections.
- VPC Administrator permissions are required for enhanced connections to use VPCs, subnets, routes, VPC peering connections.
- If you use an enhanced datasource connection, the CIDR block of the elastic resource pool or queue cannot overlap with that of the data source.
- Only queues bound with datasource connections can access datasource tables.
- Datasource tables do not support the preview function.
- When checking the connectivity of datasource connections, the constraints on IP addresses are as follows:
- The IP address must be valid, which consists of four decimal numbers separated by periods (.). The value ranges from 0 to 255.
- During the test, you can add a port after the IP address and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535.
For example, 192.168.xx.xx or 192.168.xx.xx:8181.
+@@ -137,7 +137,6 @@Constraints
- Datasource connections cannot be created for the default queue.
- Flink jobs can directly access DIS, OBS, and SMN data sources without using datasource connections.
- VPC Administrator permissions are required for enhanced connections to use VPCs, subnets, routes, VPC peering connections.
- If you use an enhanced datasource connection, the CIDR block of the elastic resource pool or queue cannot overlap with that of the data source.
- Only queues bound with datasource connections can access datasource tables.
- Datasource tables do not support the preview function.
- When checking the connectivity of datasource connections, the notes and constraints on IP addresses are:
- The IP address must be valid, which consists of four decimal numbers separated by periods (.). The value ranges from 0 to 255.
- During the test, you can add a port after the IP address and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535.
- When checking the connectivity of datasource connections, the constraints on domain names are as follows:
- The domain name can contain 1 to 255 characters. Only letters, digits, underscores (_), and hyphens (-) are allowed.
- The top-level domain name must contain at least two letters, for example, .com, .net, and .cn.
- During the test, you can add a port after the domain name and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535. +
- When checking the connectivity of datasource connections, the notes and constraints on domain names are:
- The domain name can contain 1 to 255 characters. Only letters, digits, underscores (_), and hyphens (-) are allowed.
- The top-level domain name must contain at least two letters, for example, .com, .net, and .cn.
- During the test, you can add a port after the domain name and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535.
- In the navigation pane of the DLI management console, choose Resources > Queue Management.
- Locate the queue bound with the enhanced datasource connection, click More in the Operation column, and select Test Address Connectivity.
- Enter the data source connection address and port number to test the network connectivity.
Format: IP address:Port number
- Before testing the connection, ensure that the security group of the external data source has allowed access from the CIDR block of the queue.
- Modifying the CIDR Block
-
- Elastic Queue Scaling
+- Elastic Scaling of Queues
- Scheduling CU Changes
diff --git a/docs/dli/umn/dli_01_0013.html b/docs/dli/umn/dli_01_0013.html index 62cd1b36..c2d3396b 100644 --- a/docs/dli/umn/dli_01_0013.html +++ b/docs/dli/umn/dli_01_0013.html @@ -1,6 +1,6 @@ -
Modifying Host Information
+Modifying Host Information in an Elastic Resource Pool
diff --git a/docs/dli/umn/dli_01_0228.html b/docs/dli/umn/dli_01_0228.html index 1d48ede9..f75ecdbe 100644 --- a/docs/dli/umn/dli_01_0228.html +++ b/docs/dli/umn/dli_01_0228.html @@ -5,7 +5,7 @@Scenario
Host information is the mapping between host IP addresses and domain names. After you configure host information, jobs can only use the configured domain names to access corresponding hosts. After a datasource connection is created, you can modify the host information.
When accessing the HBase cluster of MRS, you need to configure the host name (domain name) and IP address of the instance.
- Database Permission Management
- Table Permission Management
- Creating a Database or a Table
- Deleting a Database or a Table
- Changing the Owners of Databases and Tables
- Importing Data
- Exporting Data
- Viewing Metadata
- Previewing Data
-Difference Between DLI Tables and OBS Tables
- Data stored in DLI tables is applicable to delay-sensitive services, such as interactive queries.
- Data stored in OBS tables is applicable to delay-insensitive services, such as historical data statistics and analysis.
Constraints
- Database
- default is the database built in DLI. You cannot create a database named default.
- DLI supports a maximum of 50 databases.
Notes and Constraints
- Database
- default is the database built in DLI. You cannot create a database named default.
- DLI supports a maximum of 50 databases.
- Table
- DLI supports a maximum of 5,000 tables.
- DLI supports the following table types:
- MANAGED: Data is stored in a DLI table.
- EXTERNAL: Data is stored in an OBS table.
- View: A view can only be created using SQL statements.
- Datasource table: The table type is also EXTERNAL.
- You cannot specify a storage path when creating a DLI table.
- Data import
- Only OBS data can be imported to DLI or OBS.
- You can import data in CSV, Parquet, ORC, JSON, or Avro format from OBS to tables created on DLI.
- To import data in CSV format to a partitioned table, place the partition column in the last column of the data source.
- The encoding format of imported data can only be UTF-8.
Last Accessed
Last time when an operation was performed on the table.
+The last access time of a table refers only to the last time it was updated, not the time it was read (SELECT operation).
Operation
diff --git a/docs/dli/umn/dli_01_0318.html b/docs/dli/umn/dli_01_0318.html index 1d7930df..2d5289e3 100644 --- a/docs/dli/umn/dli_01_0318.html +++ b/docs/dli/umn/dli_01_0318.html @@ -3,465 +3,535 @@DLI Operations That Can Be Recorded by CTS
With CTS, you can record operations associated with DLI for later query, audit, and backtrack operations.
-Table 1 DLI operations that can be recorded by CTS Operation
++Table 1 DLI operations that can be recorded by CTS Operation
Resource Type
+Resource Type
Trace Name
Creating a database
+Creating a database
database
+database
createDatabase
Deleting a database
+Deleting a database
database
+database
deleteDatabase
Modifying the Database Owner
+Changing the database owner
database
+database
alterDatabaseOwner
Creating a table
+Creating a table
table
+table
createTable
Deleting tables
+Deleting tables
table
+table
deleteTable
Exporting table data
+Exporting table data
table
+table
exportData
Importing table data
+Importing table data
table
+table
importData
Modifying the owner of a table
+Modifying the owner of a table
table
+table
alterTableOwner
Creating a queue
+Creating a queue
queue
+queue
createQueue
Deleting a queue
+Deleting a queue
queue
+queue
dropQueue
Granting permissions to a queue
+Granting permissions to a queue
queue
+queue
shareQueue
Modifying a Queue CIDR Block
+Modifying the CIDR block of a queue
queue
+queue
replaceQueue
Restarting a queue
+Restarting a queue
queue
+queue
queueActions
Scaling out/in a queue
+Scaling out/in a queue
queue
+queue
queueActions
Submitting a job (SQL)
+Submitting a job (SQL)
queue
+queue
submitJob
Canceling a job (SQL)
+Canceling a job (SQL)
queue
+queue
cancelJob
Granting DLI the permission to access OBS buckets
+Granting DLI the permission to access OBS buckets
obs
+obs
obsAuthorize
Checking the SQL syntax
+Checking the SQL syntax
job
+job
checkSQL
Creating a job
+Creating a job
job
+job
createJob
Updating a job
+Updating a job
job
+job
updateJob
Deleting a job
+Deleting a job
job
+job
deleteJob
Purchasing CUH packages
+Creating a Flink OpenSource SQL job
order
+jobs
+createStreamSqlJob
+Updating a Flink OpenSource SQL job
+jobs
+updateStreamSqlJob
+Deleting a Flink job
+jobs
+deleteStreamJobs
+Stopping a Flink job
+jobs
+stopStreamJobs
+Submitting a Flink job
+jobs
+submitStreamJobs
+Creating a Flink Jar job
+jobs
+createStreamJarJob
+Updating a Flink Jar job
+jobs
+updateStreamJarJob
+Checking Flink jobs
+jobs
+checkStreamJob
+Importing a savepoint
+jobs
+dealSavepoint
+Purchasing CUH packages
+order
orderPackage
Freezing resources
+Freezing resources
resource
+resource
freezeResource
Unfreezing resources
+Unfreezing resources
resource
+resource
unfreezeResource
Terminating resources
+Terminating resources
resource
+resource
deleteResource
Clearing resources
+Clearing resources
resource
+resource
cleanResource
Granting data permissions
+Granting data permissions
data
+data
dataAuthorize
Granting permissions on other projects
+Granting permissions on other projects
data
+data
authorizeProjectData
Exporting query results
+Exporting query results
data
+data
storeJobResult
Saving a SQL template
+Saving a SQL template
sqlTemplate
+sqlTemplate
saveSQLTemplate
Updating a SQL template
+Updating a SQL template
sqlTemplate
+sqlTemplate
updateSQLTemplate
Deleting a SQL template
+Deleting a SQL template
sqlTemplate
+sqlTemplate
deleteSQLTemplate
Creating a Flink template
+Creating a Flink template
flinkTemplate
+template
createStreamTemplate
Updating a Flink template
+Updating a Flink template
flinkTemplate
+template
createStreamTemplate
+updateStreamTemplate
Deleting a Flink template
+Checking Flink templates
flinkTemplate
+template
+checkStreamTemplate
+Deleting a Flink template
+flinkTemplate
deleteStreamTemplate
Creating a data upload task
+Creating a data upload task
uploader
+uploader
createUploadJob
Obtaining the authentication to perform a data upload task
+Obtaining the authentication to perform a data upload task
uploader
+uploader
getUploadAuthInfo
Submitting a data upload task
+Submitting a data upload task
uploader
+uploader
commitUploadJob
Creating a datasource authentication and uploading a certificate
+Creating a datasource authentication and uploading a certificate
authInfo
+authInfo
uploadAuthInfo
Updating a datasource authentication
+Updating a datasource authentication
authInfo
+authInfo
updateAuthInfop
Deleting a datasource authentication
+Deleting a datasource authentication
authInfo
+authInfo
deleteAuthInfo
Updating the quota
+Updating the quota
quota
+quota
updateQuota
Uploading a resource package
+Uploading a resource package
pkgResource
+pkgResource
uploadResources
Deleting a resource package
+Deleting a resource package
pkgResource
+pkgResource
deleteResource
Creating a basic datasource connection
+Creating a basic datasource connection
datasource
+datasource
createDatasourceConn
Deleting a basic datasource connection
+Deleting a basic datasource connection
datasource
+datasource
deleteDatasourceConn
Reactivating a basic datasource connection
+Reactivating a basic datasource connection
datasource
+datasource
reactivateDSConnection
Creating an enhanced datasource connection
+Creating an enhanced datasource connection
datasource
+datasource
createConnection
Deleting an enhanced datasource connection
+Deleting an enhanced datasource connection
datasource
+datasource
getConnection
Binding a queue
+Binding a queue
datasource
+datasource
associateQueueToDatasourceConn
Unbinding a queue
+Unbinding a queue
datasource
+datasource
disassociateQueueToDatasourceConn
Modifying the host information
+Modifying the host information
datasource
+datasource
updateHostInfo
Adding a route
+Adding a route
datasource
+datasource
addRoute
Deleting a route
+Deleting a route
datasource
+datasource
deleteRoute
Creating a topic
+Creating a topic
smn
+smn
createTopic
Creating an agency
+Creating an agency
agency
+agency
createAgencyV2
Creating a batch processing job
+Creating a batch processing job
batch
+batch
createBatch
Canceling a batch processing job
+Canceling a batch processing job
batch
+batch
cancelBatch
Creating a session
+Creating a session
session
+session
createSession
Deleting a session
+Deleting a session
session
+session
deleteSession
Creating a statement
+Creating a statement
statement
+statement
createStatement
Canceling execution of a statement
+Canceling execution of a statement
statement
+statement
cancelStatement
Creating a global variable
+Creating a global variable
globalVar
+globalVar
createGlobalVariable
Deleting a global variable
+Deleting a global variable
globalVar
+globalVar
deleteGlobalVariable
Modifying a global variable
+Modifying a global variable
globalVar
+globalVar
updateGlobalVariable
- Select a queue from the queue list in the upper left corner of the SQL job editing window. For details about how to create a queue, see Creating a Queue.
- In the upper right corner of the SQL job editing window, select a database, for example, qw, from the Databases drop-down list.
- Create a table, for example, qw. For details about how to create a database and table, see Creating a Database or a Table.
- In the SQL job editing window, enter the following SQL statement:
-1SELECT * FROM qw.qw LIMIT 10;
Alternatively, you can double-click the table name qw. The query statement is automatically entered in the SQL job editing window.
- On top of the editing window, click More > Verify Syntax to check whether the SQL statement is correct.
- If the verification fails, check the SQL statement syntax by referring to Data Lake Insight SQL Syntax Reference.
- If the syntax verification is successful, click Execute. Read and agree to the privacy agreement. Click OK to execute the SQL statement.
- After the execution is complete, you can view the execution result in the area under the SQL job editing window.
On the Overview page, click Create Job in the upper right corner of the Spark Jobs tab or click Create Job in the upper right corner of the Spark Jobs page. The Spark job editing page is displayed.
On the Spark job editing page, a message is displayed, indicating that a temporary DLI data bucket will be created. The created bucket is used to store temporary data generated by DLI, such as job logs and job results. You cannot view job logs if you choose not to create it. The bucket will be created and the default bucket name is used.
If you do not need to create a DLI temporary data bucket and do not want to receive this message, select Do not show again and click Cancel.
-Prerequisites
- You have uploaded the dependencies to the corresponding OBS bucket on the Data Management > Package Management page. For details, see Creating a Package.
- Before creating a Spark job to access other external data sources, such as OpenTSDB, HBase, Kafka, GaussDB(DWS), RDS, CSS, CloudTable, DCS Redis, and DDS, you need to create a datasource connection to enable the network between the job running queue and external data sources.
- For details about the external data sources that can be accessed by Spark jobs, see Cross-Source Analysis Development Methods.
- For details about how to create a datasource connection, see Enhanced Datasource Connections.
On the Resources > Queue Management page, locate the queue you have created, and choose More > Test Address Connectivity in the Operation column to check whether the network connection between the queue and the data source is normal. For details, see Testing Address Connectivity.
+diff --git a/docs/dli/umn/dli_01_0397.html b/docs/dli/umn/dli_01_0397.html index a6e22a68..df3b613a 100644 --- a/docs/dli/umn/dli_01_0397.html +++ b/docs/dli/umn/dli_01_0397.html @@ -2556,156 +2556,161 @@Prerequisites
- You have uploaded the dependencies to the corresponding OBS bucket on the Data Management > Package Management page. For details, see Creating a Package.
- Before creating a Spark job to access other external data sources, such as OpenTSDB, HBase, Kafka, GaussDB(DWS), RDS, CSS, CloudTable, DCS Redis, and DDS, you need to create a datasource connection to enable the network between the job running queue and external data sources.
- For details about the external data sources that can be accessed by Spark jobs, see Cross-Source Analysis Development Methods.
- For details about how to create a datasource connection, see Enhanced Datasource Connections.
On the Resources > Queue Management page, locate the queue you have created, click More in the Operation column, and select Test Address Connectivity to check if the network connection between the queue and the data source is normal. For details, see Testing Address Connectivity.
Flink 1.15 Dependencies
Obtain information about the Flink 1.15 dependencies from the logs of a Flink job.
+- Check the logs of a Flink job.
- Log in to the DLI console. In the navigation pane on the left, choose Job Management > Flink Jobs.
- Click the name of the desired job. On the displayed page, click the Run Log tab.
- Check the latest run logs. For more logs, check the OBS bucket where the job logs are stored.
- Search for dependency information in the logs.
Search for Classpath: in the logs to check the dependencies.
+
Flink 1.12 Dependencies
-Table 4 Flink 1.12 dependencies Dependency
+Table 4 Flink 1.12 dependencies Dependency
bcpkix-jdk15on-1.60.jar
+bcpkix-jdk15on-1.60.jar
flink-json-1.12.2-ei-313001-dli-2022011002.jar
+flink-json-1.12.2-ei-313001-dli-2022011002.jar
libtensorflow-1.12.0.jar
+libtensorflow-1.12.0.jar
bcprov-jdk15on-1.60.jar
+bcprov-jdk15on-1.60.jar
flink-kubernetes_2.11-1.12.2-ei-313001-dli-2022011002.jar
+flink-kubernetes_2.11-1.12.2-ei-313001-dli-2022011002.jar
log4j-1.2-api-2.17.1.jar
+log4j-1.2-api-2.17.1.jar
clickhouse-jdbc-0.3.1-ei-313001-SNAPSHOT.jar
+clickhouse-jdbc-0.3.1-ei-313001-SNAPSHOT.jar
flink-metrics-prometheus_2.11-1.12.2-ei-313001-dli-2022011002.jar
+flink-metrics-prometheus_2.11-1.12.2-ei-313001-dli-2022011002.jar
log4j-api-2.17.1.jar
+log4j-api-2.17.1.jar
commons-codec-1.9.jar
+commons-codec-1.9.jar
flink-obs-hadoop-fs-2.0.0-20220226.034421-73.jar
+flink-obs-hadoop-fs-2.0.0-20220226.034421-73.jar
log4j-core-2.17.1.jar
+log4j-core-2.17.1.jar
commons-configuration-1.7.jar
+commons-configuration-1.7.jar
flink-s3-fs-hadoop-1.12.2.jar
+flink-s3-fs-hadoop-1.12.2.jar
log4j-slf4j-impl-2.17.1.jar
+log4j-slf4j-impl-2.17.1.jar
dataflow-fs-obs-2.0.0-20220226.034402-190.jar
+dataflow-fs-obs-2.0.0-20220226.034402-190.jar
flink-shaded-zookeeper-3.6.3-ei-313001-SNAPSHOT.jar
+flink-shaded-zookeeper-3.6.3-ei-313001-SNAPSHOT.jar
luxor-encrypt-2.0.0-20220405.072004-199.jar
+luxor-encrypt-2.0.0-20220405.072004-199.jar
deeplearning4j-core-0.9.1.jar
+deeplearning4j-core-0.9.1.jar
flink-sql-avro-1.12.2-ei-313001-dli-2022011002.jar
+flink-sql-avro-1.12.2-ei-313001-dli-2022011002.jar
luxor-fs3-2.0.0-20220405.072025-195.jar
+luxor-fs3-2.0.0-20220405.072025-195.jar
deeplearning4j-nlp-0.9.1.jar
+deeplearning4j-nlp-0.9.1.jar
flink-sql-avro-confluent-registry-1.12.2-ei-313001-dli-2022011002.jar
+flink-sql-avro-confluent-registry-1.12.2-ei-313001-dli-2022011002.jar
luxor-obs-fs3-2.0.0-20220405.072030-195.jar
+luxor-obs-fs3-2.0.0-20220405.072030-195.jar
deeplearning4j-nn-0.9.1.jar
+deeplearning4j-nn-0.9.1.jar
flink-table_2.11-1.12.2-ei-313001-dli-2022011002.jar
+flink-table_2.11-1.12.2-ei-313001-dli-2022011002.jar
manager-hadoop-security-crypter-8.1.3-313001-SNAPSHOT.jar
+manager-hadoop-security-crypter-8.1.3-313001-SNAPSHOT.jar
ejml-cdense-0.33.jar
+ejml-cdense-0.33.jar
flink-table-blink_2.11-1.12.2-ei-313001-dli-2022011002.jar
+flink-table-blink_2.11-1.12.2-ei-313001-dli-2022011002.jar
manager-wc2frm-8.1.3-313001-SNAPSHOT.jar
+manager-wc2frm-8.1.3-313001-SNAPSHOT.jar
ejml-core-0.33.jar
+ejml-core-0.33.jar
guava-18.0.jar
+guava-18.0.jar
mrs-obs-provider-3.1.1.49.jar
+mrs-obs-provider-3.1.1.49.jar
ejml-ddense-0.33.jar
+ejml-ddense-0.33.jar
guava-26.0-jre.jar
+guava-26.0-jre.jar
nd4j-api-0.9.1.jar
+nd4j-api-0.9.1.jar
ejml-dsparse-0.33.jar
+ejml-dsparse-0.33.jar
hadoop-hdfs-client-3.1.1-ei-302002.jar
+hadoop-hdfs-client-3.1.1-ei-302002.jar
nd4j-native-0.9.1.jar
+nd4j-native-0.9.1.jar
ejml-experimental-0.33.jar
+ejml-experimental-0.33.jar
hadoop-3.1.1-46.jar
+hadoop-3.1.1-46.jar
nd4j-native-api-0.9.1.jar
+nd4j-native-api-0.9.1.jar
ejml-fdense-0.33.jar
+ejml-fdense-0.33.jar
hadoop-plugins-8.1.3-313001-SNAPSHOT.jar
+hadoop-plugins-8.1.3-313001-SNAPSHOT.jar
nd4j-native-platform-0.9.1.jar
+nd4j-native-platform-0.9.1.jar
ejml-simple-0.33.jar
+ejml-simple-0.33.jar
httpasyncclient-4.1.2.jar
+httpasyncclient-4.1.2.jar
okhttp-3.14.8.jar
+okhttp-3.14.8.jar
ejml-zdense-0.33.jar
+ejml-zdense-0.33.jar
httpclient-4.5.3.jar
+httpclient-4.5.3.jar
okio-1.14.0.jar
+okio-1.14.0.jar
elsa-3.0.0-M7.jar
+elsa-3.0.0-M7.jar
httpcore-4.4.4.jar
+httpcore-4.4.4.jar
ranger-obs-client-0.1.1.jar
+ranger-obs-client-0.1.1.jar
flink-changelog-json-1.12.2-ei-313001-dli-2022011002.jar
+flink-changelog-json-1.12.2-ei-313001-dli-2022011002.jar
httpcore-nio-4.4.4.jar
+httpcore-nio-4.4.4.jar
secComponentApi-1.0.5.jar
+secComponentApi-1.0.5.jar
flink-csv-1.12.2-ei-313001-dli-2022011002.jar
+flink-csv-1.12.2-ei-313001-dli-2022011002.jar
java-xmlbuilder-1.1.jar
+java-xmlbuilder-1.1.jar
slf4j-api-1.7.26.jar
+slf4j-api-1.7.26.jar
flink-dist_2.11-1.12.2-ei-313001-dli-2022011002.jar
+flink-dist_2.11-1.12.2-ei-313001-dli-2022011002.jar
jna-4.1.0.jar
+jna-4.1.0.jar
tensorflow-1.12.0.jar
+tensorflow-1.12.0.jar
Flink 1.10 Dependencies
Only queues created after December 2020 can use the Flink 1.10 dependencies.
--Table 5 Flink 1.10 dependencies Dependency
+-Table 5 Flink 1.10 dependencies Dependency
bcpkix-jdk15on-1.60.jar
+bcpkix-jdk15on-1.60.jar
esdk-obs-java-3.20.6.1.jar
+esdk-obs-java-3.20.6.1.jar
java-xmlbuilder-1.1.jar
+java-xmlbuilder-1.1.jar
bcprov-jdk15on-1.60.jar
+bcprov-jdk15on-1.60.jar
flink-cep_2.11-1.10.0.jar
+flink-cep_2.11-1.10.0.jar
jna-4.1.0.jar
+jna-4.1.0.jar
commons-codec-1.9.jar
+commons-codec-1.9.jar
flink-cep-scala_2.11-1.10.0.jar
+flink-cep-scala_2.11-1.10.0.jar
libtensorflow-1.12.0.jar
+libtensorflow-1.12.0.jar
commons-configuration-1.7.jar
+commons-configuration-1.7.jar
flink-dist_2.11-1.10.0.jar
+flink-dist_2.11-1.10.0.jar
log4j-over-slf4j-1.7.26.jar
+log4j-over-slf4j-1.7.26.jar
deeplearning4j-core-0.9.1.jar
+deeplearning4j-core-0.9.1.jar
flink-python_2.11-1.10.0.jar
+flink-python_2.11-1.10.0.jar
logback-classic-1.2.3.jar
+logback-classic-1.2.3.jar
deeplearning4j-nlp-0.9.1.jar
+deeplearning4j-nlp-0.9.1.jar
flink-queryable-state-runtime_2.11-1.10.0.jar
+flink-queryable-state-runtime_2.11-1.10.0.jar
logback-core-1.2.3.jar
+logback-core-1.2.3.jar
deeplearning4j-nn-0.9.1.jar
+deeplearning4j-nn-0.9.1.jar
flink-sql-client_2.11-1.10.0.jar
+flink-sql-client_2.11-1.10.0.jar
nd4j-api-0.9.1.jar
+nd4j-api-0.9.1.jar
ejml-cdense-0.33.jar
+ejml-cdense-0.33.jar
flink-state-processor-api_2.11-1.10.0.jar
+flink-state-processor-api_2.11-1.10.0.jar
nd4j-native-0.9.1.jar
+nd4j-native-0.9.1.jar
ejml-core-0.33.jar
+ejml-core-0.33.jar
flink-table_2.11-1.10.0.jar
+flink-table_2.11-1.10.0.jar
nd4j-native-api-0.9.1.jar
+nd4j-native-api-0.9.1.jar
ejml-ddense-0.33.jar
+ejml-ddense-0.33.jar
flink-table-blink_2.11-1.10.0.jar
+flink-table-blink_2.11-1.10.0.jar
nd4j-native-platform-0.9.1.jar
+nd4j-native-platform-0.9.1.jar
ejml-dsparse-0.33.jar
+ejml-dsparse-0.33.jar
guava-26.0-jre.jar
+guava-26.0-jre.jar
okhttp-3.14.8.jar
+okhttp-3.14.8.jar
ejml-experimental-0.33.jar
+ejml-experimental-0.33.jar
hadoop-3.1.1-41.jar
+hadoop-3.1.1-41.jar
okio-1.14.0.jar
+okio-1.14.0.jar
ejml-fdense-0.33.jar
+ejml-fdense-0.33.jar
httpasyncclient-4.1.2.jar
+httpasyncclient-4.1.2.jar
secComponentApi-1.0.5.jar
+secComponentApi-1.0.5.jar
ejml-simple-0.33.jar
+ejml-simple-0.33.jar
httpclient-4.5.3.jar
+httpclient-4.5.3.jar
slf4j-api-1.7.26.jar
+slf4j-api-1.7.26.jar
ejml-zdense-0.33.jar
+ejml-zdense-0.33.jar
httpcore-4.4.4.jar
+httpcore-4.4.4.jar
tensorflow-1.12.0.jar
+tensorflow-1.12.0.jar
elsa-3.0.0-M7.jar
+elsa-3.0.0-M7.jar
httpcore-nio-4.4.4.jar
+httpcore-nio-4.4.4.jar
-
+-
Flink 1.7.2 Dependencies
--Table 6 Flink 1.7.2 dependencies Dependency
+-Table 6 Flink 1.7.2 dependencies Dependency
bcpkix-jdk15on-1.60.jar
+bcpkix-jdk15on-1.60.jar
esdk-obs-java-3.1.3.jar
+esdk-obs-java-3.1.3.jar
httpcore-4.4.4.jar
+httpcore-4.4.4.jar
bcprov-jdk15on-1.60.jar
+bcprov-jdk15on-1.60.jar
flink-cep_2.11-1.7.0.jar
+flink-cep_2.11-1.7.0.jar
httpcore-nio-4.4.4.jar
+httpcore-nio-4.4.4.jar
commons-codec-1.9.jar
+commons-codec-1.9.jar
flink-cep-scala_2.11-1.7.0.jar
+flink-cep-scala_2.11-1.7.0.jar
java-xmlbuilder-1.1.jar
+java-xmlbuilder-1.1.jar
commons-configuration-1.7.jar
+commons-configuration-1.7.jar
flink-dist_2.11-1.7.0.jar
+flink-dist_2.11-1.7.0.jar
jna-4.1.0.jar
+jna-4.1.0.jar
deeplearning4j-core-0.9.1.jar
+deeplearning4j-core-0.9.1.jar
flink-gelly_2.11-1.7.0.jar
+flink-gelly_2.11-1.7.0.jar
libtensorflow-1.12.0.jar
+libtensorflow-1.12.0.jar
deeplearning4j-nlp-0.9.1.jar
+deeplearning4j-nlp-0.9.1.jar
flink-gelly-scala_2.11-1.7.0.jar
+flink-gelly-scala_2.11-1.7.0.jar
log4j-over-slf4j-1.7.21.jar
+log4j-over-slf4j-1.7.21.jar
deeplearning4j-nn-0.9.1.jar
+deeplearning4j-nn-0.9.1.jar
flink-ml_2.11-1.7.0.jar
+flink-ml_2.11-1.7.0.jar
logback-classic-1.2.3.jar
+logback-classic-1.2.3.jar
ejml-cdense-0.33.jar
+ejml-cdense-0.33.jar
flink-python_2.11-1.7.0.jar
+flink-python_2.11-1.7.0.jar
logback-core-1.2.3.jar
+logback-core-1.2.3.jar
ejml-core-0.33.jar
+ejml-core-0.33.jar
flink-queryable-state-runtime_2.11-1.7.0.jar
+flink-queryable-state-runtime_2.11-1.7.0.jar
nd4j-api-0.9.1.jar
+nd4j-api-0.9.1.jar
ejml-ddense-0.33.jar
+ejml-ddense-0.33.jar
flink-shaded-curator-1.7.0.jar
+flink-shaded-curator-1.7.0.jar
nd4j-native-0.9.1.jar
+nd4j-native-0.9.1.jar
ejml-dsparse-0.33.jar
+ejml-dsparse-0.33.jar
flink-shaded-hadoop2-uber-1.7.0.jar
+flink-shaded-hadoop2-uber-1.7.0.jar
nd4j-native-api-0.9.1.jar
+nd4j-native-api-0.9.1.jar
ejml-experimental-0.33.jar
+ejml-experimental-0.33.jar
flink-table_2.11-1.7.0.jar
+flink-table_2.11-1.7.0.jar
nd4j-native-platform-0.9.1.jar
+nd4j-native-platform-0.9.1.jar
ejml-fdense-0.33.jar
+ejml-fdense-0.33.jar
guava-26.0-jre.jar
+guava-26.0-jre.jar
okhttp-3.14.8.jar
+okhttp-3.14.8.jar
ejml-simple-0.33.jar
+ejml-simple-0.33.jar
hadoop-3.1.1-41-20201014.085840-4.jar
+hadoop-3.1.1-41-20201014.085840-4.jar
okio-1.14.0.jar
+okio-1.14.0.jar
ejml-zdense-0.33.jar
+ejml-zdense-0.33.jar
httpasyncclient-4.1.2.jar
+httpasyncclient-4.1.2.jar
slf4j-api-1.7.21.jar
+slf4j-api-1.7.21.jar
elsa-3.0.0-M7.jar
+elsa-3.0.0-M7.jar
httpclient-4.5.12.jar
+httpclient-4.5.12.jar
tensorflow-1.12.0.jar
+tensorflow-1.12.0.jar
- Managing Package Permissions
- Creating a Package
- Deleting a Package
You can delete program packages in batches.
- Modifying the Owner
Constraints
- A package can be deleted, but a package group cannot be deleted.
- The following types of packages can be uploaded:
- JAR: JAR file
- PyFile: User Python file
- File: User file
- ModelFile: User AI model file
Notes and Constraints
- A package can be deleted, but a package group cannot be deleted.
- The following types of packages can be uploaded:
- JAR: JAR file
- PyFile: User Python file
- File: User file
- ModelFile: User AI model file
Package Management Page
diff --git a/docs/dli/umn/dli_01_0420.html b/docs/dli/umn/dli_01_0420.html index ef2561a9..2667210e 100644 --- a/docs/dli/umn/dli_01_0420.html +++ b/docs/dli/umn/dli_01_0420.html @@ -1,13 +1,9 @@Importing Data to a DLI Table
--Importing Data Using OBS
On the DLI management console, you can import data stored on OBS to DLI tables from Data Management > Databases and Tables > Table Management and SQL Editor pages. For details, see Importing Data to the Table.
-Importing Data Using CDM
Use the Cloud Data Migration (CDM) service to import data from OBS to DLI. You need to create a CDM queue first.
- -For details about how to create the queue, see "Migrating Data from OBS to DLI" in the Cloud Data Migration User Guide.
-Pay attention to the following configurations:
-- The VPC to which the DLI account belongs is the same as the VPC of the CDM queue.
- You need to create two links, including a DLI link and an OBS link.
- The format of the file to be transmitted can be CSV or JSON.
Importing Data Using OBS
On the DLI management console, you can import data stored in OBS into DLI tables.
+To import OBS data to a DLI table, either choose Data Management > Databases and Tables in the navigation pane on the left, locate a desired database, and click Tables in the Operation column, or choose SQL Editor in the navigation pane on the left.
+For details, see Importing Data to the Table.
Importing Data Using DIS
Use the Data Ingestion Service (DIS) service to import data to DLI. You need to create a DIS stream.
diff --git a/docs/dli/umn/dli_01_0426.html b/docs/dli/umn/dli_01_0426.html index 0600ec2e..8be3272e 100644 --- a/docs/dli/umn/dli_01_0426.html +++ b/docs/dli/umn/dli_01_0426.html @@ -10,9 +10,11 @@- Creating an Enhanced Datasource Connection
+
- Establishing a Network Connection Between DLI and Resources in a Shared VPC
+- Deleting an Enhanced Datasource Connection
-
- Modifying Host Information
+- Modifying Host Information in an Elastic Resource Pool
- Binding and Unbinding a Queue
diff --git a/docs/dli/umn/dli_01_0454.html b/docs/dli/umn/dli_01_0454.html index ad3e3c99..c10532f0 100644 --- a/docs/dli/umn/dli_01_0454.html +++ b/docs/dli/umn/dli_01_0454.html @@ -17,7 +17,7 @@
If the port of the Kafka server is listened on by the host name, you need to add the mapping between the host name and IP address of the Kafka Broker node to the datasource connection.
- CloudTable as the data input and output channel
To use CloudTable as the data input and output channel, create a cluster in CloudTable and obtain the cluster ID.
- CSS as the output channel
To use CSS as the data output channel, create a cluster in CSS and obtain the cluster's private network address. For details, see in the Cloud Search Service User Guide.
-- DCS as the output channel
To use DCS as the output channel, create a Redis cache instance in DCS and obtain the address used for Flink jobs to connect to the Redis instance. For detailed operations, see in the Distributed Cache Service User Guide.
+- DCS as the output channel
To use DCS as the output channel, create a Redis cache instance in DCS and obtain the address used for Flink jobs to connect to the Redis instance. For details, see "Buying a DCS Redis Instance" in Distributed Cache Service User Guide.
diff --git a/docs/dli/umn/dli_01_0487.html b/docs/dli/umn/dli_01_0487.html index eb29b90b..779e7c28 100644 --- a/docs/dli/umn/dli_01_0487.html +++ b/docs/dli/umn/dli_01_0487.html @@ -1,18 +1,20 @@ -Elastic Queue Scaling
+Elastic Scaling of Queues
-Prerequisites
Elastic scaling can be performed for a newly created queue only when there were jobs running in this queue.
Precautions
- Queues with 16 CUs do not support scale-out or scale-in.
- Queues with 64 CUs do not support scale-in.
- If Status of queue xxx is assigning, which is not available is displayed on the Elastic Scaling page, the queue can be scaled only after the queue resources are allocated.
Notes and Constraints
- Queues with 16 CUs do not support scale-out or scale-in.
- Queues with 64 CUs do not support scale-in.
- If Status of queue xxx is assigning, which is not available is displayed on the Elastic Scaling page, the queue can be scaled only after the queue resources are allocated.
- If there are not enough physical resources, a queue may not be able to scale out to the desired target size.
- The system does not guarantee that a queue will be scaled in to the desired target size. Typically, the system checks the resource usage before scaling in the queue to determine if there is enough space for scaling in. If the existing resources cannot be scaled in according to the minimum scaling step, the queue may not be scaled in successfully or only partially.
The scaling step may vary depending on the resource specifications, usually 16 CUs, 32 CUs, 48 CUs, 64 CUs, etc.
+For example, if the queue size is 48 CUs and job execution uses 18 CUs, the remaining 30 CUs do not meet the requirement for scaling in by the minimum step of 32 CUs. If a scaling in task is executed, it will fail.
+
Scaling Out
If the current queue specifications do not meet service requirements, you can add the number of CUs to scale out the queue.
- Scale-out is time-consuming. After you perform scale-out on the Elastic Scaling page of DLI, wait for about 10 minutes. The duration is related to the CU amount to add. After a period of time, refresh the Queue Management page and check whether values of Specifications and Actual CUs are the same to determine whether the scale-out is successful. Alternatively, on the Job Management page, check the status of the SCALE_QUEUE SQL job. If the job status is Scaling, the queue is being scaled out.
+ Scale-out is time-consuming. After you perform scale-out on the Elastic Scaling page of DLI, wait for about 10 minutes. The duration is related to the CU amount to add. After a period of time, refresh the Queue Management page and check whether values of Specifications and Actual CUs are the same to determine whether the scale-out is successful. Alternatively, on the Job Management page, check the status of the SCALE_QUEUE SQL job. If the job status is Scaling, the queue is being scaled out.
The procedure is as follows:
- On the left of the DLI management console, click Resources > Queue Management.
- Select the queue to be scaled out and choose More > Elastic Scaling in the Operation column.
- On the displayed page, select Scale-out for Operation and set the scale-out amount.
- Click .
Scaling In
If the current queue specifications are too much for your computing service, you can reduce the number of CUs to scale in the queue.
- - Scale-in is time-consuming. After you perform scale-in on the Elastic Scaling page of DLI, wait for about 10 minutes. The duration is related to the CU amount to reduce. After a period of time, refresh the Queue Management page and check whether values of Specifications and Actual CUs are the same to determine whether the scale-in is successful. Alternatively, on the Job Management page, check the status of the SCALE_QUEUE SQL job. If the job status is Scaling, the queue is being scaled in.
- The system may not fully scale in the queue to the target size. If the current queue is in use or the service volume of the queue is large, the scale-in may fail or only partial specifications may be reduced.
- By default, the minimum number of CUs is 16. That is, when the queue specifications are 16 CUs, you cannot scale in the queue.
- Scale-in is time-consuming. After you perform scale-in on the Elastic Scaling page of DLI, wait for about 10 minutes. The duration is related to the CU amount to reduce. After a period of time, refresh the Queue Management page and check whether values of Specifications and Actual CUs are the same to determine whether the scale-in is successful. Alternatively, on the Job Management page, check the status of the SCALE_QUEUE SQL job. If the job status is Scaling, the queue is being scaled in.
- By default, the minimum number of CUs is 16. That is, when the queue specifications are 16 CUs, you cannot scale in the queue.
The procedure is as follows:
- On the left of the DLI management console, click Resources > Queue Management.
- Select the queue to be scaled out, click More in the Operation column, and select Elastic Scaling.
- On the displayed page, select Scale-in for Operation and set the scale-in amount.
- Click .
Testing Address Connectivity
It can be used to test the connectivity between the DLI queue and the peer IP address specified by the user in common scenarios, or the connectivity between the DLI queue and the peer IP address bound to the datasource connection in datasource connection scenarios. The operation is as follows:
-- On the Queue Management page, locate the row containing the target queue, click More in the Operation column, and select Test Address Connectivity.
- On the Test Address Connectivity page, enter the address to be tested. The domain name and IP address are supported, and the port number can be specified.
- Click Test.
If the test address is reachable, a message is displayed on the page, indicating that the address is reachable.
-If the test address is unreachable, the system displays a message indicating that the address is unreachable. Check the network configurations and try again. Network configurations include the VPC peering and the datasource connection. Check whether they have been activated.
+- On the Queue Management page, locate the row containing the target queue, click More in the Operation column, and select Test Address Connectivity.
- On the Test Address Connectivity page, enter the address to be tested. The domain name and IP address are supported, and the port number can be specified.
- Click Test.
If the test address is reachable, a message is displayed on the page, indicating that the address is reachable.
+If the test address is unreachable, the system displays a message indicating that the address is unreachable. Check the network configurations and retry. Network configurations include the VPC peering and the datasource connection. Check whether they have been activated.
diff --git a/docs/dli/umn/dli_01_0504.html b/docs/dli/umn/dli_01_0504.html index 6c85a9c6..2f441848 100644 --- a/docs/dli/umn/dli_01_0504.html +++ b/docs/dli/umn/dli_01_0504.html @@ -21,7 +21,7 @@16–64 CUs
- High reliability and availability are not supported.
- Queue properties and job priorities cannot be set.
- Notebook instances cannot be interconnected with.
For more constraints and limitations on elastic resource pools, see Constraints.
+For more notes and constraints on elastic resource pools, see Notes and Constraints.
This edition is suitable for testing scenarios with low resource consumption and low requirements for resource reliability and availability.
64 CUs or higher
For more constraints and limitations on elastic resource pools, see Constraints.
+For more notes and constraints on elastic resource pools, see Notes and Constraints.
This edition offers powerful computing capabilities, high availability, and flexible resource management. It is suitable for large-scale computing tasks and business scenarios with long-term resource planning needs.
Constraints
- The region of an elastic resource pool cannot be changed.
- Jobs of Flink 1.10 or later can run in elastic resource pools.
- The network segment of an elastic resource pool cannot be changed after being set.
- You can only view the scaling history of resource pools in the last 30 days.
- Elastic resource pools cannot access the Internet.
+
Notes and Constraints
- The region of an elastic resource pool cannot be changed.
- Jobs of Flink 1.10 or later can run in elastic resource pools.
- The network segment of an elastic resource pool cannot be changed after being set.
- You can only view the scaling history of resource pools in the last 30 days.
- Elastic resource pools cannot access the Internet. +
- Changes to elastic resource pool CUs can occur when setting the CU, adding or deleting queues in an elastic resource pool, or modifying the scaling policies of queues in an elastic resource pool, or when the system automatically triggers elastic resource pool scaling. However, in some cases, the system cannot guarantee that the scaling will reach the target CUs as planned.
- If there are not enough physical resources, an elastic resource pool may not be able to scale out to the desired target size.
- The system does not guarantee that an elastic resource pool will be scaled in to the desired target size.
The system checks the resource usage before scaling in the elastic resource pool to determine if there is enough space for scaling in. If the existing resources cannot be scaled in according to the minimum scaling step, the pool may not be scaled in successfully or only partially.
+The scaling step may vary depending on the resource specifications, usually 16 CUs, 32 CUs, 48 CUs, 64 CUs, etc.
+For example, if the elastic resource pool has a capacity of 192 CUs and the queues in the pool are using 68 CUs due to running jobs, the plan is to scale in to 64 CUs.
+When executing a scaling in task, the system determines that there are 124 CUs remaining and scales in by the minimum step of 64 CUs. However, the remaining 60 CUs cannot be scaled in any further. Therefore, after the elastic resource pool executes the scaling in task, its capacity is reduced to 128 CUs.
+
Scenario
Resources too fixed to meet a range of requirements.
@@ -71,11 +76,11 @@
Efficiency
+Scale-out duration
You need to set scaling tasks repeatedly to improve the resource utilization.
+You will need to spend several minutes manually scaling out.
Dynamic scaling can be done in seconds.
+No manual intervention is required, as dynamic scale out can be done in seconds.
Resource utilization
diff --git a/docs/dli/umn/dli_01_0524.html b/docs/dli/umn/dli_01_0524.html index bc54155f..0b2db158 100644 --- a/docs/dli/umn/dli_01_0524.html +++ b/docs/dli/umn/dli_01_0524.html @@ -1,10 +1,7 @@Modifying Specifications
--Scenario
If CUs of a yearly/monthly elastic resource pool cannot meet your service requirements, you can modify the CUs. In this case, you will be charged based on the number of CUs exceeding that of the yearly/monthly elastic resource pool.
-For example, you have purchased an elastic resource pool with 64 CUs, and you find that most time data processing needs 128 CUs. You can add 64 CUs to the elastic resource pool and be billed based on a CU/hour basis. To save more, you can scale up your elastic resource pool to 128 CUs and be billed on a yearly/monthly basis for the 128-CU package.
-Precautions
Currently, only yearly/monthly elastic resource pools can be scaled.
+Scenario
If the current specifications of your elastic resource pool do not meet your service needs, you can modify them using the change specifications function.
Scaling Up
- In the navigation pane on the left of the console, choose Resources > Resource Pool.
- Select the elastic resource pool you want and choose More > Modify Specifications in the Operation column.
- In the Modify Specifications dialog page, set Operation to Scale-out and specify the number of CUs you want to add.
- Confirm the changes and click OK.
- Choose Job Management > SQL Jobs to view the status of the SCALE_POOL SQL job.
If the job status is Scaling, the elastic resource pool is scaling up. Wait until the job status changes to Finished.
Step 1: Prepare a Data Source
In this example, Kafka is the data source.
For more information about Flink job data, see Preparing Flink Job Data.
-Enable DIS to import Kafka data to DLI. For details, see "Buying a Kafka Instance" in the Distributed Message Service Kafka User Guide.
+Enable DIS to import Kafka data to DLI. For details, see "Buying a Kafka Instance" in Distributed Message Service Kafka User Guide.
- Create the dependent Kafka resources.Before creating a Kafka instance, ensure the availability of resources, including a virtual private cloud (VPC), subnet, security group, and security group rules.
- For details about how to create a VPC and subnet, see "Creating a VPC and Subnet" in Virtual Private Cloud User Guide. For details about how to create and use a subnet in an existing VPC, see "Create a Subnet for the VPC" in Virtual Private Cloud User Guide.
- The created VPC and the Kafka instance you will create must be in the same region.
- Retain the default settings unless otherwise specified.
- For details about how to create a security group, see "Creating a Security Group" in the Virtual Private Cloud User Guide. For details about how to add rules to a security group, see "Creating a Subnet for the VPC" in the Virtual Private Cloud User Guide.
Step 2: Prepare a Data Output Channel
To use RDS as the data output channel, create an RDS MySQL instance. For details, see "Getting Started with RDS for MySQL" in Getting Started with Relational Database Service.
- Log in to the RDS management console.
- Select a region in the upper left corner.
- Click Buy DB Instance in the upper right corner of the page and set related parameters. Retain the default values for other parameters.
- Region: Select the region where DLI is located.
- DB Instance Name: Enter rds-dliflink.
- DB Engine: Select MySQL.
- DB Engine Version: Select 8.0.
- DB Instance Type: Select Primary/Standby.
- Storage Type: Cloud SSD may be selected by default.
- Primary AZ: Select a custom AZ.
- Standby AZ: Select a custom AZ.
- Instance Class: Select a class as needed and choose 2 vCPUs | 8 GB.
- Storage Space (GB): Set it to 40.
- VPC: Select the VPC and subnet created in 1.
- Database Port: Enter 3306.
- Security Group: Select the security group created in 1.
- Administrator Password: **** (Keep the password secure. The system cannot retrieve your password.)
- Confirm Password: ****
- Parameter Template: Choose Default-MySQL-8.0.
- Click Next and confirm the specifications.
- Click Submit. The RDS DB instance is created.
- Log in to the MySQL database and create table orders in database flink.Log in to the MySQL instance, click the flink database. On the displayed page, click SQL Window. Enter the following table creation statement in the SQL editing pane to create a table.
CREATE TABLE `flink`.`orders` ( +
- Click Next and confirm the specifications.
- Click Submit. The RDS DB instance is created.
- Log in to the MySQL database and create table orders in database flink.
Log in to the MySQL instance, click the flink database. On the displayed page, click SQL Window. Enter the following table creation statement in the SQL editing pane to create a table.CREATE TABLE `flink`.`orders` ( `order_id` VARCHAR(32) NOT NULL, `order_channel` VARCHAR(32) NULL, `order_time` VARCHAR(32) NULL, @@ -124,7 +124,7 @@ insert into jdbcSink select * from kafkaSource;
- Connect to the Kafka cluster and send the following test data to the Kafka topics:
{"order_id":"202103241000000001", "order_channel":"webShop", "order_time":"2021-03-24 10:00:00", "pay_amount":"100.00", "real_pay":"100.00", "pay_time":"2021-03-24 10:02:03", "user_id":"0001", "user_name":"Alice", "area_id":"330106"} {"order_id":"202103241606060001", "order_channel":"appShop", "order_time":"2021-03-24 16:06:06", "pay_amount":"200.00", "real_pay":"180.00", "pay_time":"2021-03-24 16:10:06", "user_id":"0001", "user_name":"Alice", "area_id":"330106"}-- Run the following SQL statement in the MySQL database to view data in the table:
select * from order;
+- Run the following SQL statement in the MySQL database to view data in the table:
select * from orders;
The following is an example of the execution result copied from the MySQL database:diff --git a/docs/dli/umn/dli_01_0538.html b/docs/dli/umn/dli_01_0538.html index 8d96cd1c..95c382a3 100644 --- a/docs/dli/umn/dli_01_0538.html +++ b/docs/dli/umn/dli_01_0538.html @@ -14,7 +14,7 @@202103241000000001,webShop,2021-03-24 10:00:00,100.0,100.0,2021-03-24 10:02:03,0001,Alice,330106 202103241606060001,appShop,2021-03-24 16:06:06,200.0,180.0,2021-03-24 16:10:06,0001,Alice,330106
- Application Scenarios
-
- Constraints and Limitations
+- Notes and Constraints
- Permissions Management
diff --git a/docs/dli/umn/dli_01_0552.html b/docs/dli/umn/dli_01_0552.html index e8470cee..27082aaa 100644 --- a/docs/dli/umn/dli_01_0552.html +++ b/docs/dli/umn/dli_01_0552.html @@ -53,7 +53,7 @@
Tag key
You can specify the tag key in either of the following ways:
-- Click the text box for tag key and select a predefined tag key from the drop-down list.
To add a predefined tag, you need to create one on TMS and then select it from the Tag key drop-down list. You can click View predefined tags to go to the Predefined Tags page of the TMS console. Then, click Create Tag in the upper corner of the page to create a predefined tag.
+- Click the text box for tag key and select a predefined tag key from the drop-down list.
To add a predefined tag, you need to create one on TMS and then select it from the Tag key drop-down list. You can click View predefined tags to go to the Predefined Tags page of the TMS console. Then, click Create Tag in the upper corner of the page to create a predefined tag.
- Enter a tag key in the text box.NOTE:-
A tag key can contain a maximum of 128 characters. Only letters, digits, spaces, and special characters (_.:=+-@) are allowed, but the value cannot start or end with a space or start with _sys_.
diff --git a/docs/dli/umn/dli_01_0561.html b/docs/dli/umn/dli_01_0561.html index bfbee214..2bd52901 100644 --- a/docs/dli/umn/dli_01_0561.html +++ b/docs/dli/umn/dli_01_0561.html @@ -5,7 +5,7 @@- DEW is a comprehensive cloud-based encryption service that addresses data security, key security, and complex key management issues. You are advised to use DEW to store authentication information for data sources.
- Datasource authentication is used to manage authentication information for accessing specified data sources. After datasource authentication is configured, you do not need to repeatedly configure data source authentication information in jobs, improving data source authentication security while enabling DLI to securely access data sources.
This section describes how to use datasource authentication provided by DLI.
Constraints
- Only Spark SQL and Flink OpenSource SQL 1.12 jobs support datasource authentication.
- DLI supports four types of datasource authentication. Select an authentication type specific to each data source.
- CSS: applies to 6.5.4 or later CSS clusters with the security mode enabled.
- Kerberos: applies to MRS security clusters with Kerberos authentication enabled.
- Kafka_SSL: applies to Kafka with SSL enabled.
- Password: applies to GaussDB(DWS), RDS, DDS, and DCS.
Notes and Constraints
- Only Spark SQL and Flink OpenSource SQL 1.12 jobs support datasource authentication.
- DLI supports four types of datasource authentication. Select an authentication type specific to each data source.
- CSS: applies to 6.5.4 or later CSS clusters with the security mode enabled.
- Kerberos: applies to MRS security clusters with Kerberos authentication enabled.
- Kafka_SSL: applies to Kafka with SSL enabled.
- Password: applies to GaussDB(DWS), RDS, DDS, and DCS.
Datasource Authentication Types
DLI supports four types of datasource authentication. Select an authentication type specific to each data source.
@@ -18,7 +18,7 @@Data Source
Constraints
+Notes and Constraints
- Click the text box for tag key and select a predefined tag key from the drop-down list.
Data Source
Constraints
+Notes and Constraints
You can set Spark driver parameters to improve the scheduling efficiency of queues.
This section describes how to set queue properties on the management console.
-Constraints and Limitations
- Only SQL queues of the Spark engine support configuring queue properties.
- Setting queue properties is only supported after the queue has been created.
- Currently, only queue properties related to the Spark driver can be set.
- Queue properties cannot be set in batches.
- For a queue in an elastic resource pool, if the minimum CUs of the queue is less than 16 CUs, both Max. Spark Driver Instances and Max. Prestart Spark Driver Instances set in the queue properties do not apply.
Notes and Constraints
- Only SQL queues of the Spark engine support configuring queue properties.
- Setting queue properties is only supported after the queue has been created.
- Currently, only queue properties related to the Spark driver can be set.
- Queue properties cannot be set in batches.
- For a queue in an elastic resource pool, if the minimum CUs of the queue is less than 16 CUs, both Max. Spark Driver Instances and Max. Prestart Spark Driver Instances set in the queue properties do not apply.
Procedure
- In the navigation pane of the DLI management console, choose Resources > Queue Management.
- In the Operation column of the queue, choose More > Set Property.
- Go to the queue property setting page and set property parameters. For details about the property parameters, see Table 1.
-
Table 1 Queue properties Property
diff --git a/docs/dli/umn/dli_01_0624.html b/docs/dli/umn/dli_01_0624.html new file mode 100644 index 00000000..8dc87e3b --- /dev/null +++ b/docs/dli/umn/dli_01_0624.html @@ -0,0 +1,71 @@ + + +Establishing a Network Connection Between DLI and Resources in a Shared VPC
+++VPC Sharing Overview
VPC sharing allows sharing VPC resources created in one account with other accounts using Resource Access Manager (RAM). For example, account A can share its VPC and subnets with account B. After accepting the share, account B can view the shared VPC and subnets and use them to create resources.
++DLI Use Cases
An enterprise IT management account creates a VPC and subnets and shares them with other service accounts to facilitate centralized configuration of VPC security policies and orderly resource management.
+Service accounts use the shared VPC and subnets to create resources and want to use DLI to submit jobs and access resources in the shared VPC. To do this, they need to establish a network connection between DLI and the resources in the shared VPC.
+For example, account A is the enterprise IT management account and the owner of VPC resources. It creates the VPC and subnets and shares them with service account B.
+Account B is a service account that uses the shared VPC and subnets to create resources and uses DLI to access them.
++Prerequisites
Account A, as the resource owner, has created a VPC and subnets and designated account B as the principal.
++Establishing a Network Connection Between DLI and Resources in a Shared VPC
- Account A creates an enhanced datasource connection.
- Log in to the DLI management console using account A.
- In the navigation pane on the left, choose Datasource Connections.
- On the displayed Enhanced tab, click Create.
Set parameters based on Table 1.
+ ++
+Table 1 Parameters for creating an enhanced datasource connection Parameter
+Description
+Connection Name
+Name of the datasource connection to be created
+Resource Pool
+You do not need to set this parameter in this scenario.
+VPC
+VPC shared by account A to account B
+Subnet
+Subnet shared by account A to account B
+Host Information
+You do not need to set this parameter in this scenario.
+Tags
+Tags used to identify cloud resources. A tag includes the tag key and tag value.
+ - Click OK.
- Account A grants account B access to the enhanced datasource connection created in 1.
- In the enhanced datasource connection list, locate the row containing the newly created one, click More in the Operation column, and select Manage Permission from the drop-down list.
- In the displayed Permissions dialog box, select Grant Permission for Set Permission, enter the ID of the project account B belongs to in Project ID, and click OK.
- Account B binds a DLI elastic resource pool to the shared enhanced datasource connection.
- Log in to the DLI management console using account B.
- In the navigation pane on the left, choose Datasource Connections.
- On the displayed Enhanced tab, locate the row containing the enhanced datasource connection shared by account A, click More in the Operation column, and select Bind Resource Pool from the drop-down list.
- In the displayed Bind Resource Pool dialog box, select the created elastic resource pool for Resource Pool and click OK.
If there is no elastic resource pool available, create one by referring to Creating an Elastic Resource Pool.
+
- Account B tests the network connectivity between the elastic resource pool and resources in the VPC.+
If there are resources in the shared VPC, ensure that the security group the resources belong to has allowed access to the elastic resource pool's CIDR block.
+- Obtain the private IP address and port number of the data source in the shared VPC.
Take the RDS data source as an example. On the Instances page, click the target DB instance. On the displayed page, locate the Connection Information pane and view the private IP address. In the Connection Information pane, locate the Database Port to view the port number of the RDS DB instance.
+ - In the navigation pane of the DLI management console, choose Resources > Queue Management.
- Locate the queue under the elastic resource pool bound with the enhanced datasource connection, click More in the Operation column, and select Test Address Connectivity.
- Enter the data source connection address and port number to test the network connectivity.
If the address is reachable, it means that account B has established a network connection between the DLI resource and the resources in the shared VPC. Account B can then submit jobs to the elastic resource pool's queue and access the resources in the shared VPC.
+
- Obtain the private IP address and port number of the data source in the shared VPC.
++ diff --git a/docs/dli/umn/dli_03_0002.html b/docs/dli/umn/dli_03_0002.html deleted file mode 100644 index e2523026..00000000 --- a/docs/dli/umn/dli_03_0002.html +++ /dev/null @@ -1,11 +0,0 @@ - - -++Parent topic: Enhanced Datasource Connections+What Is DLI?
--Data Lake Insight (DLI) is a serverless data processing and analysis service fully compatible with Apache Spark and Apache Flink ecosystems. It frees you from managing any server. DLI supports standard SQL and is compatible with Spark and Flink SQL. It also supports multiple access modes, and is compatible with mainstream data formats. DLI supports SQL statements and Spark applications for heterogeneous data sources, including CloudTable, RDS, GaussDB(DWS), CSS, OBS, custom databases on ECSs, and offline databases.
--- diff --git a/docs/dli/umn/dli_03_0017.html b/docs/dli/umn/dli_03_0017.html index 75b87126..83dc65b1 100644 --- a/docs/dli/umn/dli_03_0017.html +++ b/docs/dli/umn/dli_03_0017.html @@ -1,23 +1,21 @@--Parent topic: Usage-How Do I Set the AK/SK for a Queue to Operate an OBS Table?
-- Hard-coded or plaintext AK and SK pose significant security risks. To ensure security, encrypt your AK and SK, store them in configuration files or environment variables, and decrypt them when needed.
-- If the AK and SK are obtained, set the parameters as follows:
- Create SparkContext using code
val sc: SparkContext = new SparkContext() +
+Setting Up a Spark Jar Job to Obtain the AK/SK
- To obtain the AK/SK, set the parameters as follows:
- Create a SparkContext using code.
val sc: SparkContext = new SparkContext() sc.hadoopConfiguration.set("fs.obs.access.key", ak) sc.hadoopConfiguration.set("fs.obs.secret.key", sk)- - Create SparkSession using code
val sparkSession: SparkSession = SparkSession +
- Create a SparkSession using code.
val sparkSession: SparkSession = SparkSession .builder() .config("spark.hadoop.fs.obs.access.key", ak) .config("spark.hadoop.fs.obs.secret.key", sk) .enableHiveSupport() .getOrCreate()
- Create a SparkContext using code.
- If ak, sk, and securitytoken are obtained, the temporary AK/SK and security token must be used at the same time during authentication. The setting is as follows:
- Create SparkContext using code
val sc: SparkContext = new SparkContext() +
- To obtain the AK/SK and security token and use them together for authentication, set the parameters as follows:
- Create a SparkContext using code.
val sc: SparkContext = new SparkContext() sc.hadoopConfiguration.set("fs.obs.access.key", ak) sc.hadoopConfiguration.set("fs.obs.secret.key", sk) sc.hadoopConfiguration.set("fs.obs.session.token", sts)- - Create SparkSession using code
val sparkSession: SparkSession = SparkSession +
- Create a SparkSession using code.
val sparkSession: SparkSession = SparkSession .builder() .config("spark.hadoop.fs.obs.access.key", ak) .config("spark.hadoop.fs.obs.secret.key", sk) @@ -27,6 +25,7 @@ sc.hadoopConfiguration.set("fs.obs.session.token", sts)
- Create a SparkContext using code.
- Create SparkContext using code
Parent topic: Job Developmentdiff --git a/docs/dli/umn/dli_03_0032.html b/docs/dli/umn/dli_03_0032.html index e21d76a1..f196f3b0 100644 --- a/docs/dli/umn/dli_03_0032.html +++ b/docs/dli/umn/dli_03_0032.html @@ -3,7 +3,7 @@How Do I Increase a Quota?
Increasing a Quota
The system does not support online quota adjustment. To increase a resource quota, dial the hotline or send an email to the customer service. We will process your application and inform you of the progress by phone call or email.
Before dialing the hotline number or sending an email, ensure that the following information has been obtained:
-- Domain name, project name, and project ID
Log in to the management console, click the username in the upper-right corner, choose My Credentials, and obtain the domain name, project name, and project ID.
+- Domain name, project name, and project ID
To obtain the preceding information, log in to the management console, click the username in the upper right corner, and choose My Credentials from the drop-down list.
- Quota information, including:
- Service name
- Quota type
- Required quota
Learn how to obtain the service hotline and email address.
diff --git a/docs/dli/umn/dli_03_0038.html b/docs/dli/umn/dli_03_0038.html deleted file mode 100644 index c3bb6924..00000000 --- a/docs/dli/umn/dli_03_0038.html +++ /dev/null @@ -1,53 +0,0 @@ - - -How Do I Configure Checkpoints for Flink Jar Jobs and Save the Checkpoints to OBS?
--The procedure is as follows:
-- Add the following code to the JAR file code of the Flink Jar job:
// Configure the pom file on which the StreamExecutionEnvironment depends. -StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); - - env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE); - env.getCheckpointConfig().setCheckpointInterval(40000); - env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION); - RocksDBStateBackend rocksDbBackend = new RocksDBStateBackend(new FsStateBackend("obs://${bucket}/jobs/checkpoint/my_jar"), false); - rocksDbBackend.setOptions(new OptionsFactory() { - @Override - public DBOptions createDBOptions(DBOptions currentOptions) { - return currentOptions - .setMaxLogFileSize(64 * 1024 * 1024) - .setKeepLogFileNum(3); - } - - @Override - public ColumnFamilyOptions createColumnOptions(ColumnFamilyOptions currentOptions) { - return currentOptions; - } - }); - env.setStateBackend(rocksDbBackend);-- The preceding code saves the checkpoint to the ${bucket} bucket in jobs/checkpoint/my_jar path every 40 seconds in EXACTLY_ONCE mode.
-Pay attention to the checkpoint storage path. Generally, the checkpoint is stored in the OBS bucket. The path format is as follows:
-- Path format: obs://${bucket}/xxx/xxx/xxx
- Add the following configuration to the POM file for the packages on which the StreamExecutionEnvironment depends:
<dependency> - <groupId>org.apache.flink</groupId> - <artifactId>flink-streaming-java_${scala.binary.version}</artifactId> - <version>${flink.version}</version> - <scope>provided</scope> -</dependency>-
- Configure Runtime Configuration and Restore Job from Checkpoint for a DLI Flink Jar job.
- Constraints on parameter optimization
- In the left navigation pane of the DLI console, choose Global Configuration > Service Authorization. On the page displayed, select Tenant Administrator(Global service) and click Update.
- The bucket to which data is written must be an OBS bucket created by a main account.
- Configuring Restore Job from Checkpoint
- Select Auto Restart upon Exception.
- Select Restore Job from Checkpoint and set the Checkpoint Path.
The checkpoint path is the same as that you set in JAR file code. The format is as follows:
- -
- - The checkpoint path for each Flink Jar job must be unique. Otherwise, data cannot be restored.
- DLI can access files in the checkpoint path only after DLI is authorized to access the OBS bucket.
- Constraints on parameter optimization
- Check whether the job is restored from the checkpoint.
-- diff --git a/docs/dli/umn/dli_03_0044.html b/docs/dli/umn/dli_03_0044.html index 98044b55..7ae7fa0b 100644 --- a/docs/dli/umn/dli_03_0044.html +++ b/docs/dli/umn/dli_03_0044.html @@ -1,8 +1,8 @@ ---Parent topic: Flink Jar Jobs-Does a Flink JAR Job Support Configuration File Upload? How Do I Upload a Configuration File?
-Uploading a Configuration File for a Flink JAR Job
Configuration files can be uploaded for user-defined jobs (JAR).
-- Upload the configuration file to DLI through Package Management.
- In the Other Dependencies area of the Flink JAR job, select the created DLI package.
- Load the file through ClassName.class.getClassLoader().getResource("userData/fileName") in the code. In the file name, fileName indicates the name of the file to be accessed, and ClassName indicates the name of the class that needs to access the file.
Can I Upload Configuration Files for Flink Jar Jobs?
+Uploading a Configuration File for a Flink Jar Job
You can upload configuration files for custom jobs (Jar).
+- Upload the configuration file to DLI through Package Management.
- In the Other Dependencies area of the Flink Jar job, select the created DLI package.
- Load the file through ClassName.class.getClassLoader().getResource("userData/fileName") in the code. In the file name, fileName indicates the name of the file to be accessed, and ClassName indicates the name of the class that needs to access the file.
Using a Configuration File
- Solution 1: Load the file content to the memory in the main function and broadcast the content to each taskmanager. This method is applicable to the scenario where a small number of variables need to be loaded in advance.
- Solution 2: Load the file when initializing the operator in open. A relative or absolute path can be used.
Take Kafka sink as an example. Two files (userData/kafka-sink.conf and userData/client.truststore.jks) need to be loaded.
- Example of using a relative path:
Relative path: confPath = userData/kafka-sink.conf diff --git a/docs/dli/umn/dli_03_0075.html b/docs/dli/umn/dli_03_0075.html index b059069a..e1b3d657 100644 --- a/docs/dli/umn/dli_03_0075.html +++ b/docs/dli/umn/dli_03_0075.html @@ -28,7 +28,6 @@ sk = "{{mySk}}" );
- Example of using a relative path:
- Domain name, project name, and project ID
- To obtain the AK/SK, set the parameters as follows:
- Create SparkContext using code
- Account A creates an enhanced datasource connection.
Structure of the data storage directory in OBS: obs://obs-sink/car_infos/day=xx/part-x-x.
After the data is generated, the OBS partition table can be established for subsequent batch processing through the following SQL statements:
@@ -48,11 +47,9 @@ stored as parquet location 'obs://obs-sink/car-infos';
- For details about how to create a VPC and subnet, see "Creating a VPC and Subnet" in Virtual Private Cloud User Guide. For details about how to create and use a subnet in an existing VPC, see "Create a Subnet for the VPC" in Virtual Private Cloud User Guide.
- Restore partition information from the associated OBS path.
diff --git a/docs/dli/umn/dli_03_0089.html b/docs/dli/umn/dli_03_0089.html index ab9cf67f..1165b642 100644 --- a/docs/dli/umn/dli_03_0089.html +++ b/docs/dli/umn/dli_03_0089.html @@ -30,7 +30,6 @@ sk = "{{mySk}}" );-1alter table car_infos recover partitions;
Structure of the data storage directory in OBS: obs://obs-sink/car_infos/day=xx/part-x-x.
After the data is generated, the OBS partition table can be established for subsequent batch processing through the following SQL statements:
@@ -50,11 +49,9 @@ stored as parquet location 'obs://obs-sink/car-infos';- Restore partition information from the associated OBS path.
diff --git a/docs/dli/umn/dli_03_0093.html b/docs/dli/umn/dli_03_0093.html index 4a8132e4..41bd0d9f 100644 --- a/docs/dli/umn/dli_03_0093.html +++ b/docs/dli/umn/dli_03_0093.html @@ -1,11 +1,38 @@ --1alter table car_infos recover partitions;
How Do I Eliminate Data Skew by Configuring AE Parameters?
-Scenario
If the execution of an SQL statement takes a long time, you need to access the Spark UI to check the execution status.
-If data skew occurs, the running time of a stage exceeds 20 minutes and only one task is running.
-Figure 1 Data skew example+
How Do I Do When Data Skew Occurs During the Execution of a SQL Job?
+-What Is Data Skew?
Data skew is a common issue during the execution of SQL jobs. When data is unevenly distributed, some compute nodes process significantly more data than others, which can impact the efficiency of the entire computation process.
+For example, if you notice that a SQL query is taking a long time to execute, you can check its status in SparkUI. See Figure 1. If you see a stage that has been running for over 20 minutes with only one task remaining, it is likely due to data skew.
+Procedure
- Log in to the DLI management console. Choose Job Management > SQL Jobs in the navigation pane. On the displayed page, locate the job you want to modify and click Edit in the Operation column to switch to the SQL Editor page.
- On the SQL editor page, click Set Property and add the following Spark parameters through the Settings pane:
++
Common Data Skew Scenarios
- Group By aggregation skew
During the execution of Group By aggregation, if some grouping keys have significantly more data than others, the larger groups will consume more compute resources and time during the aggregation process, resulting in slower processing speeds and data skew.
+ - JOIN operation skew
During table JOIN operations, if the keys involved in the JOIN are unevenly distributed in one of the tables, a large amount of data will be concentrated in a few tasks while others have already completed, causing data skew.
+
+Solution for Group By Data Skew
Select a subset of data and run select count(*) as sum,Key from tbl group by Key order by sum desc to identify which keys are causing data skew.
+Then, for the skewed keys, you can handle them separately by adding a salt to split them into multiple tasks for individual statistics, and finally combine the results of the separate statistics.
+For example, consider the following SQL query where Key01 is identified as the skewed key causing a single task to process a large amount of data. The following steps can be taken to handle it:
+SELECT + a.Key, + SUM(a.sum) AS Cnt +FROM + ( + SELECT + Key, + count(*) AS sum + FROM + tbl + GROUP BY + Key, + CASE + WHEN KEY = 'Key01' THEN floor(random () * 200) + ELSE 0 + END + ) a +GROUP BY + a.Key;
+Solution for JOIN Data Skew
- Log in to the DLI management console. Choose Job Management > SQL Jobs in the navigation pane. On the displayed page, locate the job you want to modify and click Edit in the Operation column to switch to the SQL Editor page.
- On the SQL editor page, click Set Property and add the following Spark parameters through the Settings pane:
The string followed by the colons (:) are the configuration parameters, and the strings following the colons are the values.
spark.sql.enableToString:false spark.sql.adaptive.join.enabled:true diff --git a/docs/dli/umn/dli_03_0100.html b/docs/dli/umn/dli_03_0100.html index 55859cd2..84f70503 100644 --- a/docs/dli/umn/dli_03_0100.html +++ b/docs/dli/umn/dli_03_0100.html @@ -2,7 +2,7 @@
How Do I Manage Fine-Grained DLI Permissions?
DLI has a comprehensive permission control mechanism and supports fine-grained authentication through Identity and Access Management (IAM). You can create policies in IAM to manage DLI permissions.
-IAM allows you to create IAM users for your employees using your cloud account, and assign permissions to control their access to specific types of resources. For example, some software developers in your enterprise need to use DLI resources but must not delete them or perform any high-risk operations. To achieve this result, you can create IAM users for the software developers and grant them only the permissions required for using DLI resources.
+With IAM, you can use your account to create IAM users for your employees, and assign permissions to the users to control their access to specific resource types. For example, some software developers in your enterprise need to use DLI resources but must not delete them or perform any high-risk operations. To achieve this result, you can create IAM users for the software developers and grant them only the permissions required for using DLI resources.
For a new user, you need to log in for the system to record the metadata before using DLI.
IAM is free to use, and you only need to pay for the resources in your account.
diff --git a/docs/dli/umn/dli_03_0119.html b/docs/dli/umn/dli_03_0119.html index 5e0d016c..dcbf069f 100644 --- a/docs/dli/umn/dli_03_0119.html +++ b/docs/dli/umn/dli_03_0119.html @@ -1,6 +1,6 @@ -Why Does the Submission Fail Due to Flink JAR File Conflict?
+Why Does a Flink Jar Package Conflict Result in Submission Failure?
Symptom
The dependency of your Flink job conflicts with a built-in dependency of the DLI Flink platform. As a result, the job submission fails.
Solution
Delete your JAR file that is the same as an existing one of the DLI Flink platform.
diff --git a/docs/dli/umn/dli_03_0126.html b/docs/dli/umn/dli_03_0126.html index 67ee6d71..45e91663 100644 --- a/docs/dli/umn/dli_03_0126.html +++ b/docs/dli/umn/dli_03_0126.html @@ -2,7 +2,7 @@How Do I Manage Tens of Thousands of Jobs Running on DLI?
If you are suggested to perform following operations to run a large number of DLI jobs:
-- Group the DLI jobs by type, and run each group on a queue.
- Alternatively, create IAM users to execute different types of jobs. .
- Group the DLI jobs by type, and run each group on a queue.
- Alternatively, create IAM users to execute different types of jobs. .
diff --git a/docs/dli/umn/dli_03_0132.html b/docs/dli/umn/dli_03_0132.html index 7666b9e8..18064afd 100644 --- a/docs/dli/umn/dli_03_0132.html +++ b/docs/dli/umn/dli_03_0132.html @@ -4,11 +4,9 @@-
-
- How Do I Configure Checkpoints for Flink Jar Jobs and Save the Checkpoints to OBS?
+ - Can I Upload Configuration Files for Flink Jar Jobs?
- - Does a Flink JAR Job Support Configuration File Upload? How Do I Upload a Configuration File?
-
- - Why Does the Submission Fail Due to Flink JAR File Conflict?
+ - Why Does a Flink Jar Package Conflict Result in Submission Failure?
- Why Does a Flink Jar Job Fail to Access GaussDB(DWS) and a Message Is Displayed Indicating Too Many Client Connections?
diff --git a/docs/dli/umn/dli_03_0138.html b/docs/dli/umn/dli_03_0138.html
index 9db67a6f..31fae707 100644
--- a/docs/dli/umn/dli_03_0138.html
+++ b/docs/dli/umn/dli_03_0138.html
@@ -1,11 +1,12 @@
- Symptom
When you create a Flink SQL job and configure the parameters, you select an OBS bucket you have created. The system displays a message indicating that the OBS bucket is not authorized. After you click Authorize, the system displays a message indicating that an internal error occurred on the server and you need to contact customer service or try again later.
- - Solution
On the settings page, press F12 to view the error details. The following is an example:
++Symptom
When you create a Flink SQL job and configure the parameters, you select an OBS bucket you have created. The system displays a message indicating that the OBS bucket is not authorized. After you click Authorize, the system displays a message indicating that an internal error occurred on the server and you need to contact customer service or try again later.
+Solution
On the settings page, press F12 to view the error details. The following is an example:
{"error_msg":"An internal error occurred. {0} Contact customer services or try again later ","error_json_opt":{"error": "Unexpected exception[NoSuchElementException: None.get]"},"error_code":"DLI.10001"}-Check whether a DLI agency has been created. If you do not have the permission to create an agency. On the DLI console, choose Global Configuration > Service Authorization, select Tenant Administrator (Global service), and click Update.
- +Check whether a DLI agency has been created. If you do not have the permission to create an agency. On the DLI console, choose Global Configuration > Service Authorization, select Tenant Administrator (Global service), and click Update.
+diff --git a/docs/dli/umn/dli_03_0139.html b/docs/dli/umn/dli_03_0139.html index 288393ba..ba7fcf96 100644 --- a/docs/dli/umn/dli_03_0139.html +++ b/docs/dli/umn/dli_03_0139.html @@ -2,7 +2,7 @@How Do I Authorize a Subuser to View Flink Jobs?
A sub-user can view queues but cannot view Flink jobs. You can authorize the sub-user using DLI or IAM.
-- Authorization on DLI
- Log in to the DLI console using a tenant account, a job owner account, or an account with the DLI Service Administrator permission.
- Choose Job Management > Flink Jobs. On the displayed page, locate the target job.
- In the Operation column of the target job, choose More > Permissions.
- On the displayed page, click Grant Permission. Enter the name of the user to be authorized and select the required permissions. Click OK. The authorized user can view the job and perform related operations.
- Authorization on DLI
- Log in to the DLI console using a tenant account, a job owner account, or an account with the DLI Service Administrator permission.
- Choose Job Management > Flink Jobs. On the displayed page, locate the target job.
- In the Operation column of the target job, choose More > Permissions.
- On the displayed page, click Grant Permission. Enter the name of the user to be authorized and select the required permissions. Click OK. The authorized user can view the job and perform related operations.
- Authorization on IAM
- Log in to the IAM console. In the navigation pane, choose Permissions > Policies/Roles. On the displayed page, click Create Custom Policy.
- Create a permission policy for the subuser to view DLI Flink jobs.
- Policy Name: Use the default name or customize a name.
- Scope: Select Project-level services.
- Policy View: Select Visual editor.
- Policy Content: Select Allow, Data Lake Insight, and dli:jobs:list_all in sequence.
Click OK to create the policy.
- Go to the User Group page, locate the user group to which the subuser to be authorized belongs and click the user group name. On the displayed page, click Assign.
- Grant permissions to the user group.
- Select Region-specific projects for Scope.
- Select the permission policy created in 2 for Permissions.
You can also select DLI Service Administrator (with all DLI permissions) for the subuser to view Flink jobs.
diff --git a/docs/dli/umn/dli_03_0140.html b/docs/dli/umn/dli_03_0140.html index 858d0a13..088f9219 100644 --- a/docs/dli/umn/dli_03_0140.html +++ b/docs/dli/umn/dli_03_0140.html @@ -2,7 +2,7 @@Why Does My Account Have Insufficient Permissions Due to Arrears?
When you submit a job, a message is displayed indicating that the job fails to be submitted due to insufficient permission caused by arrears. In this case, you need to check the roles in your token:
-- op_restrict: The account permission is restricted due to insufficient balance. If the current account balance is insufficient, all online user tokens under that account will be revoked. If they log in again, the op_restrict permission is added to the obtained tokens, limiting their operations.
- op_suspended: Your account is suspended due to arrears or other reasons. If the current account is in arrears, all online user tokens under that account will be revoked. If they log in again, the op_suspended permission is added to the obtained tokens, limiting their operations (excluding cloud service users).
- op_restrict: The account permission is restricted due to insufficient balance. If the current account balance is insufficient, all online user tokens under that account will be revoked. If they log in again, the op_restrict permission is added to the obtained tokens, limiting their operations.
- op_suspended: Your account is suspended due to arrears or other reasons. If the current account is in arrears, all online user tokens under that account will be revoked. If they log in again, the op_suspended permission is added to the obtained tokens, limiting their operations (excluding cloud service users).
If the two roles described about are in your token, user operations are restricted.
diff --git a/docs/dli/umn/dli_03_0160.html b/docs/dli/umn/dli_03_0160.html index ff120b74..40072998 100644 --- a/docs/dli/umn/dli_03_0160.html +++ b/docs/dli/umn/dli_03_0160.html @@ -3,7 +3,7 @@Why Is Error "No such user. userName:xxxx." Reported on the Flink Job Management Page When I Grant Permission to a User?
-Symptom
Choose Job Management > Flink Jobs. In the Operation column of the target job, choose More > Permissions. When a new user is authorized, No such user. userName:xxxx. is displayed.
Solution
Check whether the username exists and whether the user has logged in to the system before authorization.
+Solution
- Check whether the username exists.
- If the username exists, relog in to the management console to grant permissions.
diff --git a/docs/dli/umn/dli_03_0165.html b/docs/dli/umn/dli_03_0165.html index 3bd7d1cc..43c56634 100644 --- a/docs/dli/umn/dli_03_0165.html +++ b/docs/dli/umn/dli_03_0165.html @@ -4,7 +4,7 @@-Symptom
An exception occurred when a Flink Jar job is running. The following error information is displayed in the job log:org.apache.flink.shaded.curator.org.apache.curator.ConnectionState - Authentication failed
Possible Causes
Service authorization is not configured for the account on the Global Configuration page. When the account is used to create a datasource connection to access external data, the access fails.
+Possible Causes
Service authorization is not configured for the account on the Global Configuration page. When the account is used to create a datasource connection to access external data, the access fails.
diff --git a/docs/dli/umn/dli_03_0171.html b/docs/dli/umn/dli_03_0171.html index 64370860..46de6304 100644 --- a/docs/dli/umn/dli_03_0171.html +++ b/docs/dli/umn/dli_03_0171.html @@ -7,8 +7,7 @@Solution
- Log in to the DLI management console. Choose Global Configuration > Service Authorization in the navigation pane.
- On the Service Authorization page, select all agency permissions.
- Click Update. If the message "Agency permissions updated successfully" is displayed, the modification is successful.
- After the authorization is complete, create a datasource connection and run the job again.
-Possible Causes
The default queue is a public preset queue in the system for function trials. When multiple users submit jobs to this queue, traffic control might be triggered. As a result, the jobs fail to be submitted.
Solution
Buy a custom queue for your jobs. The procedure is as follows:
-- In the navigation pane of the DLI management console, choose Queue Management.
- In the upper right corner of the Queue Management page, click Buy Queue to create a queue.
- On the Buy Queue page, set the required parameters as you need. Especially, set Type to For SQL.
- Submit your SQL jobs to the newly created queue.
Solution
Buy a custom queue for your jobs.
diff --git a/docs/dli/umn/dli_03_0172.html b/docs/dli/umn/dli_03_0172.html index 2abde769..f3504c94 100644 --- a/docs/dli/umn/dli_03_0172.html +++ b/docs/dli/umn/dli_03_0172.html @@ -14,7 +14,7 @@ at java.lang.Thread.run(Thread.java:748)Possible Causes
When you create a migration job to DLI on the CDM console, you set Resource Queue to a DLI queue for general purpose. It should be a queue for SQL.
Solution
- On the DLI management console and click Queue Management in the navigation pane on the left. On the Queue Management page, check whether there are SQL queues. -
- Click Buy Queue to create a queue. Set Type to For SQL, set other parameters required, and click Buy.
- Go back to the CDM console and create a data migration job. Set Resource Queue to the created DLI SQL queue.
- Submit the migration job and view the job execution logs.
- Choose Resources > Resource Pool. On the displayed page, locate the purchased elastic resource pool, and click Add Queue in the Operation column. Set Type to For SQL, set other parameters, and submit the creation request.
- Go back to the CDM console and create a data migration job. Set Resource Queue to the created DLI SQL queue.
- Submit the migration job and view the job execution logs.
diff --git a/docs/dli/umn/dli_03_0179.html b/docs/dli/umn/dli_03_0179.html index c5904560..e486c053 100644 --- a/docs/dli/umn/dli_03_0179.html +++ b/docs/dli/umn/dli_03_0179.html @@ -6,7 +6,7 @@Fault Locating
The issues here are described in order of how likely they are to occur.
Troubleshoot the issue by ruling out the causes described here, one by one.
-- Check Whether a Port Number Is Added to the End of the Domain Name or IP Address
- Check Whether the Information of the Peer VPC and Subnet Are Correct.
- Check Whether the CIDR Block of the Queue Overlaps with That of the Data Source
- Check Whether the VPC Administrator Permission Is Granted to DLI
- Check Whether the Destination Security Group Allows Access from the CIDR Block of the Queue
- Check the Route Information of the VPC Peering Connection Corresponding to an Enhanced Datasource Connection
- Check Whether VPC Network ACL Rules Are Configured to Restrict Network Access
- Check Whether a Port Number Is Added to the End of the Domain Name or IP Address
- Check Whether the Information of the Peer VPC and Subnet Are Correct.
- Check Whether the CIDR Block of the Queue Overlaps with That of the Data Source
- Check Whether the DLI Datasource Connections Agency Access Permission Is Granted to DLI
- Check Whether the Destination Security Group Allows Access from the CIDR Block of the Queue
- Check the Route Information of the VPC Peering Connection Corresponding to an Enhanced Datasource Connection
- Check Whether VPC Network ACL Rules Are Configured to Restrict Network Access
-Check Whether a Port Number Is Added to the End of the Domain Name or IP Address
The port number is required for the connectivity test.
The following example tests the connectivity between a queue and a specified RDS DB instance. The RDS DB instance uses port 3306.
@@ -21,8 +21,8 @@Solution: Modify the CIDR block of the queue or create another queue.
Planing the CIDR blocks for your queues helps you to avoid this problem.
diff --git a/docs/dli/umn/dli_03_0211.html b/docs/dli/umn/dli_03_0211.html index 811df367..7a531cff 100644 --- a/docs/dli/umn/dli_03_0211.html +++ b/docs/dli/umn/dli_03_0211.html @@ -11,7 +11,7 @@Check Whether the VPC Administrator Permission Is Granted to DLI
View the connection logs to check whether there is the required permission.
-Solution: Grant DLI the VPC Administrator permission and cancel the IAM ReadOnlyAccess authorization.
+Check Whether the DLI Datasource Connections Agency Access Permission Is Granted to DLI
You can determine if a connection failure is due to insufficient permissions by checking the connection logs.
+Solution: Add the DLI Datasource Connections Agency Access authorization on the Global Configuration > Service Authorization page.
Check Whether the Destination Security Group Allows Access from the CIDR Block of the Queue
To connect to Kafka, GaussDB(DWS), and RDS instances, add security group rules for the DLI CIDR block to the security group where the instances belong. For example, to connect a queue to RDS, perform the following operations:diff --git a/docs/dli/umn/dli_03_0184.html b/docs/dli/umn/dli_03_0184.html index 3e6e94ce..75afb009 100644 --- a/docs/dli/umn/dli_03_0184.html +++ b/docs/dli/umn/dli_03_0184.html @@ -6,7 +6,7 @@- Log in to the DLI console, choose Resources > Queue Management in the navigation pane on the left. On the displayed page, select the target queue, and click
to expand the row containing the target queue to view its CIDR block. - On the Instance Management page of the RDS console, click the instance name. In the Connection Information area, locate Database Port to obtain the port number of the RDS DB instance.
- In the Connection Information area locate the Security Group and click the group name to switch to the security group management page. Select the Inbound Rules tab and click Add Rule. Set the priority to 1, protocol to TCP, port to the database port number, and source to the CIDR block of the DLI queue. Click OK.
Possible Causes
If a table exists but cannot be queried, there is a high probability that the current user does not have the permission to query or operate the table.
Solution
Contact the user who creates the table and obtain the required permissions. To assign permissions, perform the following steps:
-- Log in to the DLI management console as the user who creates the table. Choose Data Management > Databases and Tables form the navigation pane on the left.
- Click the database name. The table management page is displayed. In the Operation column of the target table, click Permissions. The table permission management page is displayed.
- Click Set Permission. In the displayed dialog box, set Authorization Object to User, set Username to the name of the user that requires the permission, and select the required permissions. For example, Select Table and Insert permissions.
- Click OK.
- Log in to the DLI console as the user that has been granted permission and check whether the table can be queried.
- Log in to the DLI management console as the user who creates the table. Choose Data Management > Databases and Tables form the navigation pane on the left.
- Click the database name. The table management page is displayed. In the Operation column of the target table, click Permissions. The table permission management page is displayed.
- Click Set Permission. In the displayed dialog box, set Authorization Object to User, set Username to the name of the user that requires the permission, and select the required permissions. For example, Select Table and Insert permissions.
- Click OK.
- Log in to the DLI console as the user that has been granted permission and check whether the table can be queried.
diff --git a/docs/dli/umn/dli_03_0196.html b/docs/dli/umn/dli_03_0196.html index 9471e4f2..550d30cf 100644 --- a/docs/dli/umn/dli_03_0196.html +++ b/docs/dli/umn/dli_03_0196.html @@ -28,7 +28,7 @@Shuffle data skew is caused by unbalanced number of key values in join.- Perform group by and count on a join to collect statistics on the number of key values of each join. The following is an example:
Join table lefttbl and table righttbl. num in the lefttbl table is the key value of the join. You can perform group by and count on lefttbl.num.
SELECT * FROM lefttbl a LEFT join righttbl b on a.num = b.int2; SELECT count(1) as count,num from lefttbl group by lefttbl.num ORDER BY count desc;
- - Use concat(cast(round(rand() * 999999999) as string) to generate a random number for each key value.
- If the skew is serious and random numbers cannot be generated, see How Do I Eliminate Data Skew by Configuring AE Parameters?
- Use concat(cast(round(rand() * 999999999) as string) to generate a random number for each key value.
- If the skew is serious and random numbers cannot be generated, see How Do I Do When Data Skew Occurs During the Execution of a SQL Job?
- How Do I View SQL Execution Records?
-
- How Do I Eliminate Data Skew by Configuring AE Parameters?
+- How Do I Do When Data Skew Occurs During the Execution of a SQL Job?
- What Can I Do If a Table Cannot Be Queried on the DLI Console?
diff --git a/docs/dli/umn/dli_03_0221.html b/docs/dli/umn/dli_03_0221.html index 46cfdd36..6cdbe06e 100644 --- a/docs/dli/umn/dli_03_0221.html +++ b/docs/dli/umn/dli_03_0221.html @@ -5,8 +5,6 @@
-
-
- What Is DLI?
- - Which Data Formats Does DLI Support?
- What Are the Differences Between MRS Spark and DLI Spark?
diff --git a/docs/dli/umn/dli_03_0227.html b/docs/dli/umn/dli_03_0227.html index 5ad8021c..564a12e2 100644 --- a/docs/dli/umn/dli_03_0227.html +++ b/docs/dli/umn/dli_03_0227.html @@ -5,7 +5,7 @@Error information: DLI.0003: Permission denied for resource 'databases.dli_test.tables.test.columns.col1', User = '{UserName}', Action = 'SELECT'
Solution
The user does not have the permission to query the table.
-In the navigation pane on the left of the DLI console page, choose Data Management > Databases and Tables, search for the desired database table, view the permission configuration, and grant the table query permission to the user who requires it.
+In the navigation pane on the left of the DLI console page, choose Data Management > Databases and Tables, search for the desired database table, view the permission configuration, and grant the table query permission to the user who requires it.
diff --git a/docs/dli/umn/dli_07_0005.html b/docs/dli/umn/dli_07_0005.html index a96dfde7..51e1ba77 100644 --- a/docs/dli/umn/dli_07_0005.html +++ b/docs/dli/umn/dli_07_0005.html @@ -1,20 +1,25 @@ -Constraints and Limitations
+Notes and Constraints
On Jobs
- Only the latest 100 jobs are displayed on DLI's SparkUI.
- A maximum of 1,000 job results can be displayed on the console. To view more or all jobs, export the job data to OBS.
- To export job run logs, you must have the permission to access OBS buckets. You need to configure a DLI job bucket on the Global Configuration > Project page in advance.
- The View Log button is not available for synchronization jobs and jobs running on the default queue.
- Only Spark jobs support custom images.
- An elastic resource pool supports a maximum of 32,000 CUs.
For details about job constraints, see Job Management.
+For more notes and constraints on jobs, see Job Management.
On Queues
- A queue named default is preset in DLI for you to experience. Resources are allocated on demand.
- Queue types:
- For SQL: Spark SQL jobs can be submitted to SQL queues.
- For general purpose: The queue is used to run Spark programs, Flink SQL jobs, and Flink Jar jobs.
The queue type cannot be changed. If you want to use another queue type, purchase a new queue.
- The region of a queue cannot be changed.
- Queues with 16 CUs do not support scale-out or scale-in.
- Queues with 64 CUs do not support scale-in.
- A newly created queue can be scaled in or out only after a job is executed on the queue.
- DLI queues cannot access the Internet.
For more constraints on using a DLI queue, see Queue Overview.
+For more notes and constraints on using a DLI queue, see Queue Overview.
On Elastic Resource Pools
- The region of an elastic resource pool cannot be changed.
- Jobs of Flink 1.10 or later can run in elastic resource pools.
- The network segment of an elastic resource pool cannot be changed after being set.
- You can only view the scaling history of resource pools in the last 30 days.
- Elastic resource pools cannot access the Internet. +
- Changes to elastic resource pool CUs can occur when setting the CU, adding or deleting queues in an elastic resource pool, or modifying the scaling policies of queues in an elastic resource pool, or when the system automatically triggers elastic resource pool scaling. However, in some cases, the system cannot guarantee that the scaling will reach the target CUs as planned.
- If there are not enough physical resources, an elastic resource pool may not be able to scale out to the desired target size.
- The system does not guarantee that an elastic resource pool will be scaled in to the desired target size.
The system checks the resource usage before scaling in the elastic resource pool to determine if there is enough space for scaling in. If the existing resources cannot be scaled in according to the minimum scaling step, the pool may not be scaled in successfully or only partially.
+The scaling step may vary depending on the resource specifications, usually 16 CUs, 32 CUs, 48 CUs, 64 CUs, etc.
+For example, if the elastic resource pool has a capacity of 192 CUs and the queues in the pool are using 68 CUs due to running jobs, the plan is to scale in to 64 CUs.
+When executing a scaling in task, the system determines that there are 124 CUs remaining and scales in by the minimum step of 64 CUs. However, the remaining 60 CUs cannot be scaled in any further. Therefore, after the elastic resource pool executes the scaling in task, its capacity is reduced to 128 CUs.
For more constraints on elastic resource pools, see Elastic Resource Pool Overview.
+
For more notes and constraints on elastic resource pools, see Elastic Resource Pool Overview.
-On Resources
- Database
- default is the database built in DLI. You cannot create a database named default.
- DLI supports a maximum of 50 databases.
- Table
- DLI supports a maximum of 5,000 tables.
- DLI supports the following table types:
- MANAGED: Data is stored in a DLI table.
- EXTERNAL: Data is stored in an OBS table.
- View: A view can only be created using SQL statements.
- Datasource table: The table type is also EXTERNAL.
- Package
- A package can be deleted, but a package group cannot be deleted.
- The following types of packages can be uploaded:
- JAR: JAR file
- PyFile: User Python file
- File: User file
- ModelFile: User AI model file
For details about constraints on resources, see Data Management.
+For more notes and constraints on resources, see Data Management.
On Enhanced Datasource Connections
- Datasource connections cannot be created for the default queue.
- Flink jobs can directly access DIS, OBS, and SMN data sources without using datasource connections.
- VPC Administrator permissions are required for enhanced connections to use VPCs, subnets, routes, VPC peering connections.
- If you use an enhanced datasource connection, the CIDR block of the elastic resource pool or queue cannot overlap with that of the data source.
- Only queues bound with datasource connections can access datasource tables.
- Datasource tables do not support the preview function.
- When checking the connectivity of datasource connections, the constraints on IP addresses are as follows:
- The IP address must be valid, which consists of four decimal numbers separated by periods (.). The value ranges from 0 to 255.
- During the test, you can add a port after the IP address and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535.
For example, 192.168.xx.xx or 192.168.xx.xx:8181.
+On Enhanced Datasource Connections
- Datasource connections cannot be created for the default queue.
- Flink jobs can directly access DIS, OBS, and SMN data sources without using datasource connections.
- VPC Administrator permissions are required for enhanced connections to use VPCs, subnets, routes, VPC peering connections.
- If you use an enhanced datasource connection, the CIDR block of the elastic resource pool or queue cannot overlap with that of the data source.
- Only queues bound with datasource connections can access datasource tables.
- Datasource tables do not support the preview function.
- When checking the connectivity of datasource connections, the notes and constraints on IP addresses are:
- The IP address must be valid, which consists of four decimal numbers separated by periods (.). The value ranges from 0 to 255.
- During the test, you can add a port after the IP address and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535.
- When checking the connectivity of datasource connections, the constraints on domain names are as follows:
- The domain name can contain 1 to 255 characters. Only letters, digits, underscores (_), and hyphens (-) are allowed.
- The top-level domain name must contain at least two letters, for example, .com, .net, and .cn.
- During the test, you can add a port after the domain name and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535. +
- When checking the connectivity of datasource connections, the notes and constraints on domain names are:
- The domain name can contain 1 to 255 characters. Only letters, digits, underscores (_), and hyphens (-) are allowed.
- The top-level domain name must contain at least two letters, for example, .com, .net, and .cn.
- During the test, you can add a port after the domain name and separate them with colons (:). The port can contain a maximum of five digits. The value ranges from 0 to 65535.
For more constraints on enhanced datasource connections, see Enhanced Datasource Connection Overview.
+For more notes and constraints on enhanced datasource connections, see Enhanced Datasource Connection Overview.
On Datasource Authentication
- Only Spark SQL and Flink OpenSource SQL 1.12 jobs support datasource authentication.
- DLI supports four types of datasource authentication. Select an authentication type specific to each data source.
- CSS: applies to 6.5.4 or later CSS clusters with the security mode enabled.
- Kerberos: applies to MRS security clusters with Kerberos authentication enabled.
- Kafka_SSL: applies to Kafka with SSL enabled.
- Password: applies to GaussDB(DWS), RDS, DDS, and DCS.
For more constraints on datasource authentication, see Datasource Authentication Introduction.
+For more notes and constraints on datasource authentication, see Datasource Authentication Introduction.
-On SQL Syntax
- Constraints on the SQL syntax:
- You are not allowed to specify a storage path when creating a DLI table using SQL statements.
- Constraints on the size of SQL statements:
- Each SQL statement should contain less than 500,000 characters.
- The size of each SQL statement must be less than 1 MB.
Other
- For details about quota constraints, see Quotas.
- Recommended browsers for logging in to DLI:
- Google Chrome 43.0 or later
- Mozilla Firefox 38.0 or later
- Internet Explorer 9.0 or later
Other
- For quota notes and constraints, see Quotas.
- Recommended browsers for logging in to DLI:
- Google Chrome 43.0 or later
- Mozilla Firefox 38.0 or later
- Internet Explorer 9.0 or later
- Authorization on DLI
Why Is a Message Displayed Indicating That the OBS Bucket Is Not Authorized When I Select an OBS Bucket for a Flink SQL Job?
-
- Group By aggregation skew
- To create a table on the Data Management page:
- You can create a table on either the Databases and Tables page or the SQL Editor page.