2020-08-11
diff --git a/docs/dms/umn/kafka-ug-0001.html b/docs/dms/umn/kafka-ug-0001.html
index 34a928ca..f4927832 100644
--- a/docs/dms/umn/kafka-ug-0001.html
+++ b/docs/dms/umn/kafka-ug-0001.html
@@ -1,21 +1,17 @@
Accessing Kafka Using a VPC Endpoint Across VPCs
-ContextVPCs are logically isolated from each other. If a Kafka instance and a Kafka client are in different VPCs within a region, they cannot communicate with each other. In this case, you can use one of the following methods to access a Kafka instance across VPCs:
+ VPCs are logically isolated from each other. If a Kafka instance and a Kafka client are in different VPCs within a region, they cannot communicate with each other. In this case, you can use one of the following methods to access a Kafka instance across VPCs:
- Establish a VPC peering connection to allow two VPCs to communicate with each other. For details, see VPC Peering Connection.
- Use VPC Endpoint (VPCEP) to establish a cross-VPC connection.
-
- ScenarioThe following describes how to use VPCEP to implement cross-VPC access.
+ The following describes how to use VPCEP to implement cross-VPC access.
VPCEP provides two types of resources: VPC endpoint services and VPC endpoints.
- A VPC endpoint service can be a Kafka instance which is accessed using VPC endpoints.
- A VPC endpoint is a secure and private channel for connecting a VPC to a VPC endpoint service.
Figure 1 Working principle of accessing a Kafka instance across VPCs
-
- ProcedureFigure 2 Process for accessing a Kafka instance across VPCs
- Creating a VPC Endpoint Service- Log in to the console.
- Click
 in the upper left corner to select a region. Select the region where your Kafka instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- In the Advanced Settings section on the Basic Information tab page, obtain the listeners IP addresses and port IDs of the instance for Cross-VPC Access.
Figure 3 Cross-VPC access–related listeners IP addresses and corresponding port IDs of the Kafka instance
- - In the Network section on the Basic Information tab page, view the VPC to which the Kafka instance belongs.
Figure 4 Viewing the VPC to which the Kafka instance belongs
- - Click the VPC to obtain the VPC ID on the VPC console.
Figure 5 Obtaining the VPC ID
+ - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- In the Advanced Settings section on the Basic Information tab page, obtain the listeners IP addresses and port IDs of the instance for Cross-VPC Access.
Figure 2 Cross-VPC access–related listeners IP addresses and corresponding port IDs of the Kafka instance
+ - In the Network section on the Basic Information tab page, view the VPC to which the Kafka instance belongs.
Figure 3 Viewing the VPC to which the Kafka instance belongs
+ - Click the VPC to obtain the VPC ID on the VPC console.
Figure 4 Obtaining the VPC ID
- Call the VPC Endpoint API to create a VPC endpoint service. For details, see Creating a VPC Endpoint Service.
POST https://{endpoint}/v1/{project_id}/vpc-endpoint-services
Set the following request parameter to the specified values, and other parameters as required.
- port_id: one of the port IDs obtained in 5.
- vpc_id: VPC ID obtained in 7.
- server_type: VM
- client_port: 9011
- server_port: 9011
- protocol: TCP
- approval_enabled: false
- service_type: interface
- endpoint: VPCEP endpoint obtained from Regions and Endpoints. The region must be the same as that of the Kafka instance.
- project_id: project ID obtained from Obtaining a Project ID. The region must be the same as that of the Kafka instance.
@@ -23,23 +19,24 @@
Record the value of service_name in the response. This parameter indicates the name of the VPC endpoint service.
- Repeat 8 to create VPC endpoint services for other port IDs obtained in 5 and record the VPC endpoint service names.
- (Optional) Adding a Whitelist RecordIf the Kafka client and Kafka instance belong to different accounts, add the ID of the account to which the Kafka client belongs to the whitelist of the endpoint service. For details, see Add a Whitelist Record.
+ (Optional) Adding a WhitelistThe VPC endpoint service can be used across accounts through a whitelist.
+ If the Kafka client and Kafka instance belong to different accounts, add the ID of the account to which the Kafka client belongs to the whitelist of the endpoint service. For details, see Add a Whitelist Record.
Creating a VPC Endpoint- Click Service List. Then choose Network > VPC Endpoint.
- Click Create VPC Endpoint.
- Set the following parameters:
- Region: Select the region that the Kafka instance is in.
- Service Category: Select Find a service by name.
- VPC Endpoint Service Name: Enter the VPC endpoint service name recorded in 8 and click Verify. If Service name found is displayed, proceed with subsequent operations.
- VPC: Select the VPC that the Kafka client is in.
- Subnet: Select the subnet that the Kafka client is in.
- Private IP Address: Select Automatic.
Retain the default values for other parameters. For details, see Creating a VPC Endpoint.
- - Click Create Now.
- Confirm the configurations and submit the request.
- Go back to the VPC endpoint list and check whether the status of the created VPC endpoint has changed to Accepted. The Accepted state means that the VPC endpoint has been connected to the VPC endpoint service.
Figure 6 Checking the VPC endpoint status
+ - Click Create Now.
- Confirm the configurations and submit the request.
- Go back to the VPC endpoint list and check whether the status of the created VPC endpoint has changed to Accepted. The Accepted state means that the VPC endpoint has been connected to the VPC endpoint service.
Figure 5 Checking the VPC endpoint status
- Click the VPC endpoint ID. On the Summary tab page, obtain the private IP address.
You can use the private IP address to access the VPC endpoint service.
-Figure 7 Viewing the private IP address
+Figure 6 Viewing the private IP address
- Repeat 1 to 7 to create a VPC endpoint for each VPC endpoint service created in 9, and view and record the private IP addresses of the VPC endpoint services.
- Changing the advertised.listeners IP Addresses- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- On the Advanced Settings section of the Basic Information tab page, click Modify for Cross-VPC Access to change the value of advertised.listeners IP address to the private IP addresses recorded in 7 and 8. Click Save.
 Each IP address must match the corresponding port ID. Otherwise, the network will be disconnected.
+ Modifying Parameter advertised.listeners IP- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- On the Advanced Settings section of the Basic Information tab page, click Modify for Cross-VPC Access to change the value of advertised.listeners IP address to the private IP addresses recorded in 7 and 8. Click Save.
Each IP address must match the corresponding port ID. Otherwise, the network will be disconnected.
-Figure 8 Changing the advertised.listeners IP addresses
+Figure 7 Changing the advertised.listeners IP addresses
Verifying ConnectivityCheck whether messages can be created and retrieved by referring to Connecting to Kafka Using the Client (Plaintext Access) or Connecting to Kafka Using the Client (Ciphertext Access).
Notes:
- - The address for connecting to a Kafka instance is in the format of "advertised.listeners IP:9011". For example, the addresses for connecting to the Kafka instance shown in Figure 8 are 10.158.0.151:9011,10.158.0.162:9011,10.158.0.164:9011.
- Configure inbound rules for the security group of the Kafka instance to allow access from 198.19.128.0/17 over port 9011.
- If a network access control list (ACL) has been configured for the subnet of this instance, configure inbound rules for the network ACL to allow access from 198.19.128.0/17 and from the subnet used by the VPC endpoint.
+ - The address for connecting to a Kafka instance is in the format of "advertised.listeners IP:9011". For example, the addresses for connecting to the Kafka instance shown in Figure 7 are 10.158.0.151:9011,10.158.0.162:9011,10.158.0.164:9011.
- Configure inbound rules for the security group of the Kafka instance to allow access from 198.19.128.0/17 over port 9011.
- If a network access control list (ACL) has been configured for the subnet of this instance, configure inbound rules for the network ACL to allow access from 198.19.128.0/17 and from the subnet used by the VPC endpoint.
198.19.128.0/17 is the network segment allocated to the VPCEP service. To use VPCEP, allow access from this network segment.
diff --git a/docs/dms/umn/kafka-ug-0002.html b/docs/dms/umn/kafka-ug-0002.html
index 4cde3d02..dda69fe7 100644
--- a/docs/dms/umn/kafka-ug-0002.html
+++ b/docs/dms/umn/kafka-ug-0002.html
@@ -1,19 +1,18 @@
Configuring Kafka Topic Permissions
- DMS supports access control list (ACL) for topics. You can differentiate user permissions by granting users different permissions in a topic.
- This section describes how to grant topic permissions to users after ciphertext access is enabled for Kafka instances. For details about how to create a user, see Configuring Kafka Users.
+ Kafka instances with ciphertext access enabled support access control list (ACL) for topics. You can differentiate user permissions by granting users different permissions in a topic.
+ This section describes how to grant topic permissions to users after ciphertext access is enabled for Kafka instances. For details about how to create a user, see Configuring Kafka ACL Users.
This function is unavailable for single-node instances.
- Constraints- If no user is granted any permission for a topic and allow.everyone.if.no.acl.found is set to true, all users can subscribe to or publish messages to the topic.
- If allow.everyone.if.no.acl.found is set to false, only the authorized users can subscribe to or publish messages to the topic. The value of allow.everyone.if.no.acl.found can be modified.
- If one or more users are granted permissions for a topic, only the authorized users can subscribe to or publish messages to the topic.
- If both the default and individual user permissions are configured for a topic, the union of the permissions is used.
+ Constraints- If parameter allow.everyone.if.no.acl.found is set to true and no topic is granted for a user, all users can subscribe to or publish messages to the topic. If permissions for a topic has been granted to one or more users, only these users can subscribe to or publish messages to the topic. The value of allow.everyone.if.no.acl.found can be modified.
- If allow.everyone.if.no.acl.found is set to false, only the authorized users can subscribe to or publish messages to the topic. The value of allow.everyone.if.no.acl.found can be modified.
- If both the default and individual user permissions are configured for a topic, the union of the permissions is used.
- Prerequisites
+
- Configuring Topic Permissions- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+ Procedure- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose Topics.
- In the row containing the desired topic, click Grant User Permission.
In the upper part of the Grant User Permission dialog box, the topic information is displayed, including the topic name, number of partitions, aging time, number of replicas, and whether synchronous flushing and replication are enabled. You can enable Default permissions to grant the same permissions for all users. You can use the search box to search for a user if there are many users. In the Users area, the list of created users is displayed. In the Selected area, grant permissions to the users.
- - Grant topic permissions to users.
- To grant the same permissions to all users, select Default permissions and then select permissions. As shown in the following figure, all users have the permission to publish messages to this topic.
Figure 1 Granting the same permissions to all users
- - To grant different permissions to different users, do not select Default permissions. In the Users area of the Grant User Permission dialog box, select target users. In the Selected area, configure permissions (Subscribe, Publish, or Publish/Subscribe) for the users. As shown in the following figure, only the test, send, and receive users can subscribe to or publish messages to this topic. The send_receive user cannot subscribe to or publish messages to this topic.
Figure 2 Granting permissions to individual users
+ - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose Topics.
- In the row containing the desired topic, click Grant User Permission.
- Grant topic permissions to users.
- To grant the same permissions to all users, select Default permissions and then select permissions. As shown in the following figure, all users have the permission to publish messages to this topic.
Figure 1 Granting the same permissions to all users
+ - To grant different permissions to different users, do not select Default permissions. In the Users area of the Grant User Permission dialog box, select target users. If there are many users, enter the username in the search box for a quick search. In the Selected area, configure permissions (Subscribe, Publish, or Publish/Subscribe) for the users. As shown in the following figure, only the test, send, and receive users can subscribe to or publish messages to this topic. The send_receive user cannot subscribe to or publish messages to this topic.
Figure 2 Granting permissions to individual users
If both the default and individual user permissions are configured for a topic, the union of the permissions is used. As shown in the following figure, the test and receive users can subscribe to and publish messages to this topic.
Figure 3 Granting topic permissions to users
@@ -21,7 +20,7 @@
Figure 4 Viewing authorized users and their permissions
- (Optional) Deleting Topic Permissions- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+ Deleting Topic Permissions- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose Topics.
- In the row containing the desired topic, click Grant User Permission.
- In the Selected area of the displayed Grant User Permission dialog box, locate the row that contains the user whose permissions are to be removed, click Delete, and click OK.
diff --git a/docs/dms/umn/kafka-ug-0003.html b/docs/dms/umn/kafka-ug-0003.html
index 2485753a..5c0f319e 100644
--- a/docs/dms/umn/kafka-ug-0003.html
+++ b/docs/dms/umn/kafka-ug-0003.html
@@ -1,7 +1,7 @@
- Configuring Kafka Users
- DMS supports access control list (ACL) for topics. You can differentiate user permissions by granting users different permissions in a topic.
+ Configuring Kafka ACL Users
+ Kafka instances with ciphertext access enabled support access control list (ACL) for topics. You can isolate users by granting them different permissions in a topic.
This section describes how to create users, reset the password, and delete users with ciphertext access enabled. For details about how to grant topic permissions for users, see Configuring Kafka Topic Permissions.
The maximum number of users that can be created for a Kafka instance is 20 or 500. Check the console for the actual limit.
There are two ways to create a user on the console. Accordingly, there are two ways to reset the user's password:
@@ -12,7 +12,28 @@
Creating a User- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- On the Users page, click Create User.
- In the displayed Create User dialog box, set the username and password, and click OK.
After the user is created, grant permissions to the user by referring to Configuring Kafka Topic Permissions.
+ - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired instance to go to the instance details page.
- On the Users page, click Create User.
- Set user information by referring to Configuring Kafka ACL Users.
+ Table 1 User creation parametersParameter
+ |
+Description
+ |
+
|---|
+
+Username
+ |
+The username used to access a Kafka instance, you can customize a name that complies with the rules: 4–64 characters; starts with a letter; can contain only letters, digits, hyphens (-), and underscores (_).
+ |
+
+Password
+ |
+The password used to access a Kafka instance. A password must meet the following requirements:
+- Contains 8 to 32 characters.
- Contains at least three of the following character types: uppercase letters, lowercase letters, digits, and special characters `~! @#$ %^&*()-_=+\|[{}];:'",<.>? and spaces, and cannot start with a hyphen (-).
- Cannot be the username spelled forward or backward.
+ |
+
+
+
+ - Click OK.
After the user is created, grant permissions to the user by referring to Configuring Kafka Topic Permissions.
Resetting the Password (for the Initial User)- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
@@ -23,16 +44,16 @@
- Resetting the User Password (for Non-initial Users)- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+ Resetting the User Password (for Non-initial Users)- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- On the Users page, click Reset Password in the row containing the desired user.
- Enter and confirm a new password, and click OK.
- If the password is successfully reset, a success message is displayed.
- If the password fails to be reset, a failure message is displayed. In this case, reset the password again. If you still fail to reset the password after multiple attempts, contact customer service.
The system will display a success message only after the password is successfully reset on all brokers.
- Deleting a User- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+ Deleting a User- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- Delete a user in either of the following ways:
- On the Users page, click Delete in the row containing the desired user.
- On the Users page, select one or more users and click Delete above the list.
+ - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- In the navigation pane, choose Users.
- Delete a user in either of the following ways:
- In the row containing the desired user, click Delete.
- Select one or more users and click Delete above the list.
The initial user set when ciphertext access is enabled for the first time cannot be deleted.
- In the displayed Delete User dialog box, click OK to delete the user.
diff --git a/docs/dms/umn/kafka-ug-0004.html b/docs/dms/umn/kafka-ug-0004.html
index b6a6de2b..4e0e184c 100644
--- a/docs/dms/umn/kafka-ug-0004.html
+++ b/docs/dms/umn/kafka-ug-0004.html
@@ -1,14 +1,14 @@
Viewing Kafka Disk Usage
- This section describes how to view the disk usage of each broker on the Kafka console.
+ This section describes how to view the disk usage of each broker of a Kafka instance on the console.
This function is unavailable for single-node instances.
- Viewing Kafka Disk Usage- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+ Procedure- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click a Kafka instance to go to the Basic Information page.
- Go to the Disk Usage Statistics page.
Figure 1 Viewing disk usage
-You can query topics that use the most disk space or topics that have used a specified amount or percentage of disk space.
+ - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click a Kafka instance to go to the Basic Information page.
- Go to the Disk Usage Statistics page.
You can query topics that use the most disk space or topics that have used a specified amount or percentage of disk space.
In the upper right corner of the page, click View Metric. On the displayed Cloud Eye page, you can view metrics of Kafka instances.
+Figure 1 Viewing disk usage
diff --git a/docs/dms/umn/kafka-ug-0006.html b/docs/dms/umn/kafka-ug-0006.html
index 37b68942..7ac3b41e 100644
--- a/docs/dms/umn/kafka-ug-0006.html
+++ b/docs/dms/umn/kafka-ug-0006.html
@@ -5,20 +5,60 @@
Changing the number of partitions does not restart the instance or affect services.
Methods for changing the partition quantity:
-
- Method 1: By using Kafka console- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+
+ Changing the Number of Topic Partitions (Console)- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose Topics.
- Modify the number of partitions using either of the following methods:
- Select one or more topics and click Edit Topic in the upper left corner.
- In the row containing the desired topic, click Edit.
- - In the Edit Topic dialog box, enter the number of partitions and click OK.
- The number of partitions can only be increased.
- To ensure performance, the Kafka console allows a maximum of 200 partitions for each topic.
- The total number of partitions of all topics cannot exceed the maximum number of partitions allowed by the instance.
+ - In the Edit Topic dialog box, enter the number of partitions and click OK.
- The number of partitions can only be increased.
- To ensure performance, the Kafka console allows a maximum of 200 partitions for each topic.
- The total partitions of all topics cannot exceed the maximum partitions of an instance. The maximum partitions of an instance varies by instance specifications. For details, see Specifications.
- Method 2: By Using Kafka CLIIf your Kafka client version is later than 2.2, you can use kafka-topics.sh to change the partition quantity.
+ Changing the Number of Topic Partitions (Client)If your Kafka client version is later than 2.2, you can use kafka-topics.sh to change the partition quantity.
For an instance with ciphertext access enabled, if allow.everyone.if.no.acl.found is set to false, topic partition quantity cannot be modified through the client.
diff --git a/docs/dms/umn/kafka-ug-0007.html b/docs/dms/umn/kafka-ug-0007.html
index 215d0b40..74ccd404 100644
--- a/docs/dms/umn/kafka-ug-0007.html
+++ b/docs/dms/umn/kafka-ug-0007.html
@@ -1,312 +1,323 @@
Modifying Kafka Instance Configuration Parameters
- ScenarioYour Kafka instances, topics, and consumers come with default configuration parameter settings. You can modify common parameters on the DMS console. For details about parameters that are not listed on the console, see the Kafka official website.
+ Your Kafka instances, topics, and consumers come with default configuration parameter settings. You can modify common parameters on the DMS console. For details about parameters that are not listed on the console, see the Kafka official website.
Kafka instances have dynamic and static parameters:
- Dynamic parameters: Modifying dynamic parameters will not restart the instance.
- Static parameters: After static parameters are modified, you must manually restart the instance.
- Configuration parameters of some old instances cannot be modified. Check whether your instance parameters can be modified on the console. If they cannot be modified, contact customer service.
- This function is not available for single-node instances.
- PrerequisitesYou can modify configuration parameters of a Kafka instance when the instance is in the Running state.
- Modifying Kafka Instance Configuration Parameters- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+ Procedure- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- On the Parameters page, click Edit in the row containing the parameter to modify.
Parameters of v1.1.0 instances are described in Table 2 and Table 1. Parameters of v2.3.0/v2.7/3.x instances are described in Table 3 and Table 4.
+ - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- On the Parameters page, click Edit in the row containing the parameter to modify.
Parameters of v1.1.0 instances are described in Table 1 and Table 2. Parameters of v2.3.0/v2.7/3.x instances are described in Table 3 and Table 4.
-Table 1 Dynamic parameters (v1.1.0 instances)Parameter
+Table 1 Dynamic parameters (v1.1.0 instances)Parameter
|
-Description
+ | Description
|
-Value Range
+ | Value Range
|
-Default Value
+ | Default Value
|
|---|
-auto.create.groups.enable
+ | auto.create.groups.enable
|
-Whether to automatically create consumer groups.
+ | Whether to automatically create consumer groups.
This parameter is not displayed on the console for some earlier instances. The function of automatically creating consumer groups is enabled by default and cannot be disabled on the console.
|
-true/false
+ | true/false
|
-true
+ | true
|
-offsets.retention.minutes
+ | offsets.retention.minutes
|
-The longest period a consumption position can be retained starts from the time of submission. Positions retained beyond this duration will be deleted. Each time a consumption position is submitted to a topic partition, its retention period resets to 0. The unit is minute.
+ | The longest period a consumption position can be retained starts from the time of submission. Positions retained beyond this duration will be deleted. Each time a consumption position is submitted to a topic partition, its retention period resets to 0.
This parameter is displayed as a static one for certain earlier instances.
|
-1440–30240
+ | 1440–30240
+Unit: minute
|
-20160
+ | 20160
|
Table 2 Static parameters (v1.1.0 instances)Parameter
+Table 2 Static parameters (v1.1.0 instances)Parameter
|
-Description
+ | Description
|
-Value Range
+ | Value Range
|
-Default Value
+ | Default Value
|
|---|
-min.insync.replicas
+ | min.insync.replicas
|
-If a producer sets the acks parameter to all (or -1), the min.insync.replicas parameter specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful.
+ | If a producer sets the acks parameter to all (or -1), the min.insync.replicas parameter specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful.
|
-1–3
+ | 1–3
|
-1
+ | 1
|
-message.max.bytes
+ | message.max.bytes
|
-Maximum length of a single message, in bytes.
+ | Maximum length of a single message.
|
-0–10,485,760
+ | 0–10,485,760
+Unit: byte
|
-10,485,760
+ | 10,485,760
|
-unclean.leader.election.enable
+ | unclean.leader.election.enable
|
-Indicates whether to allow replicas not in the ISR set to be elected as the leader as a last resort, even though doing so may result in data loss.
+ | Indicates whether to allow replicas not in the ISR set to be elected as the leader as a last resort, even though doing so may result in data loss.
|
-true or false
+ | true or false
|
-false
+ | false
|
-connections.max.idle.ms
+ | connections.max.idle.ms
|
-Idle connection timeout (in ms). Connections that are idle for the duration specified by this parameter will be closed.
+ | Idle connection timeout (in ms). Connections that are idle for the duration specified by this parameter will be closed.
|
-5000–600,000
+ | 5000–600,000
+Unit: millisecond
|
-600,000
+ | 600,000
|
-log.retention.hours
+ | log.retention.hours
|
-Duration (in hours) for retaining a log file.
+ | Maximum duration for storing log files.
This parameter takes effect only for topics that have no aging time configured. If there is aging time configured for topics, it overrides this parameter.
|
-1–168
+ | 1–168
+Unit: hour
|
-72
+ | 72
|
-max.connections.per.ip
+ | max.connections.per.ip
|
-The maximum number of connections allowed from each IP address. Request for new connections will be rejected once the limit is reached.
+ | The maximum number of connections allowed from each IP address. Request for new connections will be rejected once the limit is reached.
|
-100–20,000
+ | 100–20,000
|
-1000
+ | 1000
|
-group.max.session.timeout.ms
+ | group.max.session.timeout.ms
|
-The maximum session timeout (in ms) for consumers. A longer timeout gives consumers more time to process messages between heartbeats but results in a longer time to detect failures.
+ | Maximum session timeout for consumers. A longer timeout gives consumers more time to process messages between heartbeats but results in a longer time to detect failures.
|
-6000–1,800,000
+ | 6000–1,800,000
+Unit: millisecond
|
-1,800,000
+ | 1,800,000
|
-default.replication.factor
+ | default.replication.factor
|
-The default number of replicas configured for an automatically created topic.
+ | The default number of replicas configured for an automatically created topic.
|
-1–3
+ | 1–3
|
-3
+ | 3
|
-allow.everyone.if.no.acl.found
+ | allow.everyone.if.no.acl.found
|
-When this parameter is set to true, all users can access resources without ACL rules.
+ | When this parameter is set to true, all users can access resources without ACL rules.
This parameter is displayed only when is enabled for the instance or ciphertext access is used.
This parameter cannot be modified for certain earlier instances.
|
-true/false
+ | true/false
|
-true
+ | true
|
-num.partitions
+ | num.partitions
|
-The default number of partitions configured for each automatically created topic.
+ | The default number of partitions configured for each automatically created topic.
|
-1 ~ 200
+ | 1–200
|
-3
+ | 3
|
-group.min.session.timeout.ms
+ | group.min.session.timeout.ms
|
-The minimum session timeout (in ms) for consumers. A shorter timeout enables quicker failure detection but results in more frequent consumer heartbeating, which can overwhelm broker resources.
+ | Minimum session timeout for consumers. A shorter timeout enables quicker failure detection but results in more frequent consumer heartbeating, which can overwhelm broker resources.
|
-6000–300,000
+ | 6000–300,000
+Unit: millisecond
|
-6000
+ | 6000
|
Table 3 Dynamic parameters (2.3.0/2.7/3.x instances)Parameter
+Table 3 Dynamic parameters (2.3.0/2.7/3.x instances)Parameter
|
-Description
+ | Description
|
-Value Range
+ | Value Range
|
-Default Value
+ | Default Value
|
|---|
-min.insync.replicas
+ | min.insync.replicas
|
-If a producer sets the acks parameter to all (or -1), the min.insync.replicas parameter specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful.
+ | If a producer sets the acks parameter to all (or -1), the min.insync.replicas parameter specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful.
|
-1–3
+ | 1–3
|
-1
+ | 1
|
-message.max.bytes
+ | message.max.bytes
|
-Maximum length of a single message, in bytes.
+ | Maximum length of a single message.
|
-0–10,485,760
+ | 0–10,485,760
+Unit: byte
|
-10,485,760
+ | 10,485,760
|
-auto.create.groups.enable
+ | auto.create.groups.enable
|
-Whether to automatically create consumer groups.
+ | Whether to automatically create consumer groups.
This parameter is not displayed on the console for some earlier instances. The function of automatically creating consumer groups is enabled by default and cannot be disabled on the console.
|
-true/false
+ | true/false
|
-true
+ | true
|
-max.connections.per.ip
+ | max.connections.per.ip
|
-The maximum number of connections allowed from each IP address. Request for new connections will be rejected once the limit is reached.
+ | The maximum number of connections allowed from each IP address. Request for new connections will be rejected once the limit is reached.
|
-100–20,000
+ | 100–20,000
|
-1000
+ | 1000
|
-unclean.leader.election.enable
+ | unclean.leader.election.enable
|
-Indicates whether to allow replicas not in the ISR set to be elected as the leader as a last resort, even though doing so may result in data loss.
+ | Indicates whether to allow replicas not in the ISR set to be elected as the leader as a last resort, even though doing so may result in data loss.
|
-true or false
+ | true or false
|
-false
+ | false
|
-offsets.retention.minutes
+ | offsets.retention.minutes
|
-The longest period a consumption position can be retained starts from the time of submission. Positions retained beyond this duration will be deleted. Each time a consumption position is submitted to a topic partition, its retention period resets to 0. The unit is minute.
+ | The longest period a consumption position can be retained starts from the time of submission. Positions retained beyond this duration will be deleted. Each time a consumption position is submitted to a topic partition, its retention period resets to 0.
This parameter is displayed as a static one for certain earlier instances.
|
-1440–30240
+ | 1440–30240
+Unit: minute
|
-20160
+ | 20160
|
Table 4 Static parameters (2.3.0/2.7/3.x instances)Parameter
+Table 4 Static parameters (2.3.0/2.7/3.x instances)Parameter
|
-Description
+ | Description
|
-Value Range
+ | Value Range
|
-Default Value
+ | Default Value
|
|---|
-connections.max.idle.ms
+ | connections.max.idle.ms
|
-Idle connection timeout (in ms). Connections that are idle for the duration specified by this parameter will be closed.
+ | Idle connection timeout (in ms). Connections that are idle for the duration specified by this parameter will be closed.
|
-5000–600,000
+ | 5000–600,000
+Unit: millisecond
|
-600,000
+ | 600,000
|
-log.retention.hours
+ | log.retention.hours
|
-Duration (in hours) for retaining a log file.
+ | Maximum duration for storing log files.
This parameter takes effect only for topics that have no aging time configured. If there is aging time configured for topics, it overrides this parameter.
|
-1–168
+ | 1–168
+Unit: hour
|
-72
+ | 72
|
-group.max.session.timeout.ms
+ | group.max.session.timeout.ms
|
-The maximum session timeout (in ms) for consumers. A longer timeout gives consumers more time to process messages between heartbeats but results in a longer time to detect failures.
+ | Maximum session timeout for consumers. A longer timeout gives consumers more time to process messages between heartbeats but results in a longer time to detect failures.
|
-6000–1,800,000
+ | 6000–1,800,000
+Unit: millisecond
|
-1,800,000
+ | 1,800,000
|
-default.replication.factor
+ | default.replication.factor
|
-The default number of replicas configured for an automatically created topic.
+ | The default number of replicas configured for an automatically created topic.
|
-1–3
+ | 1–3
|
-3
+ | 3
|
-allow.everyone.if.no.acl.found
+ | allow.everyone.if.no.acl.found
|
-When this parameter is set to true, all users can access resources without ACL rules.
+ | When this parameter is set to true, all users can access resources without ACL rules.
This parameter is displayed only when is enabled for the instance or ciphertext access is used.
This parameter of some earlier instances cannot be modified.
|
-true/false
+ | true/false
|
-true
+ | true
|
-num.partitions
+ | num.partitions
|
-The default number of partitions configured for each automatically created topic.
+ | The default number of partitions configured for each automatically created topic.
|
-1 ~ 200
+ | 1 ~ 200
|
-3
+ | 3
|
-group.min.session.timeout.ms
+ | group.min.session.timeout.ms
|
-The minimum session timeout (in ms) for consumers. A shorter timeout enables quicker failure detection but results in more frequent consumer heartbeating, which can overwhelm broker resources.
+ | Minimum session timeout for consumers. A shorter timeout enables quicker failure detection but results in more frequent consumer heartbeating, which can overwhelm broker resources.
|
-6000–300,000
+ | 6000–300,000
+Unit: millisecond
|
-6000
+ | 6000
|
diff --git a/docs/dms/umn/kafka-ug-0011.html b/docs/dms/umn/kafka-ug-0011.html
index f13db34a..0a993498 100644
--- a/docs/dms/umn/kafka-ug-0011.html
+++ b/docs/dms/umn/kafka-ug-0011.html
@@ -9,7 +9,7 @@
- Querying the Kafka Consumer Group List
-- Viewing Kafka Consumer Details
+ - Viewing Kafka Consumer Information
- Viewing and Resetting Kafka Consumption Offsets
diff --git a/docs/dms/umn/kafka-ug-0012.html b/docs/dms/umn/kafka-ug-0012.html
index a43395e4..0ac22f8d 100644
--- a/docs/dms/umn/kafka-ug-0012.html
+++ b/docs/dms/umn/kafka-ug-0012.html
@@ -3,10 +3,10 @@
Deleting a Kafka Consumer Group
You can delete a consumer group in either of the following ways:
- Method 1: Delete a consumer group on the console.
- Method 2: Use Kafka CLI to delete a consumer group. (Ensure that the Kafka instance version is the same as the CLI version.)
- PrerequisitesThe status of the consumer group to be deleted is EMPTY.
- Constraints- If auto.create.groups.enable is set to true, the consumer group status is EMPTY, and no offset has been submitted, the system automatically deletes the consumer group 10 minutes later.
- If auto.create.groups.enable is set to false, the system does not automatically delete consumer groups. You can manually delete them.
- If a consumer group has never committed an offset, the group will be deleted after the Kafka instance restarts.
+ PrerequisitesThe status of the consumer group to be deleted is EMPTY.
+ Method 1: Deleting a Consumer Group on the Console- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose the Consumer Groups tab.
- Delete consumer groups using either of the following methods:
- Select one or more consumer groups and click Delete Consumer Group above the consumer group list.
- In the row containing the consumer group you want to delete, click Delete.
@@ -17,15 +17,53 @@
- In the displayed Delete Consumer Group dialog box, click OK.
Method 2: Using the CLI to Delete a Consumer GroupThe following uses Linux as an example.
- - Download Kafka CLI v1.1.0, v2.3.0, or v2.7.2, or v3.4.0. Ensure that the Kafka instance and the CLI are of the same version.
- Use the CLI to connect to the Kafka instance. For details, see Connecting to Kafka Using the Client (Plaintext Access) or Connecting to Kafka Using the Client (Ciphertext Access).
- In the /bin directory of the Kafka client, run the following command:
./kafka-consumer-groups.sh --bootstrap-server ${connection-address} --delete --group ${consumer-group-name}
-Example:
-[root@zk-server-1 bin]# ./kafka-consumer-groups.sh --bootstrap-server 192.168.1.245:9091,192.168.1.86:9091,192.168.1.128:9091 --delete --group bbbb
-Note: This will not show information about old Zookeeper-based consumers.
-Deletion of requested consumer groups ('bbbb') was successful.
- If ciphertext access is enabled for the Kafka instance, the --command-config {consumer.properties file with SASL authentication} parameter must be added to the preceding commands. For details about the consumer.properties file, see Connecting to Kafka Using the Client (Ciphertext Access).
-
-
-
+ - For a Kafka instance with ciphertext access disabled, run the following command in the /bin directory of the Kafka client:
./kafka-consumer-groups.sh --bootstrap-server ${connection-address} --delete --group ${consumer-group-name}
+Parameter description:
+- connection-address: can be obtained from the Connection area on the Basic Information page on the Kafka console.
- consumer-group-name: consumer group name.
+Example:
+[root@ecs-kafka bin]# ./kafka-consumer-groups.sh --bootstrap-server 192.168.xx.xx:9092,192.168.xx.xx:9092,192.168.xx.xx:9092 --delete --group group-01
+Deletion of requested consumer groups ('group-01') was successful.
+[root@ecs-kafka bin]#
+ - For a Kafka instance with ciphertext access enabled, do as follows:
- (Optional) Modify the client configuration file.
View Security Protocol in the Connection area on the Basic Information page on the Kafka console. The configuration settings vary depending on the protocol. - SASL_PLAINTEXT: Skip this step if the username and password are already set. Otherwise, create the ssl-user-config.properties file in the /config directory on the Kafka client and add the following content to the file:
security.protocol=SASL_PLAINTEXT
+# If the SASL mechanism is SCRAM-SHA-512, configure as follows:
+sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
+username="**********" \
+password="**********";
+sasl.mechanism=SCRAM-SHA-512
+# If the SASL mechanism is PLAIN, configure as follows:
+sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
+username="**********" \
+password="**********";
+sasl.mechanism=PLAIN
+Parameter description: username and password are the ones you set when enabling ciphertext access for the first time or when creating a user.
+ - SASL_SSL: Skip this step if the username, password, and SSL certificate are already set. Otherwise, create the ssl-user-config.properties file in the /config directory on the Kafka client and add the following content to the file:
security.protocol=SASL_SSL
+ssl.truststore.location={ssl_truststore_path}
+ssl.truststore.password=dms@kafka
+ssl.endpoint.identification.algorithm=
+# If the SASL mechanism is SCRAM-SHA-512, configure as follows:
+sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
+username="**********" \
+password="**********";
+sasl.mechanism=SCRAM-SHA-512
+# If the SASL mechanism is PLAIN, configure as follows:
+sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
+username="**********" \
+password="**********";
+sasl.mechanism=PLAIN
+Parameter description:
+- ssl.truststore.location: path for storing the client.jks certificate. Even in Windows, you need to use slashes (/) for the certificate path. Do not use backslashes (\), which are used by default for paths in Windows. Otherwise, the client will fail to obtain the certificate.
- ssl.truststore.password: server certificate password, which must be set to dms@kafka and cannot be changed.
- ssl.endpoint.identification.algorithm: whether to verify the certificate domain name. This parameter must be left blank, which indicates disabling domain name verification.
- username and password: username and password you set when enabling ciphertext access for the first time or when creating a user.
+
+ - In the /bin directory of the Kafka client, run the following command:
./kafka-consumer-groups.sh --bootstrap-server ${connection-address} --delete --group ${consumer-group-name} --command-config ../config/ssl-user-config.properties
+Parameter description:
+- connection-address: can be obtained from the Connection area on the Basic Information page on the Kafka console.
- consumer-group-name: consumer group name.
+Example:
+[root@ecs-kafka bin]# ./kafka-consumer-groups.sh --bootstrap-server 192.168.xx.xx:9093,192.168.xx.xx:9093,192.168.xx.xx:9093 --delete --group group-02 --command-config ../config/ssl-user-config.properties
+Deletion of requested consumer groups ('group-02') was successful.
+[root@ecs-kafka bin]#
+
+
+
diff --git a/docs/dms/umn/kafka-ug-0014.html b/docs/dms/umn/kafka-ug-0014.html
index 659d939e..363b71e8 100644
--- a/docs/dms/umn/kafka-ug-0014.html
+++ b/docs/dms/umn/kafka-ug-0014.html
@@ -1,7 +1,7 @@
Viewing and Resetting Kafka Consumption Offsets
- This section describes how to view and reset consumption offsets. Resetting consumption offsets is to change the consumption position for consumers.
+ A consumption offset indicates the consumption progress of a consumer. This section describes how to view and reset consumption offsets.
Messages may be retrieved more than once after the offset is reset. Exercise caution when performing this operation.
PrerequisitesThe consumer offset cannot be reset on the fly. You must first stop retrieval of the desired consumer group.
@@ -13,14 +13,79 @@
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose Consumer Groups.
- Click the name of the desired consumer group.
- On the Consumer Offset tab page, view the list of topics that the consumer group has subscribed to, total number of messages accumulated in the topic, message consumption progress in each partition of the topic (accumulated messages, offset, latest offset, consumer ID, consumer address, and client ID).
- (Optional) To query the consumer offsets of a specific topic, enter the topic name in the search box and press Enter.
Viewing Consumer Offsets (Kafka CLI)
- Resetting Consumer Offsets- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+ Resetting Consumer Offsets- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose the Consumer Groups tab.
- Click the name of the desired consumer group.
- On the Consumer Offset tab page, you can perform the following operations:
- To reset the consumer offset of all partitions of a single topic, click Reset Consumer Offset in the row containing the desired topic.
- To reset the consumer offset of a single partition of a single topic, click Reset Consumer Offset in the row containing the desired partition.
- To reset the consumer offset of all partitions in all topics, click One-touch Reset Consumer Offset above the list.
+ - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose the Consumer Groups tab.
- Click the name of the desired consumer group.
- On the Consumer Offset tab page, you can perform the following operations:
- To reset the consumer offset of all partitions of a single topic, click Reset Consumer Offset in the row containing the desired topic.
- To reset the consumer offset of a single partition of a single topic, click Reset Consumer Offset in the row containing the desired partition.
- To reset the consumer offset of all partitions in all topics, click One-touch Reset Consumer Offset above the list.
- In the displayed Reset Consumer Offset dialog box, set the parameters by referring to Table 1.
Table 1 Parameters for resetting the consumer offsetParameter
|
diff --git a/docs/dms/umn/kafka-ug-0015.html b/docs/dms/umn/kafka-ug-0015.html
index 9d4bcef7..2b2bc53d 100644
--- a/docs/dms/umn/kafka-ug-0015.html
+++ b/docs/dms/umn/kafka-ug-0015.html
@@ -1,21 +1,66 @@
-Viewing Kafka Consumer Details
-This section describes how to view the consumer list and consumer connection addresses.
+ Viewing Kafka Consumer Information
+ If a consumer group has consumers who are accessing a Kafka instance, you can view their connection information.
PrerequisitesThe consumer list and connection address can be viewed only when consumers in a consumer group are connected to the Kafka instance (that is, the consumer group is in the STABLE state).
Viewing the Consumer List (Console)- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose Consumer Groups.
- Click the name of the desired consumer group.
- On the Consumers tab page, view the consumer list.
In the consumer list, you can view the consumer ID, consumer address, and client ID.
+ - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click an instance to go to the instance details page.
- In the navigation pane, choose Consumer Groups.
- Click the name of the desired consumer group.
- On the Consumers tab page, view the consumer list.
In the consumer list, you can view the consumer ID, consumer address, and client ID.
- (Optional) To query a specific consumer, enter the consumer ID in the search box and press Enter.
Viewing the Consumer List (Kafka CLI)
- Viewing Consumer Connection Addresses (Console)- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+ Viewing Consumer Connection Addresses (Console)- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose Consumer Groups.
- Click the desired consumer group.
- On the Consumers tab page, view the consumer addresses.
diff --git a/docs/dms/umn/kafka-ug-0034.html b/docs/dms/umn/kafka-ug-0034.html
new file mode 100644
index 00000000..288c3684
--- /dev/null
+++ b/docs/dms/umn/kafka-ug-0034.html
@@ -0,0 +1,164 @@
+
+
+ Replicating Kafka Instance Data
+ Create a Smart Connect task to copy data unidirectionally or bidirectionally between two Kafka instances.
+ Data in the source Kafka instance is synchronized to the target Kafka instance in real time.
+
+ Notes and Constraints- This function is unavailable for single-node Kafka 3.x instances.
- A maximum of 18 Smart Connect tasks can be created for an instance.
- When you copy Kafka data, the two Kafka instances must be connected through the intranet. If they are in different VPCs, connect the network by referring to Accessing Kafka Using a VPC Endpoint Across VPCs or VPC Peering Connection.
- After a Smart Connect task is created, task parameters cannot be modified.
+
+
+ Procedure- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+
+ - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- In the navigation pane, choose Smart Connect.
- On the displayed page, click Create Task.
- For Task Name, enter a unique Smart Connect task name. Naming rules: 4–64 characters and only letters, digits, hyphens (-), or underscores (_).
- For Task Type, select Copy Kafka data.
- For Start Immediately, specify whether to execute the task immediately after the task is created. By default, the task is executed immediately. If you disable this option, you can enable it later in the task list.
- In the Current Kafka area, set the instance alias. Naming rules: 1–20 characters and only letters, digits, hyphens (-), or underscores (_).
The instance alias is used in the following scenarios:
+- If you enable Rename Topics and select Push or Both for Sync Direction, the alias of the current Kafka instance will be added to the topic names of the peer end Kafka instance. For example, if the alias of the current Kafka instance is A and the topic name of the peer end Kafka instance is test, the renamed topic will be A.test.
- After a Smart Connect task of Kafka data replication is created, a topic named mm2-offset-syncs.peer end Kafka instance alias.internal is generated for the current Kafka instance. If the task has Sync Consumer Offset enabled and uses Pull or Both for Sync Direction, a topic named peer end Kafka instance alias.checkpoints.internal is also created for the current Kafka instance. The two topics are used to store internal data. If they are deleted, data replication will fail.
+ - In the Peer Kafka area, configure the following parameters.
+ Table 1 Peer Kafka parametersParameter
+ |
+Description
+ |
+
|---|
+
+Instance Alias
+ |
+Naming rules: 1–20 characters and only letters, digits, hyphens (-), or underscores (_).
+The instance alias is used in the following scenarios:
+- If you enable Rename Topics and select Pull or Both for Sync Direction, the alias of the peer end Kafka instance will be added to the topic names of the current Kafka instance. For example, if the alias of the peer end Kafka instance is B and the topic name of the current Kafka instance is test01, the renamed topic will be B.test01.
- After a Smart Connect task of Kafka data replication is created, if the task has Sync Consumer Offset enabled and uses Push or Both for Sync Direction, a topic named current Kafka instance alias.checkpoints.internal is also created for the peer end Kafka instance. This topic is used to store internal data. If it is deleted, data replication will fail.
+ |
+
+Config Type
+ |
+Options:
+- Kafka address: Enter Kafka instance addresses.
- Instance name: Select an existing Kafka instance.
+ |
+
+Instance Name
+ |
+Set this parameter when Config Type is set to Instance name.
+Select an existing Kafka instance from the drop-down list.
+The peer Kafka instance and the current Kafka instance must be in the same VPC. Otherwise, they cannot be identified.
+ |
+
+Kafka Address
+ |
+Set this parameter when Config Type is set to Kafka address.
+Enter the IP addresses and port numbers for connecting to the Kafka instance.
+When you copy Kafka data, the two Kafka instances must be connected through the intranet. If they are in different VPCs, connect the network by referring to Accessing Kafka Using a VPC Endpoint Across VPCs or VPC Peering Connection.
+ |
+
+Authentication
+ |
+Options:
+- SASL_SSL: The Kafka instance has enabled SASL_SSL, clients can connect to it with SASL and the data will be encrypted using the SSL certificate.
- SASL_PLAINTEXT: The Kafka instance has enabled SASL_PLAINTEXT, clients can connect to it with SASL and the data will be transmitted in plaintext.
- PLAINTEXT: The instance is not using authentication.
+ |
+
+Authentication Mechanism
+ |
+Set this parameter when Authentication is set to SASL_SSL/SASL_PLAINTEXT.
+- SCRAM-SHA-512: uses the hash algorithm to generate credentials for usernames and passwords to verify identities. SCRAM-SHA-512 is more secure than PLAIN.
- PLAIN: a simple username and password verification mechanism.
+ |
+
+Username
+ |
+Set this parameter when Authentication is set to SASL_SSL/SASL_PLAINTEXT.
+Set in instance creation or user creation.
+ |
+
+Password
+ |
+Set this parameter when Authentication is set to SASL_SSL/SASL_PLAINTEXT.
+Set in instance creation or user creation.
+ |
+
+
+
+ After a Smart Connect task is created, modifying the authentication method or mechanism, or password of the peer end instance causes the synchronization task to fail. In this case, delete the current task and create another one.
+
+ - In the Rules area, configure the following parameters.
+ Table 2 Parameters for configuring data replication rulesParameter
+ |
+Description
+ |
+
|---|
+
+Sync Direction
+ |
+There are three synchronization directions:
+- Pull: Replicates data from the peer Kafka instance to the current Kafka instance.
- Push: Replicates data from the current Kafka instance to the peer Kafka instance.
- Both: Bidirectional replication of Kafka instance data on both ends.
+ |
+
+Topics
+ |
+Specify the topics whose data is to be replicated.
+- Regular expression: Use a regular expression to match topics.
- Enter/Select: Enter topic names. To enter multiple topic names, press Enter after entering each topic name. You can also select topics from the drop-down list. A maximum of 20 topics can be entered or selected.
+ NOTE: Data of topics whose names end with "internal" (for example, topic.internal) will not be synchronized.
+
+ |
+
+Tasks
+ |
+Number of data replication tasks. The default value is 2. You are advised to use the default value.
+If Sync Direction is set to Both, the actual number of tasks will be twice the number of tasks you configure here.
+ |
+
+Rename Topics
+ |
+Add the alias of the source Kafka instance before the target topic name to form a new name of the target topic. For example, if the alias of the source instance is A and the target topic name is test, the renamed target topic will be A.test.
+If you select Both for Sync Direction, enable Rename Topics to prevent infinite replication.
+ |
+
+Add Source Header
+ |
+The target topic receives the replicated messages. The message header contains the message source.
+If you select Both for Sync Direction, Add Source Header is enabled by default to prevent infinite replication.
+ |
+
+Sync Consumer Offset
+ |
+Enable this option to synchronize the consumer offset to the target Kafka instance.
+ NOTICE: After enabling Sync Consumer Offset, pay attention to the following:
+ - The source and target Kafka instances cannot consume messages at the same time. Otherwise, the synchronized consumer offset will be abnormal.
- The consumer offset is synchronized every minute. As a result, the consumer offset on the target end may be slightly smaller than that on the source end, and some messages are repeatedly consumed. The service logic of the consumer client must be able to handle repeated consumption.
- The offset synchronized from the source end is not the same as the offset on the target end. Instead, there is a mapping relationship. If the consumer offset is maintained by the consumer client, the consumer client does not obtain the consumer offset from the target Kafka instance after switching consumption from the source Kafka instance to the target Kafka instance. As a result, the offset may be incorrect or the consumer offset may be reset.
+
+ |
+
+Replicas
+ |
+Number of topic replicas when a topic is automatically created in the peer instance. The value of this parameter cannot exceed the number of brokers in the peer instance.
+This parameter takes precedence over the default.replication.factor parameter set in the peer instance.
+ |
+
+Start Offset
+ |
+Options:
+- Minimum offset: dumping the earliest data
- Maximum offset: dumping the latest data
+ |
+
+Compression
+ |
+Compression algorithm to use for copying messages.
+ |
+
+Topic Mapping
+ |
+Customize the target topic name.
+Maximum mappings: 20. Rename Topic and Topic Mapping cannot be configured at the same time.
+ |
+
+
+
+ - When creating a bidirectional replication task, you must enable Rename Topics or Add Source Header to prevent infinite replication. If you specify the same topic for a pull task and a push task between two instances (forming bidirectional replication), and Rename Topics and Add Source Header are not enabled for the two tasks, data will be replicated infinitely.
- If you create two or more tasks with the same configuration and enable Sync Consumer Offset for them, data will be repeatedly replicated and the consumer offset of the target topic will be abnormal.
+
+ - (Optional) In the lower right corner of the page, click Check to test the connectivity between the Kafka instances.
If "Connectivity check passed." is displayed, the Kafka instances are connected.
+ - Click Create. The Smart Connect task list page is displayed. The message "Task xxx was created successfully." is displayed in the upper right corner of the page.
- After a Smart Connect task of Kafka data replication is created, a topic named mm2-offset-syncs.peer end Kafka instance alias.internal is generated for the current Kafka instance. If the task has Sync Consumer Offset enabled and uses Pull or Both for Sync Direction, a topic named peer end Kafka instance alias.checkpoints.internal is also created for the current Kafka instance. The two topics are used to store internal data. If they are deleted, data replication will fail.
- After a Smart Connect task of Kafka data replication is created, if the task has Sync Consumer Offset enabled and uses Push or Both for Sync Direction, a topic named current Kafka instance alias.checkpoints.internal is also created for the peer end Kafka instance. This topic is used to store internal data. If it is deleted, data replication will fail.
+
+
+
+
+
+
diff --git a/docs/dms/umn/kafka-ug-0035.html b/docs/dms/umn/kafka-ug-0035.html
new file mode 100644
index 00000000..8e859491
--- /dev/null
+++ b/docs/dms/umn/kafka-ug-0035.html
@@ -0,0 +1,103 @@
+
+
+ Dumping Kafka Data to Object Storage Service (OBS)
+ Create a Smart Connect task to dump Kafka instance data to OBS for message data backup.
+ Data in the source Kafka instance is synchronized to the dumping file in real time.
+
+ Notes and Constraints- This function is unavailable for single-node Kafka 3.x instances.
- A maximum of 18 Smart Connect tasks can be created for an instance.
- After a Smart Connect task is created, task parameters cannot be modified.
+
+ Prerequisites- You have enabled Smart Connect.
- A Kafka instance has been created and is in the Running state.
- The OBS bucket must be created in the same region as the Kafka instance.
+
+ Procedure- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+
+ - Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- In the navigation pane, choose Smart Connect.
- On the displayed page, click Create Task.
- For Task Name, enter a unique Smart Connect task name. Naming rules: 4–64 characters and only letters, digits, hyphens (-), or underscores (_).
- For Task Type, select Dumping.
- For Start Immediately, specify whether to execute the task immediately after the task is created. By default, the task is executed immediately. If you disable this option, you can enable it later in the task list.
- In the Source area, retain the default setting.
- In the Topics area, set parameters based on the following table.
+ Table 1 Topic parametersParameter
+ |
+Description
+ |
+
|---|
+
+Regular expression
+ |
+A regular expression is used to subscribe to topics whose messages you want to dump.
+ |
+
+Enter/Select
+ |
+Enter or select the names of the topics to be dumped. Separate them with commas (,). A maximum of 20 topics can be entered or selected.
+ |
+
+
+
+ - In the Target area, set parameters based on the following table.
+ Table 2 Target parametersParameter
+ |
+Description
+ |
+
|---|
+
+Offset
+ |
+Options:
+- Minimum offset: dumping the earliest data
- Maximum offset: dumping the latest data
+ |
+
+Dumping Period (s)
+ |
+Interval for periodically dumping data. The time unit is second and the default interval is 300 seconds.
+No package files will be generated if there is no data within an interval.
+ |
+
+AK
+ |
+Access key ID.
+For details about how to obtain the AK, see Access Keys.
+ |
+
+SK
+ |
+Secret access key used together with the access key ID.
+For details about how to obtain the SK, see Access Keys.
+ |
+
+Dumping Address
+ |
+The OBS bucket used to store the topic data.
+- Select: You can select an existing OBS bucket from the drop-down list or click Create Dumping Address to create an OBS bucket.
- Enter: You can enter an existing OBS bucket or click Create Dumping Address to create an OBS bucket. The OBS bucket to be entered must be in the same region as the Kafka instance.
+ |
+
+Dumping Directory
+ |
+Directory for storing topic files dumped to OBS. Use slashes (/) to separate directory levels.
+ |
+
+Time Directory Format
+ |
+Data is saved to a hierarchical time directory in the dumping directory. For example, if the time directory is accurate to day, the directory will be in the format of bucket name/file directory/year/month/day.
+ |
+
+Record Separator
+ |
+Select a separator to separate OBS dumping records.
+ |
+
+Use Storage Key
+ |
+Specifies whether to dump keys.
+ |
+
+
+
+ Do not use the key of a message as the dumping file name.
+
+ - Click Create. The Smart Connect task list page is displayed. The message "Task xxx was created successfully." is displayed in the upper right corner of the page.
+
+
+
+
diff --git a/docs/dms/umn/kafka-ug-0038.html b/docs/dms/umn/kafka-ug-0038.html
index eecbf6cf..df595de2 100644
--- a/docs/dms/umn/kafka-ug-0038.html
+++ b/docs/dms/umn/kafka-ug-0038.html
@@ -2,51 +2,51 @@
Modifying Kafka Topic Configurations
This section describes how to modify following configurations of a Kafka topic on the console.
+ Modifying Synchronous Replication, Synchronous Flushing, Message Timestamp, or Max. Message Size does not require instance restart.
+
- Table 1 Kafka topic configuration parametersParameter
+Table 1 Kafka topic configuration parametersParameter
|
-Description
+ | Description
|
|---|
-Partitions
+ | Partitions
|
-Number of partitions in a topic. For details about how to change, see Changing Kafka Partition Quantity.
+ | Number of partitions in a topic. For details about how to change, see Changing Kafka Partition Quantity.
|
-Aging Time (h)
+ | Aging Time (h)
|
-Maximum message retention. For details about how to change, see Changing Kafka Message Retention Period.
+ | Maximum message retention. For details about how to change, see Changing Kafka Message Retention Period.
|
-Synchronous Replication
+ | Synchronous Replication
|
-A message is returned to the client only after the message creation request has been received and the message has been acknowledged by all replicas.
+ | A message is returned to the client only after the message creation request has been received and the message has been acknowledged by all replicas.
|
-Synchronous Flushing
+ | Synchronous Flushing
|
-- Enabled: A message is immediately flushed to disk once it is created, bringing higher reliability.
- Disabled: A message is stored in the memory instead of being immediately flushed to disk once created.
+ | - Enabled: A message is immediately flushed to disk once it is created, bringing higher reliability.
- Disabled: A message is stored in the memory instead of being immediately flushed to disk once created.
|
-Message Timestamp
+ | Message Timestamp
|
-Timestamp type of a message. Options: - CreateTime: time when the producer created the message.
- LogAppendTime: time when the broker appended the message to the log.
+ Timestamp type of a message. Options: - CreateTime: time when the producer created the message.
- LogAppendTime: time when the broker appended the message to the log.
|
|
-Max. Message Size
+ | Max. Message Size
|
-Maximum size of messages to be processed in batches. If message compression is enabled, this parameter indicates the size after compression.
+ | Maximum size of messages to be processed in batches. If message compression is enabled, this parameter indicates the size after compression.
If this value is increased and the consumer version is earlier than 0.10.2, the consumers' fetch size must also be increased so that they can obtain the latest value.
|
Modifying Synchronous Replication, Synchronous Flushing, Message Timestamp, or Max. Message Size does not require instance restart.
-
-Modifying Topic Configurations- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+ Procedure- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose Topics.
- Modify topic configurations in either of the following ways:
- Select one or more topics and click Edit Topic above the topic list.
- In the row containing the desired topic, click Edit.
- In the Edit Topic dialog box, change configurations and click OK.
- If there is only one replica, Synchronous Replication cannot be enabled.
- After enabling synchronous replication, set acks to all or –1 on the client. Otherwise, this function will not take effect.
diff --git a/docs/dms/umn/kafka-ug-0041.html b/docs/dms/umn/kafka-ug-0041.html
index 2b7312a6..e07af3ab 100644
--- a/docs/dms/umn/kafka-ug-0041.html
+++ b/docs/dms/umn/kafka-ug-0041.html
@@ -1,23 +1,27 @@
Creating a Kafka Consumer Group
- Create a consumer group on the console.
- auto.create.groups.enable: a consumer group is automatically created when a consumer attempts to enter a group that does not exist.
- - This operation is optional when auto.create.groups.enable is true in Configuring Parameters.
- A consumer group is required before consuming messages when auto.create.groups.enable is false in Configuring Parameters. Otherwise, consumption will fail.
- - If auto.create.groups.enable is set to true, the consumer group status is EMPTY, and no offset has been submitted, the system automatically deletes the consumer group 10 minutes later.
- If auto.create.groups.enable is set to false, the system does not automatically delete consumer groups. You can manually delete them.
- If a consumer group has never committed an offset, the group will be deleted after the Kafka instance restarts.
- Creating a consumer group on the console does not require instance restart.
+ A consumer subscribes to a topic. A consumer group consists of one or more consumers. Within a consumer group, each consumer can consume multiple partitions at the same time. Each partition can be consumed by one consumer at a time.
+ Figure 1 Example consumption
+ auto.create.groups.enable: a consumer group is automatically created when a consumer attempts to enter a group that does not exist.
+ - A consumer group is required before consuming messages when auto.create.groups.enable is false in Configuring Parameters. Otherwise, consumption will fail.
- A consumer group is created automatically before consuming messages when auto.create.groups.enable is true in Configuring Parameters.
+ Create a consumer group on the console.
+ Creating a consumer group on the console does not require an instance restart.
- Creating a Kafka Consumer Group- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+ Notes and Constraints- If auto.create.groups.enable is set to true, the consumer group status is EMPTY, and no offset has been submitted, the system automatically deletes the consumer group 10 minutes later.
- If auto.create.groups.enable is set to false, the system does not automatically delete consumer groups. You can manually delete them.
- If a consumer group has never committed an offset, the group will be deleted after the Kafka instance restarts.
+
+ Procedure- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view the instance details.
- In the navigation pane, choose Consumer Groups.
- Click Create Consumer Group.
- Set consumer group parameters by referring to Table 1 and click OK.
- Table 1 Consumer group parametersParameter
+Table 1 Consumer group parametersParameter
|
-Description
+ | Description
|
|---|
-Consumer Group Name
+ | Consumer Group Name
|
-Enter 3 to 64 characters, starting with a letter or underscore (_). Use only letters, digits, periods (.), hyphens (-), and underscores (_).
+ | Enter 3 to 64 characters, starting with a letter or underscore (_). Use only letters, digits, periods (.), hyphens (-), and underscores (_).
If a consumer group name starts with a special character, for example, an underscore (_) or a number sign (#), monitoring data cannot be displayed.
|
diff --git a/docs/dms/umn/kafka-ug-0053.html b/docs/dms/umn/kafka-ug-0053.html
index 4a5b1b8f..a8018aea 100644
--- a/docs/dms/umn/kafka-ug-0053.html
+++ b/docs/dms/umn/kafka-ug-0053.html
@@ -1,9 +1,8 @@
Exporting the Kafka Instance List
-ScenarioThis section describes how to export the Kafka instance list from the console.
-
- Exporting the Kafka Instance List- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+ You can export a list of instances on the DMS console.
+ Procedure- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Export the instance list using either of the following methods:
- Select the desired instances and choose Export > Export selected data to an XLSX file to export specified instances.
- Choose Export > Export all data to an XLSX file to export all instances.
diff --git a/docs/dms/umn/kafka-ug-0056.html b/docs/dms/umn/kafka-ug-0056.html
index f9ba335c..6772212b 100644
--- a/docs/dms/umn/kafka-ug-0056.html
+++ b/docs/dms/umn/kafka-ug-0056.html
@@ -1,9 +1,8 @@
Exporting Kafka Consumer Groups
- ScenarioExport the consumer group list from the console.
-
- Exporting Kafka Consumer Groups- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+ You can export a list of consumer groups in a Kafka instance.
+ Procedure- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click the desired Kafka instance to view its details.
- In the navigation pane, choose Consumer Groups.
- Export consumer groups in either of the following ways:
- Select the desired consumer groups and choose Export > Export selected data to an XLSX file to export specified consumer groups.
- Choose Export > Export all data to an XLSX file to export all consumer groups.
diff --git a/docs/dms/umn/kafka-ug-0061.html b/docs/dms/umn/kafka-ug-0061.html
index 2624f327..e4afd796 100644
--- a/docs/dms/umn/kafka-ug-0061.html
+++ b/docs/dms/umn/kafka-ug-0061.html
@@ -10,7 +10,7 @@
diff --git a/docs/dms/umn/kafka-ug-0065.html b/docs/dms/umn/kafka-ug-0065.html
new file mode 100644
index 00000000..26643117
--- /dev/null
+++ b/docs/dms/umn/kafka-ug-0065.html
@@ -0,0 +1,27 @@
+
+
+
+ Migrating Data Using Smart Connect
+
+
+
+
+
diff --git a/docs/dms/umn/kafka-ug-0069.html b/docs/dms/umn/kafka-ug-0069.html
index 560f0e56..7d9444ae 100644
--- a/docs/dms/umn/kafka-ug-0069.html
+++ b/docs/dms/umn/kafka-ug-0069.html
@@ -1,13 +1,12 @@
Process of Using Kafka
- The following figure shows the process of using a Kafka instance to produce and consume messages.
+ Distributed Message Service is a message queuing service that is based on the open-source Apache Kafka. It provides Kafka instances with isolated computing, storage, and bandwidth resources. The following figure shows the process of message production and consumption using a Kafka instance.
Figure 1 Process of using Kafka
- Creating a User and Granting DMS for Kafka Permissions
Create IAM users and grant them only the DMS for Kafka permissions required to perform a given task based on their job responsibilities.
- Creating a Kafka Instance
Kafka instances are tenant-exclusive, and physically isolated in deployment.
- Creating a Kafka Topic
Create a topic for storing messages so that producers can produce messages and consumers can subscribe to messages.
- Connecting to an Instance
The client uses commands to connect to Kafka instances in a private or public network, and produces and consumes messages.
- - Managing Messages
View messages on the console.
diff --git a/docs/dms/umn/kafka-ug-0319001.html b/docs/dms/umn/kafka-ug-0319001.html
index 616f1639..37180e93 100644
--- a/docs/dms/umn/kafka-ug-0319001.html
+++ b/docs/dms/umn/kafka-ug-0319001.html
@@ -5,7 +5,7 @@
If you no longer need public access to the instance, you can disable it as required.
Prerequisites- You can change the public access setting only when the Kafka instance is in the Running state.
- Kafka instances only support IPv4 EIPs. IPv6 EIPs are not supported.
- Enabling Public Access- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+ Enabling Public Access- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click a Kafka instance to go to the Basic Information page.
- Click
 next to Public Access to enable public access. For Elastic IP Address, select an EIP for each broker. If the number of EIPs is insufficient, click Create Elastic IP to go to the Create EIP page and create EIPs. For details, see Assigning an EIP. After the creation is complete, return to the page for enabling the public access. Click  next to Elastic IP Address, select EIPs from the drop-down list. The number of EIPs must be the same as the number of brokers. Then click  and the Background Tasks page is displayed.If the status of the task turns to Successful, public access is successfully enabled. Figure 1 Enabling public access
- Disabling Public Access- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
+ Disabling Public Access- Log in to the console.
- Click in the upper left corner to select a region.
Select the region where your Kafka instance is located.
- Click Service List and choose Application > Distributed Message Service. The Kafka instance list is displayed.
- Click a Kafka instance to go to the Basic Information page.
- Before disabling public access, disable Plaintext Access and Ciphertext Access next to Public Network Access. Then click
 next to Public Access. - Click OK. The Background Tasks page is displayed. If the status of the task turns to Successful, public access is successfully disabled.
After public access is disabled, configure security group rules listed in Table 2 before attempting to access Kafka in a VPC. For details about accessing Kafka, see Connecting to an Instance.
|
|---|
|
|---|
|---|
|
|---|
|
|---|
|
|---|
|
|---|
|
