Getting Started with Kafka for Message Production and Consumption

To achieve fine-grained management of your cloud resources, create IAM user groups and users and grant specified permissions to the users. For more information, see Creating a User and Granting DMS for Kafka Permissions.

Configure the VPC and subnet for Kafka instances as required. You can use the current account's existing VPC and subnet, or create new ones. For details about how to create a VPC and a subnet, see Creating a VPC. Note that the VPC must be in the same region as the Kafka instance.

Configure the security group for Kafka instances as required. You can use the current account's existing security groups, or create new ones. For details about how to create a security group, see Creating a Security Group.

To connect to Kafka instances, add the security group rules described in Table 1. Other rules can be added based on site requirements.

Table 1 Security group rules

Direction	Protocol	Port	Source address	Description
Inbound	TCP	9093	0.0.0.0/0	Accessing a Kafka instance over a private network within a VPC (in ciphertext)

After a security group is created, it has a default inbound rule that allows communication among ECSs within the security group and a default outbound rule that allows all outbound traffic. If you access your Kafka instance using the private network within a VPC, you do not need to add the rules described in Table 1.

It takes 3 to 15 minutes to create an instance. During this period, the instance status is Creating.

• If the instance is created successfully, its status changes to Running.
• If the instance is in the Creation failed state, delete it. Then create a new one. If the instance creation fails again, contact customer service.
  

  Instances that fail to be created do not occupy other resources.

1. In the navigation pane, click Basic Information.
2. In the Connection area, view the connection address.
   Figure 2 Kafka instance addresses (private network) for intra-VPC access
   

1. Log in to a Linux ECS.
2. Download the client.jks certificate and upload it to the /root directory on the ECS.

   To obtain the certificate: On the Kafka console, click the Kafka instance to go to the Basic Information page. Click Download next to SSL Certificate in the Connection area. Decompress the package to obtain the client certificate file client.jks.

   

   /root is the path for storing the certificate. Change it to the actual path if needed.

3. Go to the /config directory on the Kafka client.

   cd kafka_2.12-2.7.2/config

4. Add the following commands in both the consumer.properties and producer.properties files (PLAIN is used as an example).

   sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
   username="**********" \
   password="**********";        
   sasl.mechanism=PLAIN
   
   security.protocol=SASL_SSL
   ssl.truststore.location={ssl_truststore_path}
   ssl.truststore.password=dms@kafka
   ssl.endpoint.identification.algorithm=

   Description:

   • username and password are specified when enabling ciphertext access during instance creation.
   • ssl.truststore.location is the path for storing the certificate obtained in 2.b.
   • ssl.truststore.password is certified by the server, which must be set to dms@kafka and cannot be changed.
   • ssl.endpoint.identification.algorithm decides whether to verify the certificate domain name. In this example, leave this parameter blank, which indicates disabling domain name verification.

cd ../bin

./kafka-console-producer.sh --broker-list ${connection addr} --topic ${topic name} --producer.config ../config/producer.properties

Description:

• {connection addr}: the address obtained in 1.
• {topic name}: the topic name obtained in 4.

For example, 192.xxx.xxx.xxx:9093, 192.xxx.xxx.xxx:9093, 192.xxx.xxx.xxx:9093 are the connection addresses of the Kafka instance.

After running this command, you can send messages to the Kafka instance by entering the information as prompted and pressing Enter. Each line of content will be sent as a message.

[root@ecs-kafka bin]#./kafka-console-producer.sh --broker-list 192.xxx.xxx.xxx:9093,192.xxx.xxx.xxx:9093,192.xxx.xxx.xxx:9093  --topic topic-demo --producer.config ../config/producer.properties
>Hello
>DMS
>Kafka!
>^C[root@ecs-kafka bin]# 

Press Ctrl+C to cancel.

./kafka-console-consumer.sh --bootstrap-server ${connection addr} --topic ${topic name} --from-beginning  --consumer.config ../config/consumer.properties

Description:

• {connection addr}: the address obtained in 1.
• {topic name}: the topic name obtained in 4.

Sample:

[root@ecs-kafka bin]#  ./kafka-console-consumer.sh --bootstrap-server 192.xxx.xxx.xxx:9093,192.xxx.xxx.xxx:9093,192.xxx.xxx.xxx:9093 --topic topic-demo --from-beginning --consumer.config ../config/consumer.properties
Hello
Kafka!
DMS
^CProcessed a total of 3 messages
[root@ecs-kafka bin]# 

Press Ctrl+C to cancel.