This section describes how to use an open-source Kafka client to access a Kafka instance in plaintext. Two scenarios: private network within a VPC and public network. To access a Kafka instance across VPCs, see Accessing Kafka Using a VPC Endpoint Across VPCs. To access a Kafka instance using DNAT, see Accessing Kafka in a Public Network Using DNAT. To access an instance in your service code, see the Distributed Message Service Developer Guide.
For details on how to use a Kafka client in different languages, visit https://cwiki.apache.org/confluence/display/KAFKA/Clients.
Each Kafka broker allows a maximum of 1000 connections from each IP address. Excess connections will be rejected. You can change the limit by referring to Modifying Kafka Instance Configuration Parameters, that is, to modify parameter max.connections.per.ip.
To access a Kafka instance with ciphertext access disabled, configure proper security group rules. For details about security group configuration requirements, see Table 2.
Add the following lines to the .bash_profile file in the home directory as an authorized user. In this command, /opt/java/jdk1.8.0_151 is the JDK installation path. Change it to the path where you install JDK.
export JAVA_HOME=/opt/java/jdk1.8.0_151 export PATH=$JAVA_HOME/bin:$PATH
Run the source .bash_profile command for the modification to take effect.
The following uses Linux as an example.
Access the directory where the CLI package is stored and run the following command to decompress the package:
tar -zxf [kafka_tar]
In the preceding command, [kafka_tar] indicates the name of the CLI package.
For example:
tar -zxf kafka_2.12-2.7.2.tgz
In Windows, you need to access the /bin/windows directory.
./kafka-console-producer.sh --broker-list ${connection-address} --topic ${topic-name}
Parameter description:
The following example uses connection addresses 10.xx.xx.45:9094,10.xx.xx.127:9094,10.xx.xx.103:9094. After running the preceding command, you can send a message to the Kafka instance by writing it and pressing Enter. Each line of content is sent as a message.
[root@ecs-kafka bin]# ./kafka-console-producer.sh --broker-list 10.xx.xx.45:9094,10.xx.xx.127:9094,10.xx.xx.103:9094 --topic topic-demo >Hello >DMS >Kafka! >^C[root@ecs-kafka bin]#
To stop creating messages, press Ctrl+C to exit.
./kafka-console-consumer.sh --bootstrap-server ${connection-address} --topic ${topic-name} --group ${consumer-group-name} --from-beginning
Parameter description:
Example:
[root@ecs-kafka bin]# ./kafka-console-consumer.sh --bootstrap-server 10.xx.xx.45:9094,10.xx.xx.127:9094,10.xx.xx.103:9094 --topic topic-demo --group order-test --from-beginning Kafka! DMS Hello ^CProcessed a total of 3 messages [root@ecs-kafka bin]#
To stop retrieving messages, press Ctrl+C to exit.