forked from docs/doc-exports
Chen, Junjie
82285f686d
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Chen, Junjie <chenjunjie@huawei.com> Co-committed-by: Chen, Junjie <chenjunjie@huawei.com>
156 lines
15 KiB
HTML
156 lines
15 KiB
HTML
<a name="how-to-connect-kafka"></a><a name="how-to-connect-kafka"></a>
|
|
|
|
<h1 class="topictitle1">Setting Up the Java Development Environment</h1>
|
|
<div id="body1552118864471"><p id="how-to-connect-kafka__p8060118">With the information collected in <a href="Kafka-config.html">Collecting Connection Information</a> and the network environment prepared for Kafka clients, you can proceed to configuring Kafka clients. This section describes how to configure Kafka clients to produce and consume messages.</p>
|
|
<div class="section" id="how-to-connect-kafka__section18653910637"><h4 class="sectiontitle">Preparing Tools</h4><ul id="how-to-connect-kafka__ul68581118696"><li class="MsoNormal" id="how-to-connect-kafka__li17829920161514">Maven<p class="MsoNormal" id="how-to-connect-kafka__p18476921151514"><a name="how-to-connect-kafka__li17829920161514"></a><a name="li17829920161514"></a>Apache Maven 3.0.3 or later can be downloaded from the <a href="https://maven.apache.org/download.cgi" target="_blank" rel="noopener noreferrer">Maven official website</a>.</p>
|
|
</li><li class="MsoNormal" id="how-to-connect-kafka__li342101731515">JDK<p class="MsoNormal" id="how-to-connect-kafka__p2816417121511"><a name="how-to-connect-kafka__li342101731515"></a><a name="li342101731515"></a>Java Development Kit1.8.111 or later can be downloaded from the <a href="https://www.oracle.com/technetwork/java/javase/downloads/index.html" target="_blank" rel="noopener noreferrer">Oracle official website</a>.</p>
|
|
<p id="how-to-connect-kafka__p1785412212159">After the installation, configure the Java environment variables.</p>
|
|
</li><li class="MsoNormal" id="how-to-connect-kafka__li06001693150">IntelliJ IDEA<p class="MsoNormal" id="how-to-connect-kafka__p89562014191512"><a name="how-to-connect-kafka__li06001693150"></a><a name="li06001693150"></a>IntelliJ IDEA can be downloaded from the <a href="https://www.jetbrains.com/idea/" target="_blank" rel="noopener noreferrer">IntelliJ IDEA official website</a> and be installed.</p>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="how-to-connect-kafka__section1271819150319"><h4 class="sectiontitle">Procedure</h4><ol id="how-to-connect-kafka__ol154691943125519"><li id="how-to-connect-kafka__li16469124345512"><span><a href="https://dms-demo.obs.eu-de.otc.t-systems.com/Kafka-sdk-java.zip" target="_blank" rel="noopener noreferrer">Download the demo package</a>.</span><p><p id="how-to-connect-kafka__p14701043135510">Decompress the package to obtain the following files.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="how-to-connect-kafka__table16897190174718" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Files in the demo package</caption><thead align="left"><tr id="how-to-connect-kafka__row489818014475"><th align="left" class="cellrowborder" valign="top" width="26.49264926492649%" id="mcps1.3.3.2.1.2.2.2.4.1.1"><p id="how-to-connect-kafka__p178981602471">File</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="31.67316731673167%" id="mcps1.3.3.2.1.2.2.2.4.1.2"><p id="how-to-connect-kafka__p58986018474">Directory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="41.83418341834184%" id="mcps1.3.3.2.1.2.2.2.4.1.3"><p id="how-to-connect-kafka__p1489817084713">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="how-to-connect-kafka__row7898160184710"><td class="cellrowborder" valign="top" width="26.49264926492649%" headers="mcps1.3.3.2.1.2.2.2.4.1.1 "><p id="how-to-connect-kafka__p17898160104718">DmsConsumer.java</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="31.67316731673167%" headers="mcps1.3.3.2.1.2.2.2.4.1.2 "><p id="how-to-connect-kafka__p889870184713">.\src\main\java\com\dms\consumer</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.83418341834184%" headers="mcps1.3.3.2.1.2.2.2.4.1.3 "><p id="how-to-connect-kafka__p2089850144717">API for consuming messages</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="how-to-connect-kafka__row15898170154712"><td class="cellrowborder" valign="top" width="26.49264926492649%" headers="mcps1.3.3.2.1.2.2.2.4.1.1 "><p id="how-to-connect-kafka__p48987014711">DmsProducer.java</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="31.67316731673167%" headers="mcps1.3.3.2.1.2.2.2.4.1.2 "><p id="how-to-connect-kafka__p10898502470">.\src\main\java\com\dms\producer</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.83418341834184%" headers="mcps1.3.3.2.1.2.2.2.4.1.3 "><p id="how-to-connect-kafka__p168988094716">API for producing messages</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="how-to-connect-kafka__row17898907476"><td class="cellrowborder" valign="top" width="26.49264926492649%" headers="mcps1.3.3.2.1.2.2.2.4.1.1 "><p id="how-to-connect-kafka__p4898208477">dms.sdk.consumer.properties</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="31.67316731673167%" headers="mcps1.3.3.2.1.2.2.2.4.1.2 "><p id="how-to-connect-kafka__p38981103477">.\src\main\resources</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.83418341834184%" headers="mcps1.3.3.2.1.2.2.2.4.1.3 "><p id="how-to-connect-kafka__p108984014715">Configuration information for consuming messages</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="how-to-connect-kafka__row14898702475"><td class="cellrowborder" valign="top" width="26.49264926492649%" headers="mcps1.3.3.2.1.2.2.2.4.1.1 "><p id="how-to-connect-kafka__p2089811054718">dms.sdk.producer.properties</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="31.67316731673167%" headers="mcps1.3.3.2.1.2.2.2.4.1.2 "><p id="how-to-connect-kafka__p38983012477">.\src\main\resources</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.83418341834184%" headers="mcps1.3.3.2.1.2.2.2.4.1.3 "><p id="how-to-connect-kafka__p138994094715">Configuration information for producing messages</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="how-to-connect-kafka__row1189970194712"><td class="cellrowborder" valign="top" width="26.49264926492649%" headers="mcps1.3.3.2.1.2.2.2.4.1.1 "><p id="how-to-connect-kafka__p889930184714">client.truststore.jks</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="31.67316731673167%" headers="mcps1.3.3.2.1.2.2.2.4.1.2 "><p id="how-to-connect-kafka__p1789980124713">.\src\main\resources</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.83418341834184%" headers="mcps1.3.3.2.1.2.2.2.4.1.3 "><p id="how-to-connect-kafka__p589940154713">SSL certificate, used for SASL connection</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="how-to-connect-kafka__row1623641494915"><td class="cellrowborder" valign="top" width="26.49264926492649%" headers="mcps1.3.3.2.1.2.2.2.4.1.1 "><p id="how-to-connect-kafka__p9237191484920">DmsConsumerTest.java</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="31.67316731673167%" headers="mcps1.3.3.2.1.2.2.2.4.1.2 "><p id="how-to-connect-kafka__p162371141499">.\src\test\java\com\dms\consumer</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.83418341834184%" headers="mcps1.3.3.2.1.2.2.2.4.1.3 "><p id="how-to-connect-kafka__p17237314144917">Test code of consuming messages</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="how-to-connect-kafka__row651411464916"><td class="cellrowborder" valign="top" width="26.49264926492649%" headers="mcps1.3.3.2.1.2.2.2.4.1.1 "><p id="how-to-connect-kafka__p13515151420490">DmsProducerTest.java</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="31.67316731673167%" headers="mcps1.3.3.2.1.2.2.2.4.1.2 "><p id="how-to-connect-kafka__p6515131418495">.\src\test\java\com\dms\producer</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.83418341834184%" headers="mcps1.3.3.2.1.2.2.2.4.1.3 "><p id="how-to-connect-kafka__p15159142490">Test code of producing messages</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="how-to-connect-kafka__row79213153494"><td class="cellrowborder" valign="top" width="26.49264926492649%" headers="mcps1.3.3.2.1.2.2.2.4.1.1 "><p id="how-to-connect-kafka__p793715114914">pom.xml</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="31.67316731673167%" headers="mcps1.3.3.2.1.2.2.2.4.1.2 "><p id="how-to-connect-kafka__p1493161554910">.\</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="41.83418341834184%" headers="mcps1.3.3.2.1.2.2.2.4.1.3 "><p id="how-to-connect-kafka__p19317151493">Maven configuration file, containing the Kafka client dependencies</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="how-to-connect-kafka__li11470144314552"><span>In IntelliJ IDEA, import the demo project.</span><p><div class="p" id="how-to-connect-kafka__p117771615353">The demo project is a Java project built in Maven. Therefore, you need the JDK and the Maven plugin in IDEA.<div class="fignone" id="how-to-connect-kafka__fig12105131513352"><span class="figcap"><b>Figure 1 </b>Click <strong id="how-to-connect-kafka__b6235122816456">Import Project</strong>.</span><br><span><img id="how-to-connect-kafka__image610511514352" src="en-us_image_0171660980.png"></span></div>
|
|
<div class="fignone" id="how-to-connect-kafka__fig110571543514"><span class="figcap"><b>Figure 2 </b>Choose <strong id="how-to-connect-kafka__b17766124294516">Maven</strong>.</span><br><span><img id="how-to-connect-kafka__image1110510157353" src="en-us_image_0171660966.png"></span></div>
|
|
</div>
|
|
<div class="fignone" id="how-to-connect-kafka__fig115294167375"><span class="figcap"><b>Figure 3 </b>Select the JDK.</span><br><span><img id="how-to-connect-kafka__image14105191515352" src="en-us_image_0171660975.png"></span></div>
|
|
<p id="how-to-connect-kafka__p17105181519352">You can select other options or retain the default settings. Click <strong id="how-to-connect-kafka__b232805517565">Finish</strong>.</p>
|
|
<p id="how-to-connect-kafka__p19484162217363">The demo project has been imported.</p>
|
|
<p id="how-to-connect-kafka__p185285103610"><span><img id="how-to-connect-kafka__image18982152463710" src="en-us_image_0171660939.png"></span></p>
|
|
</p></li><li id="how-to-connect-kafka__li54701543155512"><span>Configure Maven.</span><p><p class="MsoNormal" id="how-to-connect-kafka__p1256543315148">Choose <strong id="how-to-connect-kafka__b3285165201814">Files</strong> > <strong id="how-to-connect-kafka__b1159718138181">Settings</strong>, set <strong id="how-to-connect-kafka__b11644172152210">Maven home directory</strong> correctly, and select the required <strong id="how-to-connect-kafka__b155112732216">settings.xml</strong> file.</p>
|
|
<p id="how-to-connect-kafka__p201751061418"><span><img id="how-to-connect-kafka__image38584015142" src="en-us_image_0171660970.png"></span></p>
|
|
</p></li><li id="how-to-connect-kafka__li10470164310559"><span>Specify Kafka configurations.</span><p><p id="how-to-connect-kafka__p278910111468">The following is a configuration example for producing messages. Replace the information in bold with the actual values.</p>
|
|
<pre class="screen" id="how-to-connect-kafka__screen25135224477">#The information in bold is specific to different Kafka instances and must be modified. Other parameters can also be added.
|
|
#The topic name is in the specific production and consumption code.
|
|
#######################
|
|
#Information about Kafka brokers. <strong id="how-to-connect-kafka__b1333422317215">ip:port</strong> are the connection addresses and ports used by the instance. The values can be obtained by referring to the "Collecting Connection Information" section. Example: bootstrap.servers=100.xxx.xxx.87:909x,100.xxx.xxx.69:909x,100.xxx.xxx.155:909x
|
|
bootstrap.servers=<strong id="how-to-connect-kafka__b13819311161719">ip1:port1,ip2:port2,ip3:port3</strong>
|
|
#Producer acknowledgement
|
|
acks=all
|
|
#Method of turning the key into bytes
|
|
key.serializer=org.apache.kafka.common.serialization.StringSerializer
|
|
#Method of turning the value into bytes
|
|
value.serializer=org.apache.kafka.common.serialization.StringSerializer
|
|
#Memory available to the producer for buffering
|
|
buffer.memory=33554432
|
|
#Number of retries
|
|
retries=0
|
|
#######################
|
|
#Comment out the following parameters if SASL access is not enabled.
|
|
#######################
|
|
#Configure the JAAS username and password. <em id="how-to-connect-kafka__i393707281"><strong id="how-to-connect-kafka__b19318014283">username</strong></em> and <em id="how-to-connect-kafka__i6944052814"><strong id="how-to-connect-kafka__b13936012287">password</strong></em> are set when you enable Kafka SASL_SSL during instance creation or when you create a SASL_SSL user.
|
|
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
|
|
username="<strong id="how-to-connect-kafka__b1381921119175">username</strong>" \
|
|
password="<strong id="how-to-connect-kafka__b381911113171">password</strong>";
|
|
#SASL mechanism
|
|
sasl.mechanism=PLAIN
|
|
#Encryption protocol. Currently, only SASL_SSL is supported.
|
|
security.protocol=SASL_SSL
|
|
#Location of ssl.truststore
|
|
ssl.truststore.location=<strong id="how-to-connect-kafka__b5820911111714">E:\\temp\\client.truststore.jks</strong>
|
|
#Password of ssl.truststore, which must be set to dms@kafka and cannot be changed.
|
|
ssl.truststore.password=dms@kafka</pre>
|
|
</p></li><li id="how-to-connect-kafka__li153111734619"><span>In the down left corner of IDEA, click <strong id="how-to-connect-kafka__b323811566572">Terminal</strong>. In terminal, run the <strong id="how-to-connect-kafka__b181421630141516">mvn test</strong> command to see how the demo project goes.</span><p><div class="fignone" id="how-to-connect-kafka__fig13579842143916"><span class="figcap"><b>Figure 4 </b>Opening terminal in IDEA</span><br><span><img id="how-to-connect-kafka__image1567249144013" src="en-us_image_0171660984.png"></span></div>
|
|
<p id="how-to-connect-kafka__p93961749163812">The following information is displayed for the producer:</p>
|
|
<pre class="screen" id="how-to-connect-kafka__screen1396174914381">-------------------------------------------------------
|
|
T E S T S
|
|
-------------------------------------------------------
|
|
Running com.dms.producer.DmsProducerTest
|
|
produce msg:The msg is 0
|
|
produce msg:The msg is 1
|
|
produce msg:The msg is 2
|
|
produce msg:The msg is 3
|
|
produce msg:The msg is 4
|
|
produce msg:The msg is 5
|
|
produce msg:The msg is 6
|
|
produce msg:The msg is 7
|
|
produce msg:The msg is 8
|
|
produce msg:The msg is 9
|
|
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 138.877 sec</pre>
|
|
<p id="how-to-connect-kafka__p1839624993812">The following information is displayed for the consumer:</p>
|
|
<pre class="screen" id="how-to-connect-kafka__screen8397134983816">-------------------------------------------------------
|
|
T E S T S
|
|
-------------------------------------------------------
|
|
Running com.dms.consumer.DmsConsumerTest
|
|
the numbers of topic:0
|
|
the numbers of topic:0
|
|
the numbers of topic:6
|
|
ConsumerRecord(topic = topic-0, partition = 2, offset = 0, CreateTime = 1557059377179, serialized key size = -1, serialized value size = 12, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = The msg is 2)
|
|
ConsumerRecord(topic = topic-0, partition = 2, offset = 1, CreateTime = 1557059377195, serialized key size = -1, serialized value size = 12, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = The msg is 5)</pre>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="kafka-java.html">Java</a></div>
|
|
</div>
|
|
</div>
|
|
|