vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide-lts/mrs_01_0376.html
Yang, Tong 3f5759eed2 MRS comp-lts 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2023-01-19 17:08:45 +00:00

31 lines
14 KiB
HTML
Raw Blame History

<a name="mrs_01_0376"></a><a name="mrs_01_0376"></a>
<h1 class="topictitle1">Managing Kafka Topics</h1>
<div id="body8662426"><div class="section" id="mrs_01_0376__en-us_topic_0000001219230523_se26ecb53194e405f958c4ce3378383a7"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_0376__en-us_topic_0000001219230523_aa3144edd1295421b83ed08a78c1bd310">You can manage Kafka topics on a cluster client based on service requirements. Management permission is required for clusters with Kerberos authentication enabled.</p>
</div>
<div class="section" id="mrs_01_0376__en-us_topic_0000001219230523_sd37d6d7758764152ac4d177b37f3127c"><h4 class="sectiontitle">Prerequisites</h4><p id="mrs_01_0376__en-us_topic_0000001219230523_a1179bbe9a620407fb0d1dcc5676fde5b">You have installed the Kafka client.</p>
</div>
<div class="section" id="mrs_01_0376__en-us_topic_0000001219230523_sf23385a46fce4ee3baaacbabc3ae5219"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_0376__en-us_topic_0000001219230523_o05c947a6157c429488f1a6bcdc1d66bd"><li id="mrs_01_0376__en-us_topic_0000001219230523_li20357144681413"><span>Access the ZooKeeper instance page.</span><p><p id="mrs_01_0376__p48001164237">log in to FusionInsight Manager. For details, see <a href="mrs_01_2124.html">Accessing FusionInsight Manager</a>. Choose <strong id="mrs_01_0376__en-us_topic_0000001219230523_b49214164532">Cluster</strong> &gt; <em id="mrs_01_0376__en-us_topic_0000001219230523_i592151655320">Name of the desired cluster</em> &gt; <strong id="mrs_01_0376__en-us_topic_0000001219230523_b29313163533">Services</strong> &gt; <strong id="mrs_01_0376__en-us_topic_0000001219230523_b5931416175311">ZooKeeper</strong> &gt; <strong id="mrs_01_0376__en-us_topic_0000001219230523_b179331619531">Instance</strong>.</p>
</p></li><li id="mrs_01_0376__en-us_topic_0000001219230523_l2f4a4c4b2bbb427a890864b6ae357ea8"><span>View the IP addresses of the ZooKeeper role instance.</span><p><p id="mrs_01_0376__en-us_topic_0000001219230523_a2c7ffa9191bf467cb17b66c27ef78e5a">Record any IP address of the ZooKeeper instance.</p>
</p></li><li id="mrs_01_0376__en-us_topic_0000001219230523_l9b2a798722044c01975eaaee4bebd7dd"><span>Prepare the client based on service requirements. Log in to the node where the client is installed.</span></li><li id="mrs_01_0376__en-us_topic_0000001219230523_l14680ebec7694c71a1e3056f9ae0f72b"><span>Run the following command to switch to the client directory, for example, /<strong id="mrs_01_0376__en-us_topic_0000001219230523_b49244712513">opt</strong><strong id="mrs_01_0376__en-us_topic_0000001219230523_b15924974258"></strong><strong id="mrs_01_0376__en-us_topic_0000001219230523_b169246782516">/client/Kafka/kafka/bin</strong>.</span><p><p id="mrs_01_0376__en-us_topic_0000001219230523_af3976b170fc4424ba173ec0c73381a13"><strong id="mrs_01_0376__en-us_topic_0000001219230523_en-us_topic_0053946969_b208077520224">cd /opt/client/Kafka/kafka/bin</strong></p>
</p></li><li id="mrs_01_0376__en-us_topic_0000001219230523_l9974bee6f19240e692c4157eeab1a308"><span>Run the following command to configure environment variables:</span><p><p id="mrs_01_0376__en-us_topic_0000001219230523_abf436e581670449ca4950d9225d58143"><strong id="mrs_01_0376__en-us_topic_0000001219230523_a64fbd5f493b8412ca7e963f5bdbff088">source /opt/client/bigdata_env</strong></p>
</p></li><li id="mrs_01_0376__en-us_topic_0000001219230523_lfa00bf6e182d432dbbe25004a768ab74"><span>Run the following command to perform user authentication (skip this step in normal mode):</span><p><p id="mrs_01_0376__en-us_topic_0000001219230523_a3161a2660a72482c80662e8cd46c2554"><strong id="mrs_01_0376__en-us_topic_0000001219230523_b72205782213">kinit</strong> <em id="mrs_01_0376__en-us_topic_0000001219230523_i15717577224">Component service user</em></p>
</p></li><li id="mrs_01_0376__en-us_topic_0000001219230523_li8151829112811"><span>Use <strong id="mrs_01_0376__en-us_topic_0000001219230523_b18634102883712">kafka-topics.sh</strong> to manage Kafka topics.</span><p><ul id="mrs_01_0376__en-us_topic_0000001219230523_ul168653182912"><li id="mrs_01_0376__en-us_topic_0000001219230523_li25388190164715">Creating a topic:<p id="mrs_01_0376__en-us_topic_0000001219230523_p111779572114"><a name="mrs_01_0376__en-us_topic_0000001219230523_li25388190164715"></a><a name="en-us_topic_0000001219230523_li25388190164715"></a>By default, partitions of a topic are distributed based on the number of partitions on the node and disk. To distribute partitions based on the disk capacity, set <strong id="mrs_01_0376__en-us_topic_0000001219230523_b166891232192018">log.partition.strategy</strong> to <span class="parmvalue" id="mrs_01_0376__en-us_topic_0000001219230523_parmvalue7695183141211"><b>capacity</b></span> for the Kafka service.</p>
<ul id="mrs_01_0376__en-us_topic_0000001219230523_ul22051777018"><li id="mrs_01_0376__en-us_topic_0000001219230523_li88278578191"><strong id="mrs_01_0376__en-us_topic_0000001219230523_b5362949132116">./kafka-topics.sh --create --topic </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i1636824911210">topic name </em><strong id="mrs_01_0376__en-us_topic_0000001219230523_b2036914920217">--partitions </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i137019491215">number of partitions occupied by the topic</em><strong id="mrs_01_0376__en-us_topic_0000001219230523_b12372749102111">--replication-factor </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i3373104992119">number of replicas of the topic</em><strong id="mrs_01_0376__en-us_topic_0000001219230523_b8374114911219">--zookeeper </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i1737564917215">IP address of any ZooKeeper node:clientPort</em><strong id="mrs_01_0376__en-us_topic_0000001219230523_b1637617497213">/kafka</strong></li><li id="mrs_01_0376__li827476152719"><strong id="mrs_01_0376__en-us_topic_0000001219230523_b227945519216">./kafka-topics.sh --create --topic </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i1328516555214">topic name </em><strong id="mrs_01_0376__en-us_topic_0000001219230523_b528635592111">--partitions </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i13287185582118">number of partitions occupied by the topic</em><strong id="mrs_01_0376__en-us_topic_0000001219230523_b52881955102118">--replication-factor </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i228935542112">number of replicas of the topic</em> <strong id="mrs_01_0376__en-us_topic_0000001219230523_b1529018551215">--</strong><strong id="mrs_01_0376__en-us_topic_0000001219230523_b182911755172112">bootstrap-server </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i829215552213">IP address of the Kafka</em><em id="mrs_01_0376__en-us_topic_0000001219230523_i122941555182119">cluster:21007</em> <strong id="mrs_01_0376__en-us_topic_0000001219230523_b12295755132118">--command-config ../config/client.properties</strong></li></ul>
</li><li id="mrs_01_0376__en-us_topic_0000001219230523_li194992217202">List of topics:<ul id="mrs_01_0376__en-us_topic_0000001219230523_ul184931324142015"><li id="mrs_01_0376__en-us_topic_0000001219230523_l66e92762768d4a3493c515f9bd00760b"><strong id="mrs_01_0376__en-us_topic_0000001219230523_a72125dcd6e6b4b94ab66472ec993e209">./kafka-topics.sh --list --zookeeper </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_a15a17c4cf72d4c7cb49b0820dba942ab">service IP address of any ZooKeeper node</em><em id="mrs_01_0376__en-us_topic_0000001219230523_i48711232204811">:clientPort</em><strong id="mrs_01_0376__en-us_topic_0000001219230523_a52beb0e66fbf49adbef304bc48fef974">/kafka</strong></li><li id="mrs_01_0376__en-us_topic_0000001219230523_li1492983018209"><strong id="mrs_01_0376__en-us_topic_0000001219230523_b11562826172111">./kafka-topics.sh --list --bootstrap-server </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i2811612102114">IP address of the Kafka</em><em id="mrs_01_0376__en-us_topic_0000001219230523_i381110128219">cluster:21007</em> <strong id="mrs_01_0376__en-us_topic_0000001219230523_b16177142215216">--command-config ../config/client.properties</strong></li></ul>
</li><li id="mrs_01_0376__en-us_topic_0000001219230523_li164951610132914">Viewing the topic:<ul id="mrs_01_0376__en-us_topic_0000001219230523_ul1772491742915"><li id="mrs_01_0376__en-us_topic_0000001219230523_li553110165299"><strong id="mrs_01_0376__en-us_topic_0000001219230523_b775644152914">./kafka-topics.sh --describe --zookeeper</strong> <em id="mrs_01_0376__en-us_topic_0000001219230523_i79811489298">service IP address of any ZooKeeper node</em><em id="mrs_01_0376__en-us_topic_0000001219230523_i8977204618503">:clientPort</em><strong id="mrs_01_0376__en-us_topic_0000001219230523_b69864802917">/kafka</strong> --<strong id="mrs_01_0376__en-us_topic_0000001219230523_b167411123152215">topic</strong> <em id="mrs_01_0376__en-us_topic_0000001219230523_i44753576296">topic name</em></li><li id="mrs_01_0376__en-us_topic_0000001219230523_li9238185910295"><strong id="mrs_01_0376__en-us_topic_0000001219230523_b1877203473019">./kafka-topics.sh --describe --bootstrap-server</strong> <em id="mrs_01_0376__en-us_topic_0000001219230523_i960761253010">IP address of the Kafka</em><em id="mrs_01_0376__en-us_topic_0000001219230523_i86074126307">cluster:21007</em> <strong id="mrs_01_0376__en-us_topic_0000001219230523_b11114115512308">--command-config ../config/client.properties</strong> <strong id="mrs_01_0376__en-us_topic_0000001219230523_b7706638103017">--topic</strong> <em id="mrs_01_0376__en-us_topic_0000001219230523_i9876141512303">topic name</em></li></ul>
</li><li id="mrs_01_0376__en-us_topic_0000001219230523_li54651651132113">Modifying a topic:<ul id="mrs_01_0376__en-us_topic_0000001219230523_ul381045111218"><li id="mrs_01_0376__en-us_topic_0000001219230523_l1147f56f270147e2bea176493ba889ca"><strong id="mrs_01_0376__en-us_topic_0000001219230523_a0e86a80de59c41d8882708f194e971db">./kafka-topics.sh --alter --topic </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_a5aaa44a9b26c49c1be9905757aa99128">topic name</em><strong id="mrs_01_0376__en-us_topic_0000001219230523_a3d2326ee01fc4bbe9ef8c5aeb7770f7b">--config </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_aa19f8377e45d4f87a238c36693ce9e2a">configuration item=</em><em id="mrs_01_0376__en-us_topic_0000001219230523_a465f39ac4d7b45e295c2d2356f7d7999">configuration value </em><strong id="mrs_01_0376__en-us_topic_0000001219230523_a9e3d3011eb884b32b9cb28562cc9c920">--zookeeper </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_a1921e229f7884d08baba2e84a20f9656">service IP address of any ZooKeeper node</em><em id="mrs_01_0376__en-us_topic_0000001219230523_i11910154135212">:clientPort</em><strong id="mrs_01_0376__en-us_topic_0000001219230523_a08bc4aa581624350922e6467eb730d1f">/kafka</strong></li></ul>
</li><li id="mrs_01_0376__en-us_topic_0000001219230523_li226143511720">Expanding partitions:<ul id="mrs_01_0376__en-us_topic_0000001219230523_ul38891245131713"><li id="mrs_01_0376__en-us_topic_0000001219230523_li14921756121920"><strong id="mrs_01_0376__en-us_topic_0000001219230523_b1854159151519">./kafka-topics.sh --alter --topic </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i457959191519">topic name</em><strong id="mrs_01_0376__en-us_topic_0000001219230523_b81712871617"> --zookeeper </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i131741083163">service IP address of any ZooKeeper node:clientPort</em><strong id="mrs_01_0376__en-us_topic_0000001219230523_b10976119171611">/kafka --command-config Kafka/kafka/config/client.properties --partitions </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i199781019111610">number of partitions after the expansion</em></li><li id="mrs_01_0376__en-us_topic_0000001219230523_li172251459179"><strong id="mrs_01_0376__en-us_topic_0000001219230523_b1254502618163">./kafka-topics.sh --alter --topic</strong> <em id="mrs_01_0376__en-us_topic_0000001219230523_i19548192691618">topic name</em><strong id="mrs_01_0376__en-us_topic_0000001219230523_b10805133421619"> --bootstrap-server</strong> <em id="mrs_01_0376__en-us_topic_0000001219230523_i14812113410166">IP address of the Kafka cluster</em><em id="mrs_01_0376__en-us_topic_0000001219230523_i1072944115165">:21007</em><strong id="mrs_01_0376__en-us_topic_0000001219230523_b1250510473167"> --command-config Kafka/kafka/config/client.properties --partitions </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i4510164781610">number of partitions after the expansion</em></li></ul>
</li><li id="mrs_01_0376__en-us_topic_0000001219230523_li42542952613">Deleting a topic<ul id="mrs_01_0376__en-us_topic_0000001219230523_ul18488133192610"><li id="mrs_01_0376__en-us_topic_0000001219230523_li05741922122618"><strong id="mrs_01_0376__en-us_topic_0000001219230523_b45741522132618">./kafka-topics.sh --delete --topic </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i857582262617">topic name </em><strong id="mrs_01_0376__en-us_topic_0000001219230523_b1557522262613">--zookeeper </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i257562219265"><em id="mrs_01_0376__en-us_topic_0000001219230523_i1857542212268"><em id="mrs_01_0376__en-us_topic_0000001219230523_i45754227267">Service IP address of any ZooKeeper node</em></em>:clientPort</em><strong id="mrs_01_0376__en-us_topic_0000001219230523_b4575172214261">/kafka</strong></li><li id="mrs_01_0376__en-us_topic_0000001219230523_li9620173418260"><strong id="mrs_01_0376__en-us_topic_0000001219230523_b1541691718273">./kafka-topics.sh --delete --topic </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i2416161714271">topic name</em><strong id="mrs_01_0376__en-us_topic_0000001219230523_b1880122342715">--bootstrap-server </strong><em id="mrs_01_0376__en-us_topic_0000001219230523_i088042332720">IP address of the Kafka cluster</em><em id="mrs_01_0376__en-us_topic_0000001219230523_i108801723112719">:21007</em> <strong id="mrs_01_0376__en-us_topic_0000001219230523_b888072311272">--command-config ../config/client.properties</strong></li></ul>
</li></ul>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0375.html">Using Kafka</a></div>
</div>
</div>
View Git Blame Copy Permalink