vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dms/umn/Kafka-specification.html
Chen, Junjie ca9677b121 DMS UMN 20230201 Version 
Reviewed-by: Antonova, Ekaterina <ekantono@noreply.gitea.eco.tsi-dev.otc-service.com>
Co-authored-by: Chen, Junjie <chenjunjie@huawei.com>
Co-committed-by: Chen, Junjie <chenjunjie@huawei.com>
2024-07-22 13:20:04 +00:00

342 lines
33 KiB
HTML
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<a name="Kafka-specification"></a><a name="Kafka-specification"></a>
<h1 class="topictitle1">Specifications</h1>
<div id="body1552013649930"><div class="section" id="Kafka-specification__section21958537463"><h4 class="sectiontitle">Kafka Instance Specifications</h4><p id="Kafka-specification__p712114892412">Kafka instances are compatible with open-source Kafka v1.1.0, v2.3.0, v2.7, and v3.x. Kafka instances are classified into cluster and single-node types. A cluster instance consists of three or more brokers and a single-node one has one broker.</p>
<p id="Kafka-specification__p85014108284">Kafka instances are classified based on instance ECS flavors as follows:</p>
<ul id="Kafka-specification__ul7311715132811"><li id="Kafka-specification__li41011033132818">Cluster<ul id="Kafka-specification__ul1599233102815"><li id="Kafka-specification__li7291144032811">kafka.2u4g.cluster.small</li><li id="Kafka-specification__li148218445289">kafka.2u4g.cluster</li><li id="Kafka-specification__li1527619476287">kafka.4u8g.cluster</li><li id="Kafka-specification__li1776394818284">kafka.8u16g.cluster</li><li id="Kafka-specification__li1759485122816">kafka.12u24g.cluster</li><li id="Kafka-specification__li782165610288">kafka.16u32g.cluster</li><li id="Kafka-specification__li7989155732413">kafka.2u4g.cluster.beta</li><li id="Kafka-specification__li5924686255">kafka.4u8g.cluster.beta</li></ul>
</li><li id="Kafka-specification__li3570183652911">Single-node<ul id="Kafka-specification__ul1256843612919"><li id="Kafka-specification__li122624142912">kafka.2u4g.single.small</li><li id="Kafka-specification__li7568183692919">kafka.2u4g.single</li></ul>
</li></ul>
<div class="note" id="Kafka-specification__note04898151210"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="Kafka-specification__p192525251121">For Kafka instances, the number of transactions per second (TPS) is the maximum number of messages that can be written per second. In the following table, transactions per second (TPS) are calculated assuming that the size of a message is 1 KB. The test scenario is private access in plaintext. The disk type is ultra-high I/O. </p>
<p id="Kafka-specification__p1910002214468">Cluster Kafka instances support v1.1.0, v2.3.0, v2.7, and v3.x. Single-node Kafka instances support v2.7.</p>
</div></div>
<div class="tablenoborder"><a name="Kafka-specification__table152020206204"></a><a name="table152020206204"></a><table cellpadding="4" cellspacing="0" summary="" id="Kafka-specification__table152020206204" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Kafka instance specifications (v1.1.0/v2.3.0/v2.7 cluster instances)</caption><thead align="left"><tr id="Kafka-specification__row652112022012"><th align="left" class="cellrowborder" valign="top" width="13.598640135986404%" id="mcps1.3.1.6.2.9.1.1"><p id="Kafka-specification__p15704194442115">Flavor</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="8.35916408359164%" id="mcps1.3.1.6.2.9.1.2"><p id="Kafka-specification__p127043444212">Brokers</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="10.338966103389662%" id="mcps1.3.1.6.2.9.1.3"><p id="Kafka-specification__p7704144102118">Maximum TPS per Broker</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="9.829017098290171%" id="mcps1.3.1.6.2.9.1.4"><p id="Kafka-specification__p470434420218">Maximum Partitions per Broker</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="10.108989101089891%" id="mcps1.3.1.6.2.9.1.5"><p id="Kafka-specification__p370414442112">Recommended Consumer Groups per Broker</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="13.20867913208679%" id="mcps1.3.1.6.2.9.1.6"><p id="Kafka-specification__p157043445213">Maximum Client Connections per Broker</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="18.388161183881614%" id="mcps1.3.1.6.2.9.1.7"><p id="Kafka-specification__p147040446217">Storage Space</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="16.168383161683835%" id="mcps1.3.1.6.2.9.1.8"><p id="Kafka-specification__p9443133815139">Traffic per Broker (MB/s)</p>
</th>
</tr>
</thead>
<tbody><tr id="Kafka-specification__row1639511615016"><td class="cellrowborder" valign="top" width="13.598640135986404%" headers="mcps1.3.1.6.2.9.1.1 "><p id="Kafka-specification__p639518616500">kafka.2u4g.cluster.small</p>
</td>
<td class="cellrowborder" valign="top" width="8.35916408359164%" headers="mcps1.3.1.6.2.9.1.2 "><p id="Kafka-specification__p939606195017">3–30</p>
</td>
<td class="cellrowborder" valign="top" width="10.338966103389662%" headers="mcps1.3.1.6.2.9.1.3 "><p id="Kafka-specification__p1139696155014">20,000</p>
</td>
<td class="cellrowborder" valign="top" width="9.829017098290171%" headers="mcps1.3.1.6.2.9.1.4 "><p id="Kafka-specification__p73962610509">100</p>
</td>
<td class="cellrowborder" valign="top" width="10.108989101089891%" headers="mcps1.3.1.6.2.9.1.5 "><p id="Kafka-specification__p1639610685011">15</p>
</td>
<td class="cellrowborder" valign="top" width="13.20867913208679%" headers="mcps1.3.1.6.2.9.1.6 "><p id="Kafka-specification__p839619616507">2000</p>
</td>
<td class="cellrowborder" valign="top" width="18.388161183881614%" headers="mcps1.3.1.6.2.9.1.7 "><p id="Kafka-specification__p0396167508">300 GB–300,000 GB</p>
</td>
<td class="cellrowborder" valign="top" width="16.168383161683835%" headers="mcps1.3.1.6.2.9.1.8 "><p id="Kafka-specification__p1044393881310">40</p>
</td>
</tr>
<tr id="Kafka-specification__row185211420182019"><td class="cellrowborder" valign="top" width="13.598640135986404%" headers="mcps1.3.1.6.2.9.1.1 "><p id="Kafka-specification__p1679155462118">kafka.2u4g.cluster</p>
</td>
<td class="cellrowborder" valign="top" width="8.35916408359164%" headers="mcps1.3.1.6.2.9.1.2 "><p id="Kafka-specification__p2791554142112">3–30</p>
</td>
<td class="cellrowborder" valign="top" width="10.338966103389662%" headers="mcps1.3.1.6.2.9.1.3 "><p id="Kafka-specification__p079175422116">30,000</p>
</td>
<td class="cellrowborder" valign="top" width="9.829017098290171%" headers="mcps1.3.1.6.2.9.1.4 "><p id="Kafka-specification__p15791854192115">250</p>
</td>
<td class="cellrowborder" valign="top" width="10.108989101089891%" headers="mcps1.3.1.6.2.9.1.5 "><p id="Kafka-specification__p18793545210">20</p>
</td>
<td class="cellrowborder" valign="top" width="13.20867913208679%" headers="mcps1.3.1.6.2.9.1.6 "><p id="Kafka-specification__p47975418211">2000</p>
</td>
<td class="cellrowborder" valign="top" width="18.388161183881614%" headers="mcps1.3.1.6.2.9.1.7 "><p id="Kafka-specification__p1679115419210">300 GB–300,000 GB</p>
</td>
<td class="cellrowborder" valign="top" width="16.168383161683835%" headers="mcps1.3.1.6.2.9.1.8 "><p id="Kafka-specification__p1944323871316">100</p>
</td>
</tr>
<tr id="Kafka-specification__row1652272020202"><td class="cellrowborder" valign="top" width="13.598640135986404%" headers="mcps1.3.1.6.2.9.1.1 "><p id="Kafka-specification__p1179554132119">kafka.4u8g.cluster</p>
</td>
<td class="cellrowborder" valign="top" width="8.35916408359164%" headers="mcps1.3.1.6.2.9.1.2 "><p id="Kafka-specification__p177975413216">3–30</p>
</td>
<td class="cellrowborder" valign="top" width="10.338966103389662%" headers="mcps1.3.1.6.2.9.1.3 "><p id="Kafka-specification__p20792543217">100,000</p>
</td>
<td class="cellrowborder" valign="top" width="9.829017098290171%" headers="mcps1.3.1.6.2.9.1.4 "><p id="Kafka-specification__p1079654192117">500</p>
</td>
<td class="cellrowborder" valign="top" width="10.108989101089891%" headers="mcps1.3.1.6.2.9.1.5 "><p id="Kafka-specification__p1979155412118">100</p>
</td>
<td class="cellrowborder" valign="top" width="13.20867913208679%" headers="mcps1.3.1.6.2.9.1.6 "><p id="Kafka-specification__p157912546219">4000</p>
</td>
<td class="cellrowborder" valign="top" width="18.388161183881614%" headers="mcps1.3.1.6.2.9.1.7 "><p id="Kafka-specification__p87985482111">300 GB–600,000 GB</p>
</td>
<td class="cellrowborder" valign="top" width="16.168383161683835%" headers="mcps1.3.1.6.2.9.1.8 "><p id="Kafka-specification__p18443163841316">200</p>
</td>
</tr>
<tr id="Kafka-specification__row14522820112013"><td class="cellrowborder" valign="top" width="13.598640135986404%" headers="mcps1.3.1.6.2.9.1.1 "><p id="Kafka-specification__p1879135416219">kafka.8u16g.cluster</p>
</td>
<td class="cellrowborder" valign="top" width="8.35916408359164%" headers="mcps1.3.1.6.2.9.1.2 "><p id="Kafka-specification__p160542123317">3–50</p>
</td>
<td class="cellrowborder" valign="top" width="10.338966103389662%" headers="mcps1.3.1.6.2.9.1.3 "><p id="Kafka-specification__p479155452113">150,000</p>
</td>
<td class="cellrowborder" valign="top" width="9.829017098290171%" headers="mcps1.3.1.6.2.9.1.4 "><p id="Kafka-specification__p0799545213">1000</p>
</td>
<td class="cellrowborder" valign="top" width="10.108989101089891%" headers="mcps1.3.1.6.2.9.1.5 "><p id="Kafka-specification__p1179254102115">150</p>
</td>
<td class="cellrowborder" valign="top" width="13.20867913208679%" headers="mcps1.3.1.6.2.9.1.6 "><p id="Kafka-specification__p10798546218">4000</p>
</td>
<td class="cellrowborder" valign="top" width="18.388161183881614%" headers="mcps1.3.1.6.2.9.1.7 "><p id="Kafka-specification__p17715354123319">300 GB–1,500,000 GB</p>
</td>
<td class="cellrowborder" valign="top" width="16.168383161683835%" headers="mcps1.3.1.6.2.9.1.8 "><p id="Kafka-specification__p159621537201410">250</p>
</td>
</tr>
<tr id="Kafka-specification__row13522112012013"><td class="cellrowborder" valign="top" width="13.598640135986404%" headers="mcps1.3.1.6.2.9.1.1 "><p id="Kafka-specification__p1579205452110">kafka.12u24g.cluster</p>
</td>
<td class="cellrowborder" valign="top" width="8.35916408359164%" headers="mcps1.3.1.6.2.9.1.2 "><p id="Kafka-specification__p2311428203514">3–50</p>
</td>
<td class="cellrowborder" valign="top" width="10.338966103389662%" headers="mcps1.3.1.6.2.9.1.3 "><p id="Kafka-specification__p88035420212">200,000</p>
</td>
<td class="cellrowborder" valign="top" width="9.829017098290171%" headers="mcps1.3.1.6.2.9.1.4 "><p id="Kafka-specification__p48014544218">1500</p>
</td>
<td class="cellrowborder" valign="top" width="10.108989101089891%" headers="mcps1.3.1.6.2.9.1.5 "><p id="Kafka-specification__p11801254172112">200</p>
</td>
<td class="cellrowborder" valign="top" width="13.20867913208679%" headers="mcps1.3.1.6.2.9.1.6 "><p id="Kafka-specification__p1880135417219">4000</p>
</td>
<td class="cellrowborder" valign="top" width="18.388161183881614%" headers="mcps1.3.1.6.2.9.1.7 "><p id="Kafka-specification__p473011816366">300 GB–1,500,000 GB</p>
</td>
<td class="cellrowborder" valign="top" width="16.168383161683835%" headers="mcps1.3.1.6.2.9.1.8 "><p id="Kafka-specification__p0934164510144">375</p>
</td>
</tr>
<tr id="Kafka-specification__row1852319202204"><td class="cellrowborder" valign="top" width="13.598640135986404%" headers="mcps1.3.1.6.2.9.1.1 "><p id="Kafka-specification__p48025416215">kafka.16u32g.cluster</p>
</td>
<td class="cellrowborder" valign="top" width="8.35916408359164%" headers="mcps1.3.1.6.2.9.1.2 "><p id="Kafka-specification__p390133113356">3–50</p>
</td>
<td class="cellrowborder" valign="top" width="10.338966103389662%" headers="mcps1.3.1.6.2.9.1.3 "><p id="Kafka-specification__p1280195462119">250,000</p>
</td>
<td class="cellrowborder" valign="top" width="9.829017098290171%" headers="mcps1.3.1.6.2.9.1.4 "><p id="Kafka-specification__p18055452117">2000</p>
</td>
<td class="cellrowborder" valign="top" width="10.108989101089891%" headers="mcps1.3.1.6.2.9.1.5 "><p id="Kafka-specification__p68065410212">200</p>
</td>
<td class="cellrowborder" valign="top" width="13.20867913208679%" headers="mcps1.3.1.6.2.9.1.6 "><p id="Kafka-specification__p68055413219">4000</p>
</td>
<td class="cellrowborder" valign="top" width="18.388161183881614%" headers="mcps1.3.1.6.2.9.1.7 "><p id="Kafka-specification__p177571811113616">300 GB–1,500,000 GB</p>
</td>
<td class="cellrowborder" valign="top" width="16.168383161683835%" headers="mcps1.3.1.6.2.9.1.8 "><p id="Kafka-specification__p11359505144">500</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="Kafka-specification__table189471434374" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Kafka instance specifications (v3.x cluster instances)</caption><thead align="left"><tr id="Kafka-specification__row1594719353718"><th align="left" class="cellrowborder" valign="top" width="13.598640135986404%" id="mcps1.3.1.7.2.9.1.1"><p id="Kafka-specification__p8947123133715">Flavor</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="8.349165083491652%" id="mcps1.3.1.7.2.9.1.2"><p id="Kafka-specification__p394712311371">Brokers</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="10.34896510348965%" id="mcps1.3.1.7.2.9.1.3"><p id="Kafka-specification__p194716333716">TPS per Broker</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="9.829017098290171%" id="mcps1.3.1.7.2.9.1.4"><p id="Kafka-specification__p199476383716">Maximum Partitions per Broker</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="10.108989101089891%" id="mcps1.3.1.7.2.9.1.5"><p id="Kafka-specification__p12947833373">Recommended Consumer Groups per Broker</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="13.20867913208679%" id="mcps1.3.1.7.2.9.1.6"><p id="Kafka-specification__p1194718353716">Maximum Client Connections per Broker</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="18.388161183881614%" id="mcps1.3.1.7.2.9.1.7"><p id="Kafka-specification__p494733183712">Storage Space</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="16.168383161683835%" id="mcps1.3.1.7.2.9.1.8"><p id="Kafka-specification__p594719313718">Traffic per Broker (MB/s)</p>
</th>
</tr>
</thead>
<tbody><tr id="Kafka-specification__row189471310372"><td class="cellrowborder" valign="top" width="13.598640135986404%" headers="mcps1.3.1.7.2.9.1.1 "><p id="Kafka-specification__p1052313653715">kafka.2u4g.cluster.beta</p>
</td>
<td class="cellrowborder" valign="top" width="8.349165083491652%" headers="mcps1.3.1.7.2.9.1.2 "><p id="Kafka-specification__p2523143615377">3</p>
</td>
<td class="cellrowborder" valign="top" width="10.34896510348965%" headers="mcps1.3.1.7.2.9.1.3 "><p id="Kafka-specification__p7523153663716">30,000</p>
</td>
<td class="cellrowborder" valign="top" width="9.829017098290171%" headers="mcps1.3.1.7.2.9.1.4 "><p id="Kafka-specification__p95234369376">250</p>
</td>
<td class="cellrowborder" valign="top" width="10.108989101089891%" headers="mcps1.3.1.7.2.9.1.5 "><p id="Kafka-specification__p2052310366378">20</p>
</td>
<td class="cellrowborder" valign="top" width="13.20867913208679%" headers="mcps1.3.1.7.2.9.1.6 "><p id="Kafka-specification__p6523203603712">2000</p>
</td>
<td class="cellrowborder" valign="top" width="18.388161183881614%" headers="mcps1.3.1.7.2.9.1.7 "><p id="Kafka-specification__p8523636193712">300 GB</p>
</td>
<td class="cellrowborder" valign="top" width="16.168383161683835%" headers="mcps1.3.1.7.2.9.1.8 "><p id="Kafka-specification__p9523163616376">100</p>
</td>
</tr>
<tr id="Kafka-specification__row1594723123713"><td class="cellrowborder" valign="top" width="13.598640135986404%" headers="mcps1.3.1.7.2.9.1.1 "><p id="Kafka-specification__p7523193663712">kafka.4u8g.cluster.beta</p>
</td>
<td class="cellrowborder" valign="top" width="8.349165083491652%" headers="mcps1.3.1.7.2.9.1.2 "><p id="Kafka-specification__p115231136103711">3</p>
</td>
<td class="cellrowborder" valign="top" width="10.34896510348965%" headers="mcps1.3.1.7.2.9.1.3 "><p id="Kafka-specification__p1152363693717">100,000</p>
</td>
<td class="cellrowborder" valign="top" width="9.829017098290171%" headers="mcps1.3.1.7.2.9.1.4 "><p id="Kafka-specification__p1052316368375">500</p>
</td>
<td class="cellrowborder" valign="top" width="10.108989101089891%" headers="mcps1.3.1.7.2.9.1.5 "><p id="Kafka-specification__p4523163618373">100</p>
</td>
<td class="cellrowborder" valign="top" width="13.20867913208679%" headers="mcps1.3.1.7.2.9.1.6 "><p id="Kafka-specification__p165231836143715">4000</p>
</td>
<td class="cellrowborder" valign="top" width="18.388161183881614%" headers="mcps1.3.1.7.2.9.1.7 "><p id="Kafka-specification__p145231936193717">300 GB</p>
</td>
<td class="cellrowborder" valign="top" width="16.168383161683835%" headers="mcps1.3.1.7.2.9.1.8 "><p id="Kafka-specification__p75239365375">200</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="tablenoborder"><a name="Kafka-specification__table960115533719"></a><a name="table960115533719"></a><table cellpadding="4" cellspacing="0" summary="" id="Kafka-specification__table960115533719" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Kafka instance specifications (single-node)</caption><thead align="left"><tr id="Kafka-specification__row16601165533710"><th align="left" class="cellrowborder" valign="top" width="13.598640135986404%" id="mcps1.3.1.8.2.9.1.1"><p id="Kafka-specification__p5601155519372">Flavor</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="8.35916408359164%" id="mcps1.3.1.8.2.9.1.2"><p id="Kafka-specification__p060116553372">Brokers</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="10.338966103389662%" id="mcps1.3.1.8.2.9.1.3"><p id="Kafka-specification__p1601255173719">TPS per Broker</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="9.829017098290171%" id="mcps1.3.1.8.2.9.1.4"><p id="Kafka-specification__p66013555378">Maximum Partitions per Broker</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="10.108989101089891%" id="mcps1.3.1.8.2.9.1.5"><p id="Kafka-specification__p136011655103714">Recommended Consumer Groups per Broker</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="13.20867913208679%" id="mcps1.3.1.8.2.9.1.6"><p id="Kafka-specification__p156011955173718">Maximum Client Connections per Broker</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="18.388161183881614%" id="mcps1.3.1.8.2.9.1.7"><p id="Kafka-specification__p19601155523712">Storage Space</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="16.168383161683835%" id="mcps1.3.1.8.2.9.1.8"><p id="Kafka-specification__p8601165510378">Traffic per Broker (MB/s)</p>
</th>
</tr>
</thead>
<tbody><tr id="Kafka-specification__row86011855173717"><td class="cellrowborder" valign="top" width="13.598640135986404%" headers="mcps1.3.1.8.2.9.1.1 "><p id="Kafka-specification__p0601115515375">kafka.2u4g.single.small</p>
</td>
<td class="cellrowborder" valign="top" width="8.35916408359164%" headers="mcps1.3.1.8.2.9.1.2 "><p id="Kafka-specification__p9601125553712">1</p>
</td>
<td class="cellrowborder" valign="top" width="10.338966103389662%" headers="mcps1.3.1.8.2.9.1.3 "><p id="Kafka-specification__p17601185512375">20,000</p>
</td>
<td class="cellrowborder" valign="top" width="9.829017098290171%" headers="mcps1.3.1.8.2.9.1.4 "><p id="Kafka-specification__p156019550379">100</p>
</td>
<td class="cellrowborder" valign="top" width="10.108989101089891%" headers="mcps1.3.1.8.2.9.1.5 "><p id="Kafka-specification__p18601125512374">15</p>
</td>
<td class="cellrowborder" valign="top" width="13.20867913208679%" headers="mcps1.3.1.8.2.9.1.6 "><p id="Kafka-specification__p1060185583712">2000</p>
</td>
<td class="cellrowborder" valign="top" width="18.388161183881614%" headers="mcps1.3.1.8.2.9.1.7 "><p id="Kafka-specification__p96014557379">100 GB–10,000 GB</p>
</td>
<td class="cellrowborder" valign="top" width="16.168383161683835%" headers="mcps1.3.1.8.2.9.1.8 "><p id="Kafka-specification__p1960105519373">40</p>
</td>
</tr>
<tr id="Kafka-specification__row2060110559374"><td class="cellrowborder" valign="top" width="13.598640135986404%" headers="mcps1.3.1.8.2.9.1.1 "><p id="Kafka-specification__p960135553715">kafka.2u4g.single</p>
</td>
<td class="cellrowborder" valign="top" width="8.35916408359164%" headers="mcps1.3.1.8.2.9.1.2 "><p id="Kafka-specification__p86011855163710">1</p>
</td>
<td class="cellrowborder" valign="top" width="10.338966103389662%" headers="mcps1.3.1.8.2.9.1.3 "><p id="Kafka-specification__p116018555372">30,000</p>
</td>
<td class="cellrowborder" valign="top" width="9.829017098290171%" headers="mcps1.3.1.8.2.9.1.4 "><p id="Kafka-specification__p1960155583710">250</p>
</td>
<td class="cellrowborder" valign="top" width="10.108989101089891%" headers="mcps1.3.1.8.2.9.1.5 "><p id="Kafka-specification__p19601155510371">20</p>
</td>
<td class="cellrowborder" valign="top" width="13.20867913208679%" headers="mcps1.3.1.8.2.9.1.6 "><p id="Kafka-specification__p060111556376">2000</p>
</td>
<td class="cellrowborder" valign="top" width="18.388161183881614%" headers="mcps1.3.1.8.2.9.1.7 "><p id="Kafka-specification__p17601195520371">100 GB–10,000 GB</p>
</td>
<td class="cellrowborder" valign="top" width="16.168383161683835%" headers="mcps1.3.1.8.2.9.1.8 "><p id="Kafka-specification__p26011555193717">100</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="Kafka-specification__section1861114217413"><h4 class="sectiontitle">Instance Specifications and Network Bandwidth</h4><p id="Kafka-specification__p11306155616">The network bandwidth of a Kafka instance consists of the following:</p>
<ol id="Kafka-specification__ol1140217251682"><li id="Kafka-specification__li1240211251587">Network bandwidth used by the instance brokers</li><li id="Kafka-specification__li2029172717817">Bandwidth of the disk used by the instance brokers. For details, see <a href="https://docs.otc.t-systems.com/en-us/usermanual/evs/en-us_topic_0014580744.html" target="_blank" rel="noopener noreferrer">Disk Types and Performance</a>.</li></ol>
<p id="Kafka-specification__p12812101815152">Note:</p>
<ul id="Kafka-specification__ul11506101317302"><li id="Kafka-specification__li250614131300">By default, Kafka tests are performed in the tail read scenario (that is, only the latest production data is consumed) instead of the cold read scenario (that is, historical data is consumed from the beginning).</li><li id="Kafka-specification__li202222295304">The bandwidth of an instance with an old flavor (such as 100 MB/s) is the total network bandwidth of the instance's all brokers.</li></ul>
<p id="Kafka-specification__p1699072112156"><strong id="Kafka-specification__b175656261161">Traffic calculation of instances with new flavors (such as kafka.2u4g.cluster) is described as follows:</strong></p>
<ul id="Kafka-specification__ul3360764317"><li id="Kafka-specification__li113617114317">The read/write ratio is 1:1.</li><li id="Kafka-specification__li11274433124312">The default number of topic replicas is 3.</li><li id="Kafka-specification__li5800153912439">Total network traffic = Traffic per broker x Broker quantity</li><li id="Kafka-specification__li1461613448436">Total instance traffic = Service traffic + Data replication traffic between brokers</li></ul>
<p id="Kafka-specification__p2514117144818">Assume that the current flavor is kafka.2u4g.cluster, the traffic per broker is 100 MB/s, and the number of brokers is 3. What are the total network traffic, maximum read traffic, and maximum write traffic of the instance?</p>
<ol id="Kafka-specification__ol17602315175610"><li id="Kafka-specification__li0602151535612">Total network traffic = Traffic per broker x Broker quantity = 100 MB/s x 3 = 300 MB/s</li><li id="Kafka-specification__li3703131713560">Maximum read traffic = Total instance network traffic/Default number of replicas/2 = 300 MB/s/3/2= 50 MB/s</li><li id="Kafka-specification__li20391521155612">Maximum write traffic = Total instance network traffic/Default number of replicas/2 = 300 MB/s/3/2 = 50 MB/s</li></ol>
</div>
<div class="section" id="Kafka-specification__section91861161300"><h4 class="sectiontitle">Mapping Between Old and New Flavors</h4><p id="Kafka-specification__p17107192317216"><a href="#Kafka-specification__table16919145814010">Table 4</a> compares the old and new Kafka instance flavors.</p>
<div class="tablenoborder"><a name="Kafka-specification__table16919145814010"></a><a name="table16919145814010"></a><table cellpadding="4" cellspacing="0" summary="" id="Kafka-specification__table16919145814010" frame="border" border="1" rules="all"><caption><b>Table 4 </b>Mapping between old and new Kafka instance flavors</caption><thead align="left"><tr id="Kafka-specification__row16919158103"><th align="left" class="cellrowborder" colspan="2" valign="top" id="mcps1.3.3.3.2.5.1.1"><p id="Kafka-specification__p1891917589019">Old Flavor</p>
</th>
<th align="left" class="cellrowborder" colspan="2" valign="top" id="mcps1.3.3.3.2.5.1.2"><p id="Kafka-specification__p1663594119244">New Flavor</p>
</th>
</tr>
</thead>
<tbody><tr id="Kafka-specification__row7478122224"><td class="cellrowborder" valign="top" width="19.939999999999998%" headers="mcps1.3.3.3.2.5.1.1 "><p id="Kafka-specification__p4478421211">Flavor</p>
</td>
<td class="cellrowborder" valign="top" width="25.21%" headers="mcps1.3.3.3.2.5.1.1 "><p id="Kafka-specification__p188771424947">Total Instance Network Traffic</p>
</td>
<td class="cellrowborder" valign="top" width="32.33%" headers="mcps1.3.3.3.2.5.1.2 "><p id="Kafka-specification__p143225342034">Flavor</p>
</td>
<td class="cellrowborder" valign="top" width="22.52%" headers="mcps1.3.3.3.2.5.1.2 "><p id="Kafka-specification__p1239919277419">Total Instance Network Traffic</p>
</td>
</tr>
<tr id="Kafka-specification__row189191758209"><td class="cellrowborder" valign="top" width="19.939999999999998%" headers="mcps1.3.3.3.2.5.1.1 "><p id="Kafka-specification__p1358442212423">100 MB/s</p>
</td>
<td class="cellrowborder" valign="top" width="25.21%" headers="mcps1.3.3.3.2.5.1.1 "><p id="Kafka-specification__p9362199263">100 MB/s</p>
</td>
<td class="cellrowborder" valign="top" width="32.33%" headers="mcps1.3.3.3.2.5.1.2 "><p id="Kafka-specification__p145488418279">kafka.2u4g.cluster.small * 3</p>
</td>
<td class="cellrowborder" valign="top" width="22.52%" headers="mcps1.3.3.3.2.5.1.2 "><p id="Kafka-specification__p9505729662">120 MB/s</p>
</td>
</tr>
<tr id="Kafka-specification__row149198581803"><td class="cellrowborder" valign="top" width="19.939999999999998%" headers="mcps1.3.3.3.2.5.1.1 "><p id="Kafka-specification__p103410336428">300 MB/s</p>
</td>
<td class="cellrowborder" valign="top" width="25.21%" headers="mcps1.3.3.3.2.5.1.1 "><p id="Kafka-specification__p113621091616">300 MB/s</p>
</td>
<td class="cellrowborder" valign="top" width="32.33%" headers="mcps1.3.3.3.2.5.1.2 "><p id="Kafka-specification__p22222012102816">kafka.2u4g.cluster * 3</p>
</td>
<td class="cellrowborder" valign="top" width="22.52%" headers="mcps1.3.3.3.2.5.1.2 "><p id="Kafka-specification__p150515298618">300 MB/s</p>
</td>
</tr>
<tr id="Kafka-specification__row59196581603"><td class="cellrowborder" valign="top" width="19.939999999999998%" headers="mcps1.3.3.3.2.5.1.1 "><p id="Kafka-specification__p859920410423">600 MB/s</p>
</td>
<td class="cellrowborder" valign="top" width="25.21%" headers="mcps1.3.3.3.2.5.1.1 "><p id="Kafka-specification__p6362591864">600 MB/s</p>
</td>
<td class="cellrowborder" valign="top" width="32.33%" headers="mcps1.3.3.3.2.5.1.2 "><p id="Kafka-specification__p15635134152410">kafka.4u8g.cluster * 3</p>
</td>
<td class="cellrowborder" valign="top" width="22.52%" headers="mcps1.3.3.3.2.5.1.2 "><p id="Kafka-specification__p750512296619">600 MB/s</p>
</td>
</tr>
<tr id="Kafka-specification__row9919758808"><td class="cellrowborder" valign="top" width="19.939999999999998%" headers="mcps1.3.3.3.2.5.1.1 "><p id="Kafka-specification__p1169134954214">1200 MB/s</p>
</td>
<td class="cellrowborder" valign="top" width="25.21%" headers="mcps1.3.3.3.2.5.1.1 "><p id="Kafka-specification__p2363119662">1200 MB/s</p>
</td>
<td class="cellrowborder" valign="top" width="32.33%" headers="mcps1.3.3.3.2.5.1.2 "><p id="Kafka-specification__p663513412244">kafka.4u8g.cluster * 6</p>
</td>
<td class="cellrowborder" valign="top" width="22.52%" headers="mcps1.3.3.3.2.5.1.2 "><p id="Kafka-specification__p15051629769">1250 MB/s</p>
</td>
</tr>
</tbody>
</table>
</div>
<p id="Kafka-specification__p16328451173">Instances with new flavors have the following features:</p>
<ul id="Kafka-specification__ul864017121587"><li id="Kafka-specification__li931815383">Better performance and cost effectiveness: They use exclusive resources (except for kafka.2u4g.cluster.small). By contrast, old flavors use non-exclusive resources. If the load is heavy, resources conflicts will occur.</li><li id="Kafka-specification__li108706183818">Latest functions, for example, reassigning partitions and changing the SSL setting.</li><li id="Kafka-specification__li20880165018172">Flexible flavor changes: For example, you can increase the broker flavor.</li><li id="Kafka-specification__li486216144252">Flexible disk capacity: Only related to the broker quantity, and not to the flavor.</li><li id="Kafka-specification__li6893173415285">More specification options: A wider range of combinations of broker flavor (over 10,000 MB/s) and quantity are available.</li></ul>
</div>
<div class="section" id="Kafka-specification__section19307643183916"><h4 class="sectiontitle">Flavor Selection</h4><ul id="Kafka-specification__ul15944162914402"><li id="Kafka-specification__li12786173310511">kafka.2u4g.cluster.small with 3 brokers<p id="Kafka-specification__p1077015215514"><a name="Kafka-specification__li12786173310511"></a><a name="li12786173310511"></a>Recommended for up to 6000 client connections, 45 consumer groups, and 60,000 TPS</p>
</li><li id="Kafka-specification__li5945162916401">kafka.2u4g.cluster with 3 brokers<p id="Kafka-specification__p1713214954119"><a name="Kafka-specification__li5945162916401"></a><a name="li5945162916401"></a>Recommended for up to 6000 client connections, 60 consumer groups, and 90,000 TPS</p>
</li><li id="Kafka-specification__li12476204604011">kafka.4u8g.cluster with 3 brokers<p id="Kafka-specification__p146971827144217"><a name="Kafka-specification__li12476204604011"></a><a name="li12476204604011"></a>Recommended for up to 12,000 client connections, 300 consumer groups, and 300,000 TPS</p>
</li><li id="Kafka-specification__li1282216512414">kafka.8u16g.cluster with 3 brokers<p id="Kafka-specification__p1657704564316"><a name="Kafka-specification__li1282216512414"></a><a name="li1282216512414"></a>Recommended for up to 12,000 client connections, 450 consumer groups, and 450,000 TPS</p>
</li><li id="Kafka-specification__li149931417154118">kafka.12u24g.cluster with 3 brokers<p id="Kafka-specification__p61831089443"><a name="Kafka-specification__li149931417154118"></a><a name="li149931417154118"></a>Recommended for up to 12,000 client connections, 600 consumer groups, and 600,000 TPS</p>
</li><li id="Kafka-specification__li1851642944118">kafka.16u32g.cluster with 3 brokers<p id="Kafka-specification__p1884611288445"><a name="Kafka-specification__li1851642944118"></a><a name="li1851642944118"></a>Recommended for up to 12,000 client connections, 600 consumer groups, and 750,000 TPS</p>
</li></ul>
</div>
<div class="section" id="Kafka-specification__section2792821194417"><h4 class="sectiontitle">Storage Space Selection</h4><p id="Kafka-specification__p4545191495211">Kafka instances can store messages in multiple replicas. The storage space is consumed by message replicas, logs, and metadata. When creating an instance, specify its storage space based on the expected service message size, the number of replicas, and reserved disk space. Each Kafka broker reserves 33 GB disk space for storing logs and metadata.</p>
<p id="Kafka-specification__p22993455134">For example, if the expected service message size is 100 GB, the number of replicas is 2, and the number of brokers is 3, the disk size should be at least 299 GB (100 GB x 2 + 33 GB x 3).</p>
<p id="Kafka-specification__p1750795212148">The storage space can be expanded as your service grows.</p>
</div>
<div class="section" id="Kafka-specification__section8847182451410"><h4 class="sectiontitle">Topic Quantity</h4><p id="Kafka-specification__p347717213107">There are limits on the topic quantity and the aggregate number of partitions in the topics. When the partition quantity limit is reached, you can no longer create topics.</p>
<p id="Kafka-specification__p9441156121114">The number of topics is related to the maximum number of partitions allowed (see <a href="#Kafka-specification__fig1391435820518">Figure 1</a>) and the specified number of partitions in each topic (see <a href="#Kafka-specification__table152020206204">Table 1</a> and <a href="#Kafka-specification__table960115533719">Table 3</a>).</p>
<div class="fignone" id="Kafka-specification__fig1391435820518"><a name="Kafka-specification__fig1391435820518"></a><a name="fig1391435820518"></a><span class="figcap"><b>Figure 1 </b>Setting the number of partitions</span><br><span><img id="Kafka-specification__image189156581054" src="en-us_image_0000001755301270.png"></span></div>
<p id="Kafka-specification__p0126122818389"><strong id="Kafka-specification__b91142061988">The maximum number of partitions allowed for an instance with kafka.2u4g.cluster and 3 brokers is 750.</strong></p>
<ul id="Kafka-specification__ul182747568414"><li id="Kafka-specification__li7552142104219">If the number of partitions of each topic in the instance is 3, the maximum number of topics is 750/3 = 250.</li><li id="Kafka-specification__li1155222114219">If the number of partitions of each topic in the instance is 1, the maximum number of topics is 750/1 = 750.</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="kafka-ug-0723001.html">Service Overview</a></div>
</div>
</div>
View Git Blame Copy Permalink