vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dms/umn/kafka-ug-0001.html
Chen, Junjie dd8a3a658b DMS UMN Initial Version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Chen, Junjie <chenjunjie@huawei.com>
Co-committed-by: Chen, Junjie <chenjunjie@huawei.com>
2022-12-08 00:33:11 +00:00

53 lines
16 KiB
HTML
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<a name="kafka-ug-0001"></a><a name="kafka-ug-0001"></a>
<h1 class="topictitle1">Cross-VPC Access to a Kafka Instance</h1>
<div id="body0000001096588567"><div class="section" id="kafka-ug-0001__section128927211065"><h4 class="sectiontitle">Context</h4><p id="kafka-ug-0001__p1449521416556">VPCs are logically isolated from each other. If a Kafka instance and a Kafka client are in different VPCs within a region, they cannot communicate with each other. In this case, you can use one of the following methods to access a Kafka instance across VPCs:</p>
<ul id="kafka-ug-0001__ul97214158109"><li id="kafka-ug-0001__li5721515171018">Establish a VPC peering connection to allow two VPCs to communicate with each other. For details, see section "VPC Peering Connection" in <em id="kafka-ug-0001__i13401524184513">Virtual Private Cloud User Guide</em>.</li><li id="kafka-ug-0001__li113217487569">Use VPC Endpoint (VPCEP) to establish a cross-VPC connection.</li></ul>
</div>
<div class="section" id="kafka-ug-0001__section1856313495332"><h4 class="sectiontitle">Scenario</h4><p id="kafka-ug-0001__p19184161715520">The following describes how to use VPCEP to implement cross-VPC access.</p>
<p id="kafka-ug-0001__p190851815389">VPCEP provides two types of resources: VPC endpoint services and VPC endpoints.</p>
<ul id="kafka-ug-0001__ul321112193919"><li id="kafka-ug-0001__li13214126397">A VPC endpoint service can be a Kafka instance which is accessed using VPC endpoints.</li><li id="kafka-ug-0001__li7537121183910">A VPC endpoint is a secure and private channel for connecting a VPC to a VPC endpoint service.</li></ul>
<div class="fignone" id="kafka-ug-0001__fig17225659161619"><span class="figcap"><b>Figure 1 </b>Working principle of accessing a Kafka instance across VPCs</span><br><span><img class="eddx" id="kafka-ug-0001__image169875400199" src="en-us_image_0000001376864660.png"></span></div>
</div>
<div class="section" id="kafka-ug-0001__section1677691110218"><h4 class="sectiontitle">Procedure</h4><div class="fignone" id="kafka-ug-0001__fig12258108164017"><span class="figcap"><b>Figure 2 </b>Process for accessing a Kafka instance across VPCs</span><br><span><img class="eddx" id="kafka-ug-0001__image1225820804011" src="en-us_image_0000001382159745.png"></span></div>
</div>
<div class="section" id="kafka-ug-0001__section171840351866"><h4 class="sectiontitle">Creating a VPC Endpoint Service</h4><ol id="kafka-ug-0001__ol16700104831910"><li id="kafka-ug-0001__li10427115412419"><span>Log in to the management console.</span></li><li id="kafka-ug-0001__li14905725134512"><span>Click <span><img id="kafka-ug-0001__image106626547332" src="en-us_image_0143929918.png"></span> in the upper left corner to select a region.</span><p><div class="note" id="kafka-ug-0001__note596412409275"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="kafka-ug-0001__p11964174020277">Select the region where your Kafka instance is located.</p>
</div></div>
</p></li><li id="kafka-ug-0001__li189561034172215"><span>Click <strong id="kafka-ug-0001__b8197419395652">Service List</strong> and choose <strong id="kafka-ug-0001__b208156690895652">Application</strong> &gt; <strong id="kafka-ug-0001__b126641929395652">Distributed Message Service</strong>. The Kafka instance list is displayed.</span></li><li id="kafka-ug-0001__li1933311013310"><span>Click the desired Kafka instance to view the instance details.</span></li><li id="kafka-ug-0001__li1470016488194"><a name="kafka-ug-0001__li1470016488194"></a><a name="li1470016488194"></a><span>In the <strong id="kafka-ug-0001__b6166141111476">Advanced Settings</strong> section on the <strong id="kafka-ug-0001__b1848582144719">Basic Information</strong> tab page, obtain the listeners IP addresses and port IDs of the instance for <strong id="kafka-ug-0001__b1522495205117">Cross-VPC Access</strong>.</span><p><div class="fignone" id="kafka-ug-0001__fig25911341876"><span class="figcap"><b>Figure 3 </b>Cross-VPC access–related listeners IP addresses and corresponding port IDs of the Kafka instance</span><br><span><img id="kafka-ug-0001__image959123411716" src="en-us_image_0000001328948884.png"></span></div>
</p></li><li id="kafka-ug-0001__li42609420212"><span>In the <strong id="kafka-ug-0001__b186011912105317">Network</strong> section on the <strong id="kafka-ug-0001__b10988161818536">Basic Information</strong> tab page, view the VPC to which the Kafka instance belongs.</span><p><div class="fignone" id="kafka-ug-0001__fig23264316235"><span class="figcap"><b>Figure 4 </b>Viewing the VPC to which the Kafka instance belongs</span><br><span><img id="kafka-ug-0001__image176598590259" src="en-us_image_0000001244372389.png"></span></div>
</p></li><li id="kafka-ug-0001__li19701310122315"><a name="kafka-ug-0001__li19701310122315"></a><a name="li19701310122315"></a><span>Click the VPC to obtain the VPC ID on the VPC console.</span><p><div class="fignone" id="kafka-ug-0001__fig99101255497"><span class="figcap"><b>Figure 5 </b>Obtaining the VPC ID</span><br><span><img id="kafka-ug-0001__image2091014550918" src="en-us_image_0000001328950348.png"></span></div>
</p></li><li id="kafka-ug-0001__li11323122315289"><a name="kafka-ug-0001__li11323122315289"></a><a name="li11323122315289"></a><span>Call the VPC Endpoint API to create a VPC endpoint service. For details, see "Creating a VPC Endpoint Service" in <em id="kafka-ug-0001__i16707459475">VPC Endpoint API Reference</em>.</span><p><pre class="screen" id="kafka-ug-0001__screen52371811174315">curl -i -k -H 'Accept:application/json' -H 'Content-Type:application/json;charset=utf8' -X POST -H "X-Auth-Token:$token" -d '{"port_id":"38axxxeac","vpc_id":"706xxx888","ports":[{"protocol":"TCP","client_port":9011,"server_port":9011 }],"approval_enabled":false,"service_type":"interface","server_type":"VM"}' https://{endpoint}/v1/{project_id}/vpc-endpoint-services</pre>
<p id="kafka-ug-0001__p712372418441">Parameter description:</p>
<ul id="kafka-ug-0001__ul109721527154415"><li id="kafka-ug-0001__li5972192734419"><strong id="kafka-ug-0001__b122616314441">token</strong>: an access credential issued to an IAM user to bear its identity and permissions. For details on how to obtain a token, see <a href="https://docs.otc.t-systems.com/en-us/api/iam/en-us_topic_0057845583.html" target="_blank" rel="noopener noreferrer">Obtaining a User Token</a>.</li><li id="kafka-ug-0001__li15324130161810"><strong id="kafka-ug-0001__b1123819614449">port_id</strong>: one of the port IDs obtained in <a href="#kafka-ug-0001__li1470016488194">5</a>.</li><li id="kafka-ug-0001__li14232192411218"><strong id="kafka-ug-0001__b22112913445">vpc_id</strong>: VPC ID obtained in <a href="#kafka-ug-0001__li19701310122315">7</a>.</li><li id="kafka-ug-0001__li163603457186"><strong id="kafka-ug-0001__b14850154020452">endpoint</strong>: VPCEP endpoint obtained from <a href="https://docs.otc.t-systems.com/en-us/endpoint/index.html" target="_blank" rel="noopener noreferrer">Regions and Endpoints</a>. The region must be the same as that of the Kafka instance.</li><li id="kafka-ug-0001__li124628547309"><strong id="kafka-ug-0001__b83401937184811">project_id</strong>: project ID obtained from "Obtaining a Project ID". The region must be the same as that of the Kafka instance. For details about how to obtain the value, see section "Common Parameters" &gt; "Obtaining a Project ID" in the <em id="kafka-ug-0001__i1557141218493">VPC Endpoint API Reference</em>.</li></ul>
<p id="kafka-ug-0001__p16995118192714"></p>
<p id="kafka-ug-0001__p163352316615">Record the value of <strong id="kafka-ug-0001__b185461917134213">service_name</strong> in the response. This parameter indicates the name of the VPC endpoint service.</p>
</p></li><li id="kafka-ug-0001__li7368125918119"><a name="kafka-ug-0001__li7368125918119"></a><a name="li7368125918119"></a><span>Repeat <a href="#kafka-ug-0001__li11323122315289">8</a> to create VPC endpoint services for other port IDs obtained in <a href="#kafka-ug-0001__li1470016488194">5</a> and record the VPC endpoint service names.</span></li></ol>
</div>
<div class="section" id="kafka-ug-0001__section386115711369"><h4 class="sectiontitle">(Optional) Adding a Whitelist Record</h4><p id="kafka-ug-0001__p195501184377">If the Kafka client and Kafka instance belong to different accounts, add the ID of the account to which the Kafka client belongs to the whitelist of the endpoint service. For details, see <a href="https://docs.otc.t-systems.com/usermanual/vpcep/vpcep_02_02034.html" target="_blank" rel="noopener noreferrer">Add a Whitelist Record</a>.</p>
</div>
<div class="section" id="kafka-ug-0001__section1649245213366"><h4 class="sectiontitle">Creating a VPC Endpoint</h4><ol id="kafka-ug-0001__ol5270192011377"><li id="kafka-ug-0001__li182701720183719"><a name="kafka-ug-0001__li182701720183719"></a><a name="li182701720183719"></a><span>Click <strong id="kafka-ug-0001__b86875993513">Service List</strong>. Then choose <strong id="kafka-ug-0001__b0523185773415">Networking</strong> &gt; <strong id="kafka-ug-0001__b1952325711343">VPC Endpoint</strong>.</span></li><li id="kafka-ug-0001__li1097665924012"><span>Click <strong id="kafka-ug-0001__b3597125717507">Create VPC Endpoint</strong>.</span></li><li id="kafka-ug-0001__li8149409405"><span>Set the following parameters:</span><p><ul id="kafka-ug-0001__ul165655051517"><li id="kafka-ug-0001__li1256500171513"><strong id="kafka-ug-0001__b14242221408">Region</strong>: Select the region that the Kafka instance is in.</li><li id="kafka-ug-0001__li1689535162214"><strong id="kafka-ug-0001__b176289169117">Service Category</strong>: Select <strong id="kafka-ug-0001__b82312504015">Find a service by name</strong>.</li><li id="kafka-ug-0001__li5677124012253"><strong id="kafka-ug-0001__b204221111829">VPC Endpoint Service Name</strong>: Enter the VPC endpoint service name recorded in <a href="#kafka-ug-0001__li11323122315289">8</a> and click <strong id="kafka-ug-0001__b865911297120">Verify</strong>. If <strong id="kafka-ug-0001__b87887283210">Service name found</strong> is displayed, proceed with subsequent operations.</li><li id="kafka-ug-0001__li188779365268"><strong id="kafka-ug-0001__b18689638927">VPC</strong>: Select the VPC that the Kafka instance is in.</li><li id="kafka-ug-0001__li19754165123816"><strong id="kafka-ug-0001__b911185212219">Subnet</strong>: Select the subnet that the Kafka instance is in.</li><li id="kafka-ug-0001__li194121351184516"><strong id="kafka-ug-0001__b84235270611353">Private IP Address</strong>: Select <strong id="kafka-ug-0001__b84235270684224">Automatic</strong>.</li></ul>
<p id="kafka-ug-0001__p10346253114517">Retain the default values for other parameters. For details, see <a href="https://docs.otc.t-systems.com/usermanual/vpcep/en-us_topic_0131645189.html" target="_blank" rel="noopener noreferrer">Creating a VPC Endpoint</a>.</p>
</p></li><li id="kafka-ug-0001__li74938503146"><span>Click <strong id="kafka-ug-0001__b10156256175114">Create Now</strong>.</span></li><li id="kafka-ug-0001__li14345185919339"><span>Confirm the configurations and submit the request.</span></li><li id="kafka-ug-0001__li1265111693415"><span>Go back to the VPC endpoint list and check whether the status of the created VPC endpoint has changed to <strong id="kafka-ug-0001__b1034371911618">Accepted</strong>. The <strong id="kafka-ug-0001__b1653455420611">Accepted</strong> state means that the VPC endpoint has been connected to the VPC endpoint service.</span><p><div class="fignone" id="kafka-ug-0001__fig855195817158"><span class="figcap"><b>Figure 6 </b>Checking the VPC endpoint status</span><br><span><img id="kafka-ug-0001__image15551958101511" src="en-us_image_0000001380194201.png"></span></div>
</p></li><li id="kafka-ug-0001__li1942253845112"><a name="kafka-ug-0001__li1942253845112"></a><a name="li1942253845112"></a><span>Click the VPC endpoint ID. On the <strong id="kafka-ug-0001__b33123781">Summary</strong> tab page, obtain the private IP address.</span><p><p id="kafka-ug-0001__p979320487510">You can use the private IP address to access the VPC endpoint service.</p>
<div class="fignone" id="kafka-ug-0001__fig51201236201616"><span class="figcap"><b>Figure 7 </b>Viewing the private IP address</span><br><span><img id="kafka-ug-0001__image1012013369164" src="en-us_image_0000001328954164.png"></span></div>
</p></li><li id="kafka-ug-0001__li923645116109"><a name="kafka-ug-0001__li923645116109"></a><a name="li923645116109"></a><span>Repeat <a href="#kafka-ug-0001__li182701720183719">1</a> to <a href="#kafka-ug-0001__li1942253845112">7</a> to create a VPC endpoint for each VPC endpoint service created in <a href="#kafka-ug-0001__li7368125918119">9</a>, and view and record the private IP addresses of the VPC endpoint services.</span></li></ol>
</div>
<div class="section" id="kafka-ug-0001__section159510175154"><h4 class="sectiontitle">Changing the advertised.listeners IP Address</h4><ol id="kafka-ug-0001__ol131014476156"><li id="kafka-ug-0001__li296363971814"><span>Click <strong id="kafka-ug-0001__b26820673795652">Service List</strong> and choose <strong id="kafka-ug-0001__b179920916395652">Application</strong> &gt; <strong id="kafka-ug-0001__b126737532595652">Distributed Message Service</strong>. The Kafka instance list is displayed.</span></li><li id="kafka-ug-0001__li024010166196"><span>Click the desired Kafka instance to view the instance details.</span></li><li id="kafka-ug-0001__li202449304196"><span>On the <strong id="kafka-ug-0001__b14364133213810">Advanced Settings</strong> section of the <strong id="kafka-ug-0001__b1191815561766">Basic Information</strong> tab page, click <strong id="kafka-ug-0001__b123891311599">Modify</strong> for <strong id="kafka-ug-0001__b169194562064">Cross-VPC Access</strong> to change the value of <strong id="kafka-ug-0001__b09191556069">advertised.listeners IP address</strong> to the private IP addresses recorded in <a href="#kafka-ug-0001__li1942253845112">7</a> and <a href="#kafka-ug-0001__li923645116109">8</a>. Click <strong id="kafka-ug-0001__b755432213914">Save</strong>.</span><p><div class="notice" id="kafka-ug-0001__note8247191763718"><span class="noticetitle"><img src="public_sys-resources/notice_3.0-en-us.png"> </span><div class="noticebody"><p id="kafka-ug-0001__p224821712375">Each IP address must match the corresponding port ID. Otherwise, the network will be disconnected.</p>
</div></div>
<div class="fignone" id="kafka-ug-0001__fig6446112151915"><a name="kafka-ug-0001__fig6446112151915"></a><a name="fig6446112151915"></a><span class="figcap"><b>Figure 8 </b>Changing the advertised.listeners IP addresses</span><br><span><img id="kafka-ug-0001__image5446226193" src="en-us_image_0000001380118889.png"></span></div>
</p></li></ol>
</div>
<div class="section" id="kafka-ug-0001__section72114271643"><h4 class="sectiontitle">Verifying Connectivity</h4><p id="kafka-ug-0001__p2063111531619">Check whether messages can be created and retrieved by referring to <a href="kafka-ug-180604020.html">Accessing a Kafka Instance Without SASL</a> or <a href="kafka-ug-180801001.html">Accessing a Kafka Instance with SASL</a>.</p>
<p id="kafka-ug-0001__p14394610154411">Notes:</p>
<ul id="kafka-ug-0001__ul469613431451"><li id="kafka-ug-0001__li388874315198">The address for connecting to a Kafka instance is in the format of "<em id="kafka-ug-0001__i7877652125319">advertised.listeners IP</em><strong id="kafka-ug-0001__b158781652105314">:9011</strong>". For example, the addresses for connecting to the Kafka instance shown in <a href="#kafka-ug-0001__fig6446112151915">Figure 8</a> are <strong id="kafka-ug-0001__b4879165213532">10.158.0.151:9011,10.158.0.162:9011,10.158.0.164:9011</strong>.</li><li id="kafka-ug-0001__li14696124317455">Configure inbound rules for the security group of the Kafka instance to allow access from <strong id="kafka-ug-0001__b197257441927">198.19.128.0/17</strong> over port <strong id="kafka-ug-0001__b66263456219">9011</strong>.</li><li id="kafka-ug-0001__li98341331018">If a network access control list (ACL) has been configured for the subnet of this instance, configure inbound rules for the network ACL to allow access from <strong id="kafka-ug-0001__b1987561781813">198.19.128.0/17</strong> and from the subnet used by the VPC endpoint.</li></ul>
<div class="note" id="kafka-ug-0001__note14901185218139"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="kafka-ug-0001__p1190175211319"><strong id="kafka-ug-0001__b5294743192118">198.19.128.0/17</strong> is the network segment allocated to the VPCEP service. To use VPCEP, allow access from this network segment.</p>
</div></div>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="kafka-ug190605003.html">Accessing a Kafka Instance</a></div>
</div>
</div>
View Git Blame Copy Permalink