Yang, Tong 48706b7552 MRS COMP-LTS 320-lts.1 version
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2024-04-12 12:51:10 +00:00

96 lines
20 KiB
HTML

<a name="mrs_01_17511"></a><a name="mrs_01_17511"></a>
<h1 class="topictitle1">Connecting Hive with External RDS</h1>
<div id="body8662426"><div class="note" id="mrs_01_17511__en-us_topic_0000001219350615_note835285022717"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_17511__en-us_topic_0000001219350615_ul4356143102519"><li id="mrs_01_17511__en-us_topic_0000001219350615_li33261346284">RDS indicates the relational database in this section. This section describes how to connect Hive with the open-source MySQL and Postgres databases.</li><li id="mrs_01_17511__en-us_topic_0000001219350615_li133561243112515">After an external metadata database is deployed in a cluster with Hive data, the original metadata table will not be automatically synchronized. Therefore, before installing Hive, you need to check whether the metadata is external RDS or built-in DBService. If the metadata is external RDS, you need to configure the metadata to external RDS when installing Hive or when there is no Hive data. After the installation, the metadata cannot be modified. Otherwise, the original metadata will be lost.</li></ul>
</div></div>
<p id="mrs_01_17511__en-us_topic_0000001219350615_p93596469572"><strong id="mrs_01_17511__en-us_topic_0000001219350615_b13474515162416">Hive supports access to open source MySQL and Postgres metabases.</strong></p>
<ol id="mrs_01_17511__en-us_topic_0000001219350615_ol173272291573"><li id="mrs_01_17511__en-us_topic_0000001219350615_li1132702919577"><span>Install the open source MySQL or Postgres database.</span><p><div class="note" id="mrs_01_17511__en-us_topic_0000001219350615_note13389269303"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_17511__en-us_topic_0000001219350615_p134281153183010">The node where the database is installed must be in the same network segment as the cluster, so that they can access each other.</p>
</div></div>
</p></li><li id="mrs_01_17511__en-us_topic_0000001219350615_li161241356122611"><span>Upload the driver package.</span><p><ul id="mrs_01_17511__en-us_topic_0000001219350615_ul1198031112275"><li id="mrs_01_17511__en-us_topic_0000001219350615_li13980171152714">Postgres: Use the open source Postgres driver package to replace the existing one of the cluster. Upload the Postgres driver package <strong id="mrs_01_17511__en-us_topic_0000001219350615_b229618193195">postgresql-42.2.5.jar</strong> to the <em id="mrs_01_17511__en-us_topic_0000001219350615_i6208125217194">${BIGDATA_HOME}</em><strong id="mrs_01_17511__en-us_topic_0000001219350615_b14257114910191">/third_lib/Hive</strong> directory on all MetaStore instance nodes. To download the open-source driver package, visit <a href="https://repo1.maven.org/maven2/org/postgresql/postgresql/42.2.5/" target="_blank" rel="noopener noreferrer">https://repo1.maven.org/maven2/org/postgresql/postgresql/42.2.5/</a>.</li><li id="mrs_01_17511__en-us_topic_0000001219350615_li1498011112712">MySQL: Go to the MySQL official website (https://www.mysql.com/), select <strong id="mrs_01_17511__en-us_topic_0000001219350615_b5742165712244">Downloads</strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b34421259142414">Community</strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b1863115122511">MySQL Connectors</strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b127461679255">Connector/J</strong> to download the driver package of the corresponding version, and upload the driver package to the <em id="mrs_01_17511__en-us_topic_0000001219350615_i171251217142514">${BIGDATA_HOME}</em><strong id="mrs_01_17511__en-us_topic_0000001219350615_b913312208253">/third_lib/Hive</strong> directory on all RDSMetastore nodes.</li></ul>
</p></li><li id="mrs_01_17511__en-us_topic_0000001219350615_li836403843219"><span>Create a database in RDS as the specified database of Hive metadata.</span><p><p id="mrs_01_17511__en-us_topic_0000001219350615_p20514342184216">Run the following command in Postgres or MySQL:</p>
<p id="mrs_01_17511__en-us_topic_0000001219350615_p178576423321"><strong id="mrs_01_17511__en-us_topic_0000001219350615_b1213184923510">create database </strong><em id="mrs_01_17511__en-us_topic_0000001219350615_i815104913513">databasename</em><strong id="mrs_01_17511__en-us_topic_0000001219350615_b131314497351">;</strong></p>
<div class="note" id="mrs_01_17511__en-us_topic_0000001219350615_note8312130123616"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_17511__en-us_topic_0000001219350615_p21131913610">In the preceding command, <em id="mrs_01_17511__en-us_topic_0000001219350615_i123275712289">databasename</em> indicates the database name.</p>
</div></div>
</p></li><li id="mrs_01_17511__en-us_topic_0000001219350615_li15848182614321"><span>Import the SQL statements for creating metadata tables.</span><p><ul id="mrs_01_17511__en-us_topic_0000001219350615_ul65215412364"><li id="mrs_01_17511__en-us_topic_0000001219350615_li1052041113619">Path of the Postgres SQL file: <strong id="mrs_01_17511__en-us_topic_0000001219350615_b11991744195215">${BIGDATA_HOME}/FusionInsight_HD_<span id="mrs_01_17511__en-us_topic_0000001219350615_text15995182919377">8.1.2.2</span>/install/FusionInsight-Hive-3.1.0/hive-3.1.0/scripts/metastore/upgrade/postgres/hive-schema-3.1.0.postgres.sql</strong><p id="mrs_01_17511__en-us_topic_0000001219350615_p1585615754315">Run the following command to import the SQL file to Postgres:</p>
<p id="mrs_01_17511__en-us_topic_0000001219350615_p565025716374"><strong id="mrs_01_17511__en-us_topic_0000001219350615_b4418654123314">./bin/psql -U </strong><em id="mrs_01_17511__en-us_topic_0000001219350615_i1777715493314">username</em><strong id="mrs_01_17511__en-us_topic_0000001219350615_b4919143573411"> -d </strong><em id="mrs_01_17511__en-us_topic_0000001219350615_i6207193620348">databasename</em><strong id="mrs_01_17511__en-us_topic_0000001219350615_b19919143543410"> -f hive-schema-3.1.0.postgres.sql</strong></p>
<p id="mrs_01_17511__en-us_topic_0000001219350615_p185671510443">Specifically:</p>
<p id="mrs_01_17511__en-us_topic_0000001219350615_p5791111110445"><strong id="mrs_01_17511__en-us_topic_0000001219350615_b76420113390">./bin/psql</strong> is in the Postgres installation directory.</p>
<p id="mrs_01_17511__en-us_topic_0000001219350615_p116322187448"><strong id="mrs_01_17511__en-us_topic_0000001219350615_b1338992134012">username</strong> indicates the username for logging in to Postgres.</p>
<p id="mrs_01_17511__en-us_topic_0000001219350615_p2572113573312"><strong id="mrs_01_17511__en-us_topic_0000001219350615_b16951204312447"><em id="mrs_01_17511__en-us_topic_0000001219350615_i52386436448">databasename</em></strong> indicates the database name.</p>
</li><li id="mrs_01_17511__en-us_topic_0000001219350615_li016050103610">Path of the MySQL file: <strong id="mrs_01_17511__en-us_topic_0000001219350615_b14593175511526">${BIGDATA_HOME}/FusionInsight_HD_<span id="mrs_01_17511__en-us_topic_0000001219350615_text65634383381">8.1.2.2</span>/install/FusionInsight-Hive-3.1.0/hive-3.1.0/scripts/metastore/upgrade/mysql/hive-schema-3.1.0.mysql.sql</strong><p id="mrs_01_17511__en-us_topic_0000001219350615_p197905154410">Run the following command to import the SQL file to the MySQL database:</p>
<p id="mrs_01_17511__en-us_topic_0000001219350615_p20506384455"><strong id="mrs_01_17511__en-us_topic_0000001219350615_b182511815174018">./bin/mysql -u </strong><em id="mrs_01_17511__en-us_topic_0000001219350615_i8254161584010">username</em><strong id="mrs_01_17511__en-us_topic_0000001219350615_b12704155064014"> -p</strong><em id="mrs_01_17511__en-us_topic_0000001219350615_i8264952144010">password</em><strong id="mrs_01_17511__en-us_topic_0000001219350615_b783131324117"> -D</strong><em id="mrs_01_17511__en-us_topic_0000001219350615_i150762494114">databasename</em><strong id="mrs_01_17511__en-us_topic_0000001219350615_b38315131413">&lt;hive-schema-3.1.0.mysql.sql</strong></p>
<p id="mrs_01_17511__en-us_topic_0000001219350615_p109222205459">Specifically:</p>
<p id="mrs_01_17511__en-us_topic_0000001219350615_p08629287456"><strong id="mrs_01_17511__en-us_topic_0000001219350615_b963318228479">./bin/mysql</strong> is in the MySQL installation directory.</p>
<p id="mrs_01_17511__en-us_topic_0000001219350615_p176931832124515"><strong id="mrs_01_17511__en-us_topic_0000001219350615_b15656737124712"><em id="mrs_01_17511__en-us_topic_0000001219350615_i3310819104316">username</em></strong> indicates the user name for logging in to MySQL.</p>
<p id="mrs_01_17511__en-us_topic_0000001219350615_p17281191543919"><strong id="mrs_01_17511__en-us_topic_0000001219350615_b763111474810"><em id="mrs_01_17511__en-us_topic_0000001219350615_i166251214134820">databasename</em></strong> indicates the database name.</p>
</li></ul>
</p></li><li id="mrs_01_17511__en-us_topic_0000001219350615_li18339443124"><span>To connect Hive to the open-source database, log in to FusionInsight Manager. For details, see <a href="mrs_01_2124.html">Accessing FusionInsight Manager</a>. Choose <strong id="mrs_01_17511__en-us_topic_0000001219350615_b12798172615271">Cluster</strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b14769191134914"><em id="mrs_01_17511__en-us_topic_0000001219350615_i10693122992713">Name of the desired cluster</em></strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b20661113332712">Services</strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b1498993782714">Hive</strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b83451540142717">Configurations</strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b6395445192719">All Configurations</strong> &gt;<strong id="mrs_01_17511__en-us_topic_0000001219350615_b1509747162713"> Hive(Service)</strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b97451450192715">MetaDB</strong>, modify the following parameters, and click <strong id="mrs_01_17511__en-us_topic_0000001219350615_b10288155920421">Save</strong>.</span><p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_17511__en-us_topic_0000001219350615_table1999612014189" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_17511__en-us_topic_0000001219350615_row11997110101813"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.5.2.1.2.4.1.1"><p id="mrs_01_17511__en-us_topic_0000001219350615_p999716018189">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.5.2.1.2.4.1.2"><p id="mrs_01_17511__en-us_topic_0000001219350615_p189977014185">Default Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.5.2.1.2.4.1.3"><p id="mrs_01_17511__en-us_topic_0000001219350615_p59971509184">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_17511__en-us_topic_0000001219350615_row199971906187"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.5.2.1.2.4.1.1 "><p id="mrs_01_17511__en-us_topic_0000001219350615_p109974014180">javax.jdo.option.ConnectionDriverName</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.5.2.1.2.4.1.2 "><p id="mrs_01_17511__en-us_topic_0000001219350615_p8997106184">org.postgresql.Driver</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.5.2.1.2.4.1.3 "><p id="mrs_01_17511__en-us_topic_0000001219350615_p127931248184119">Driver class for connecting metadata on MetaStore</p>
<ul id="mrs_01_17511__en-us_topic_0000001219350615_ul64983344215"><li id="mrs_01_17511__en-us_topic_0000001219350615_li1738115441021">If an external MySQL database is used, the value is:<p id="mrs_01_17511__en-us_topic_0000001219350615_p16597100136"><a name="mrs_01_17511__en-us_topic_0000001219350615_li1738115441021"></a><a name="en-us_topic_0000001219350615_li1738115441021"></a><strong id="mrs_01_17511__en-us_topic_0000001219350615_b1881216377916">com.mysql.jdbc.Driver</strong></p>
</li><li id="mrs_01_17511__en-us_topic_0000001219350615_li17989205012220">If an external Postgres database is used, the value is:<p id="mrs_01_17511__en-us_topic_0000001219350615_p198141557822"><a name="mrs_01_17511__en-us_topic_0000001219350615_li17989205012220"></a><a name="en-us_topic_0000001219350615_li17989205012220"></a><strong id="mrs_01_17511__en-us_topic_0000001219350615_b1941117288916">org.postgresql.Driver</strong></p>
</li></ul>
</td>
</tr>
<tr id="mrs_01_17511__en-us_topic_0000001219350615_row59975013188"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.5.2.1.2.4.1.1 "><p id="mrs_01_17511__en-us_topic_0000001219350615_p6997110121818">javax.jdo.option.ConnectionURL</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.5.2.1.2.4.1.2 "><p id="mrs_01_17511__en-us_topic_0000001219350615_p499760191811">jdbc:postgresql://%{DBSERVICE_FLOAT_IP}%{DBServer}:%{DBSERVICE_CPORT}/hivemeta?socketTimeout=60</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.5.2.1.2.4.1.3 "><p id="mrs_01_17511__en-us_topic_0000001219350615_p09971702184">URL of the JDBC link of the MetaStore metadata</p>
<ul id="mrs_01_17511__en-us_topic_0000001219350615_ul1596171094317"><li id="mrs_01_17511__en-us_topic_0000001219350615_li596121054310">If an external MySQL database is used, the value is:<p id="mrs_01_17511__en-us_topic_0000001219350615_p58521847115413"><a name="mrs_01_17511__en-us_topic_0000001219350615_li596121054310"></a><a name="en-us_topic_0000001219350615_li596121054310"></a><strong id="mrs_01_17511__en-us_topic_0000001219350615_b69231443442">jdbc:mysql://</strong><em id="mrs_01_17511__en-us_topic_0000001219350615_i139154154312">MySQL IP address</em>:<em id="mrs_01_17511__en-us_topic_0000001219350615_i18605185717436">MySQL port number</em><strong id="mrs_01_17511__en-us_topic_0000001219350615_b1520018110444">/test? useSSL=false</strong></p>
</li><li id="mrs_01_17511__en-us_topic_0000001219350615_li6751715184318">If an external Postgres database is used, the value is:<p id="mrs_01_17511__en-us_topic_0000001219350615_p661172011429"><a name="mrs_01_17511__en-us_topic_0000001219350615_li6751715184318"></a><a name="en-us_topic_0000001219350615_li6751715184318"></a><strong id="mrs_01_17511__en-us_topic_0000001219350615_b13519163344417">jdbc:postgresql://</strong><em id="mrs_01_17511__en-us_topic_0000001219350615_i20865123784410">Postgres IP address</em><strong id="mrs_01_17511__en-us_topic_0000001219350615_b64019416443">:</strong><em id="mrs_01_17511__en-us_topic_0000001219350615_i11950164414446">Postgres port number</em><strong id="mrs_01_17511__en-us_topic_0000001219350615_b15741447174415">/test</strong></p>
</li></ul>
</td>
</tr>
<tr id="mrs_01_17511__en-us_topic_0000001219350615_row189973011820"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.5.2.1.2.4.1.1 "><p id="mrs_01_17511__en-us_topic_0000001219350615_p199700131813">javax.jdo.option.ConnectionUserName</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.5.2.1.2.4.1.2 "><p id="mrs_01_17511__en-us_topic_0000001219350615_p6997909183">hive${SERVICE_INDEX}${SERVICE_INDEX}</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.5.2.1.2.4.1.3 "><p id="mrs_01_17511__en-us_topic_0000001219350615_p5997408180">Username for connecting to the metadata database on Metastore</p>
</td>
</tr>
</tbody>
</table>
</div>
</p></li><li id="mrs_01_17511__en-us_topic_0000001219350615_li11909455124315"><span>Change the Postgres database password in MetaStore. Choose <strong id="mrs_01_17511__en-us_topic_0000001219350615_b9836916192917">Cluster</strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b11809939105117"><em id="mrs_01_17511__en-us_topic_0000001219350615_i772162012299">Name of the desired cluster</em></strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b154641923112919">Services</strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b273472715291">Hive</strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b1256417308299">Configurations</strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b122351934142912">All Configurations</strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b99291136152912">MetaStore(Role)</strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b4463174019298">MetaDB</strong>, modify the following parameters, and click <strong id="mrs_01_17511__en-us_topic_0000001219350615_b8547115711514">Save</strong>.</span><p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_17511__en-us_topic_0000001219350615_table10910115517438" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_17511__en-us_topic_0000001219350615_row1910175564313"><th align="left" class="cellrowborder" valign="top" width="33.373337333733375%" id="mcps1.3.3.6.2.1.2.4.1.1"><p id="mrs_01_17511__en-us_topic_0000001219350615_p16910125511433">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.29332933293329%" id="mcps1.3.3.6.2.1.2.4.1.2"><p id="mrs_01_17511__en-us_topic_0000001219350615_p191025518434">Default Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.6.2.1.2.4.1.3"><p id="mrs_01_17511__en-us_topic_0000001219350615_p1591025519435">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_17511__en-us_topic_0000001219350615_row1991018554439"><td class="cellrowborder" valign="top" width="33.373337333733375%" headers="mcps1.3.3.6.2.1.2.4.1.1 "><p id="mrs_01_17511__en-us_topic_0000001219350615_p1591018553430">javax.jdo.option.extend.ConnectionPassword</p>
</td>
<td class="cellrowborder" valign="top" width="33.29332933293329%" headers="mcps1.3.3.6.2.1.2.4.1.2 "><p id="mrs_01_17511__en-us_topic_0000001219350615_p1091015554313">******</p>
</td>
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.6.2.1.2.4.1.3 "><p id="mrs_01_17511__en-us_topic_0000001219350615_p7910055174313">User password for connecting to the external metadata database on Metastore. The password is encrypted in the background.</p>
</td>
</tr>
</tbody>
</table>
</div>
</p></li><li id="mrs_01_17511__en-us_topic_0000001219350615_li437619388210"><span>Log in to each MetaStore background node and check whether the local directory <strong id="mrs_01_17511__en-us_topic_0000001219350615_b67392512311">/opt/Bigdata/tmp</strong> exists.</span><p><ul id="mrs_01_17511__en-us_topic_0000001219350615_ul15522114205916"><li id="mrs_01_17511__en-us_topic_0000001219350615_li12522121485912">If yes, go to <a href="#mrs_01_17511__en-us_topic_0000001219350615_li24241321154318">8</a>.</li><li id="mrs_01_17511__en-us_topic_0000001219350615_li1955520265111">If no, run the following commands to create one:<p id="mrs_01_17511__en-us_topic_0000001219350615_p45551826815"><a name="mrs_01_17511__en-us_topic_0000001219350615_li1955520265111"></a><a name="en-us_topic_0000001219350615_li1955520265111"></a><strong id="mrs_01_17511__en-us_topic_0000001219350615_b205551261719">mkdir -p /opt/</strong><strong id="mrs_01_17511__en-us_topic_0000001219350615_b5555162617115">Bigdata/tmp</strong></p>
<p id="mrs_01_17511__en-us_topic_0000001219350615_p455517268118"><strong id="mrs_01_17511__en-us_topic_0000001219350615_b255517263118">chmod 755 /opt/</strong><strong id="mrs_01_17511__en-us_topic_0000001219350615_b5555126310">Bigdata/tmp</strong></p>
</li></ul>
</p></li><li id="mrs_01_17511__en-us_topic_0000001219350615_li24241321154318"><a name="mrs_01_17511__en-us_topic_0000001219350615_li24241321154318"></a><a name="en-us_topic_0000001219350615_li24241321154318"></a><span>Save the configuration. Choose <strong id="mrs_01_17511__en-us_topic_0000001219350615_b37311688226">Dashboard</strong> &gt; <strong id="mrs_01_17511__en-us_topic_0000001219350615_b187371852218">More</strong> &gt;<strong id="mrs_01_17511__en-us_topic_0000001219350615_b167378852213"> Restart Service</strong>, and enter the password to restart the Hive service.</span></li></ol>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0581.html">Using Hive</a></div>
</div>
</div>