forked from docs/doc-exports
Yang, Tong
3f5759eed2
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
110 lines
16 KiB
HTML
110 lines
16 KiB
HTML
<a name="mrs_01_2302"></a><a name="mrs_01_2302"></a>
|
|
|
|
<h1 class="topictitle1">Redis-based CacheStore of HiveMetaStore</h1>
|
|
<div id="body8662426"><div class="section" id="mrs_01_2302__en-us_topic_0000001219149683_section897072914375"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_2302__en-us_topic_0000001219149683_p162521931153713">The MetaStore service of Hive can cache the metadata of some tables in Redis.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_2302__en-us_topic_0000001219149683_section13562315184415"><h4 class="sectiontitle">Prerequisites</h4><p id="mrs_01_2302__en-us_topic_0000001219149683_p0312171854420">The Redis service has been installed in a cluster.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_2302__en-us_topic_0000001219149683_section182122772710"><h4 class="sectiontitle">Configure Parameters Related to Metastore</h4><ol id="mrs_01_2302__en-us_topic_0000001219149683_ol1671143288"><li id="mrs_01_2302__en-us_topic_0000001219149683_li07754449304"><span>Log in to FusionInsight Manager. For details, see <a href="mrs_01_2124.html">Accessing FusionInsight Manager</a>. Choose <strong id="mrs_01_2302__en-us_topic_0000001219149683_b6671241379">Cluster</strong> > <em id="mrs_01_2302__en-us_topic_0000001219149683_i2033020272371">Name of the desired cluster</em> > <strong id="mrs_01_2302__en-us_topic_0000001219149683_b1263953013718">Services</strong> > <strong id="mrs_01_2302__en-us_topic_0000001219149683_b1300163419376">Hive</strong> > <strong id="mrs_01_2302__en-us_topic_0000001219149683_b18241337133710">Configurations</strong> > <strong id="mrs_01_2302__en-us_topic_0000001219149683_b440911418372">All Configurations</strong> > <strong id="mrs_01_2302__en-us_topic_0000001219149683_b1657117433379">MetaStore(Role)</strong> > <strong id="mrs_01_2302__en-us_topic_0000001219149683_b45975465371">Customization</strong>. Modify the following parameters to interconnect the cache of MetaStore to the Redis service.</span><p>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_2302__en-us_topic_0000001219149683_table1392765193117" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters</caption><thead align="left"><tr id="mrs_01_2302__en-us_topic_0000001219149683_row20927185183111"><th align="left" class="cellrowborder" valign="top" width="27.532753275327533%" id="mcps1.3.3.2.1.2.1.2.4.1.1"><p id="mrs_01_2302__en-us_topic_0000001219149683_p159271751143113">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="34.283428342834284%" id="mcps1.3.3.2.1.2.1.2.4.1.2"><p id="mrs_01_2302__en-us_topic_0000001219149683_p29271151183111">Value</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="38.18381838183818%" id="mcps1.3.3.2.1.2.1.2.4.1.3"><p id="mrs_01_2302__en-us_topic_0000001219149683_p1492755110316">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_2302__en-us_topic_0000001219149683_row9927851163111"><td class="cellrowborder" valign="top" width="27.532753275327533%" headers="mcps1.3.3.2.1.2.1.2.4.1.1 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p992716513313">hive.metastore.rawstore.impl</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="34.283428342834284%" headers="mcps1.3.3.2.1.2.1.2.4.1.2 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p119271751163119">org.apache.hadoop.hive.metastore.cache.redis.RedisCachedStore</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.3.2.1.2.1.2.4.1.3 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p10927165112317">(Mandatory) Implementation class of CachedStore. Use the customized RedisCachedStore.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_2302__en-us_topic_0000001219149683_row692725173116"><td class="cellrowborder" valign="top" width="27.532753275327533%" headers="mcps1.3.3.2.1.2.1.2.4.1.1 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p192785119319">redis.cluster.host.and.port</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="34.283428342834284%" headers="mcps1.3.3.2.1.2.1.2.4.1.2 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p19927105117313">xxx.xxx.xxx.xxx:22400;xxx.xxx.xxx.xxx.xxx:22401</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.3.2.1.2.1.2.4.1.3 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p492785193115">(Mandatory) IP address and port number of any node in the Redis cluster. The format is <em id="mrs_01_2302__en-us_topic_0000001219149683_i143962040133817">ip:port. ip:port</em>. The value cannot end with a semicolon (;).</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_2302__en-us_topic_0000001219149683_row59271851163113"><td class="cellrowborder" valign="top" width="27.532753275327533%" headers="mcps1.3.3.2.1.2.1.2.4.1.1 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p1592895118318">metastore.cached.rawstore.cached.object.whitelist</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="34.283428342834284%" headers="mcps1.3.3.2.1.2.1.2.4.1.2 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p16928251203111">catalog.database.table,catalog.database.table</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.3.2.1.2.1.2.4.1.3 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p98453811114">(Optional) Cache table whitelist. The configured tables are cached to Redis. Multiple tables are separated by commas (,). The default separator is <strong id="mrs_01_2302__en-us_topic_0000001219149683_b6821740195618">.*</strong>, that is, all tables are cached.</p>
|
|
<div class="note" id="mrs_01_2302__en-us_topic_0000001219149683_note143912444910"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_2302__en-us_topic_0000001219149683_p4401944399">The table name consists of <strong id="mrs_01_2302__en-us_topic_0000001219149683_b1841019349429">catalog.database.table</strong>. The default catalog is <strong id="mrs_01_2302__en-us_topic_0000001219149683_b1532245104413">hive</strong>.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_2302__en-us_topic_0000001219149683_row1592875119311"><td class="cellrowborder" valign="top" width="27.532753275327533%" headers="mcps1.3.3.2.1.2.1.2.4.1.1 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p1992819514317">metastore.cached.rawstore.cached.object.blacklist</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="34.283428342834284%" headers="mcps1.3.3.2.1.2.1.2.4.1.2 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p1565955816216">catalog.database.table,catalog.database.table</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.3.2.1.2.1.2.4.1.3 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p9928125111318">(Optional) Cache table blacklist. Tables that are not configured in the blacklist are not cached in Redis. Multiple tables are separated by commas (,). By default, this parameter is left blank.</p>
|
|
<div class="note" id="mrs_01_2302__en-us_topic_0000001219149683_note291043611011"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_2302__en-us_topic_0000001219149683_p14929336191010">The table name consists of <strong id="mrs_01_2302__en-us_topic_0000001219149683_b1872737124520">catalog.database.table</strong>. The default catalog is <strong id="mrs_01_2302__en-us_topic_0000001219149683_b1731037174512">hive</strong>.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_2302__en-us_topic_0000001219149683_row3928451193120"><td class="cellrowborder" valign="top" width="27.532753275327533%" headers="mcps1.3.3.2.1.2.1.2.4.1.1 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p49288511317">redis.cache.prewarm.cron</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="34.283428342834284%" headers="mcps1.3.3.2.1.2.1.2.4.1.2 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p592885163113">cronTab expression, for example, 0 0 16 * *?</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.3.2.1.2.1.2.4.1.3 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p7928105117318">(Optional) Periodically execute the corn expression of prewarm to update the data cached in the metabase to redisCache for synchronization.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_2302__en-us_topic_0000001219149683_row169281651113115"><td class="cellrowborder" valign="top" width="27.532753275327533%" headers="mcps1.3.3.2.1.2.1.2.4.1.1 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p1792812513314">metastore.cached.rawstore.catalogs</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="34.283428342834284%" headers="mcps1.3.3.2.1.2.1.2.4.1.2 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p1992813512317">hive</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.3.2.1.2.1.2.4.1.3 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p19928651133117">(Optional) Catalog to be cached. The default value is <strong id="mrs_01_2302__en-us_topic_0000001219149683_b1036320583467">hive</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_2302__en-us_topic_0000001219149683_row159284515310"><td class="cellrowborder" valign="top" width="27.532753275327533%" headers="mcps1.3.3.2.1.2.1.2.4.1.1 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p0928135143112">jedis.pool.max.wait.mills</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="34.283428342834284%" headers="mcps1.3.3.2.1.2.1.2.4.1.2 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p17928851153118">30,000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.3.2.1.2.1.2.4.1.3 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p89282513319">(Optional) Obtain the Redis connection timeout interval. In security mode, the timeout interval can be longer. The unit is ms. The default value is 30,000 ms.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_2302__en-us_topic_0000001219149683_row10362951797"><td class="cellrowborder" valign="top" width="27.532753275327533%" headers="mcps1.3.3.2.1.2.1.2.4.1.1 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p16362205694">jedis.pool.max.idle</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="34.283428342834284%" headers="mcps1.3.3.2.1.2.1.2.4.1.2 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p7362551590">200</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.3.2.1.2.1.2.4.1.3 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p0362165997">(Optional) Maximum number of idle connections in the Jedis connection pool. You are advised to set this parameter to the value of <strong id="mrs_01_2302__en-us_topic_0000001219149683_b8822028114713">max.total</strong>. The default value is 200.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_2302__en-us_topic_0000001219149683_row2767128992"><td class="cellrowborder" valign="top" width="27.532753275327533%" headers="mcps1.3.3.2.1.2.1.2.4.1.1 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p137672816914">jedis.pool.max.total</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="34.283428342834284%" headers="mcps1.3.3.2.1.2.1.2.4.1.2 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p77671818917">200</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.3.2.1.2.1.2.4.1.3 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p20767981390">(Optional) Maximum number of connections in the Jedis connection pool. The default value is 200.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_2302__en-us_topic_0000001219149683_row4706143192614"><td class="cellrowborder" valign="top" width="27.532753275327533%" headers="mcps1.3.3.2.1.2.1.2.4.1.1 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p10707113182611">redis.security.enabled</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="34.283428342834284%" headers="mcps1.3.3.2.1.2.1.2.4.1.2 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p370793172618">true or false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.18381838183818%" headers="mcps1.3.3.2.1.2.1.2.4.1.3 "><p id="mrs_01_2302__en-us_topic_0000001219149683_p77071336264">(Optional) Whether to enable the Redis cache security mode. The default value is <strong id="mrs_01_2302__en-us_topic_0000001219149683_b646943184817">true</strong>, indicating that the security mode is enabled.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="note" id="mrs_01_2302__en-us_topic_0000001219149683_note12292355122918"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_2302__en-us_topic_0000001219149683_p14292755122915">If the cluster is installed in non-security mode, choose <strong id="mrs_01_2302__en-us_topic_0000001219149683_b282191010489">Cluster</strong> > <em id="mrs_01_2302__en-us_topic_0000001219149683_i13723313154820">Name of the desired cluster</em> > <strong id="mrs_01_2302__en-us_topic_0000001219149683_b15535171654815">Services</strong> > <strong id="mrs_01_2302__en-us_topic_0000001219149683_b14241720194815">Redis</strong> > <strong id="mrs_01_2302__en-us_topic_0000001219149683_b113152312482">Configurations</strong> > <strong id="mrs_01_2302__en-us_topic_0000001219149683_b92041727114818">All Configurations</strong> > <strong id="mrs_01_2302__en-us_topic_0000001219149683_b59441733104810">Redis</strong> > <strong id="mrs_01_2302__en-us_topic_0000001219149683_b20271193894813">Security</strong> and check whether the value of <strong id="mrs_01_2302__en-us_topic_0000001219149683_b78041742144811">REDIS_SECURITY_ENABLED</strong> is <strong id="mrs_01_2302__en-us_topic_0000001219149683_b0674124718481">false</strong>. If not, change it to <strong id="mrs_01_2302__en-us_topic_0000001219149683_b176871051114811">false</strong>.Otherwise, the Redis service does not comply with the non-security mode of the current Metastore.</p>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_2302__en-us_topic_0000001219149683_li13402316143018"><span>Save the configuration and choose <strong id="mrs_01_2302__en-us_topic_0000001219149683_b2026418681103543">Dashboard</strong> > <strong id="mrs_01_2302__en-us_topic_0000001219149683_b1433727877103543">More</strong> ><strong id="mrs_01_2302__en-us_topic_0000001219149683_b140685088103543"> Restart Service</strong> to restart the Hive service.</span></li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_2302__en-us_topic_0000001219149683_section198472313515"><h4 class="sectiontitle">Precautions</h4><p id="mrs_01_2302__en-us_topic_0000001219149683_p19191162910369">If Redis is switched back to the native non-cache mode and then switched back after a period of time, the added, deleted, or modified metadata cannot be synchronized to the Redis when the database is used. Therefore, before switching back, you must clear the cache table in the Redis and synchronize the metadata again in either of the following two clearing modes:</p>
|
|
<ul id="mrs_01_2302__en-us_topic_0000001219149683_ul1956719398120"><li id="mrs_01_2302__en-us_topic_0000001219149683_li4567339101216">Log in to the Redis client and run the <strong id="mrs_01_2302__en-us_topic_0000001219149683_b14822234125116">flushall</strong> command on all Redis nodes.</li><li id="mrs_01_2302__en-us_topic_0000001219149683_li7719123615317">Log in to the Redis client and run the following commands to change the two Redis identifiers. <em id="mrs_01_2302__en-us_topic_0000001219149683_i1144410462514">{hiveServiceName}</em> is the value of <strong id="mrs_01_2302__en-us_topic_0000001219149683_b11185195013512">HIVE_DEFAULT_GROUP</strong> in the Metastore configuration file <strong id="mrs_01_2302__en-us_topic_0000001219149683_b1561255319517">ENV_VARS</strong>. The default value is <strong id="mrs_01_2302__en-us_topic_0000001219149683_b196204585514">hive</strong>.<p id="mrs_01_2302__en-us_topic_0000001219149683_p4497125310419"><strong id="mrs_01_2302__en-us_topic_0000001219149683_b0301672307">set </strong><em id="mrs_01_2302__en-us_topic_0000001219149683_i257967113018">{hiveServiceName}</em><strong id="mrs_01_2302__en-us_topic_0000001219149683_b18301127183016">-hive-isRedisAvailable false</strong></p>
|
|
<p id="mrs_01_2302__en-us_topic_0000001219149683_p1049765317413"><strong id="mrs_01_2302__en-us_topic_0000001219149683_b1699131217302">del </strong><em id="mrs_01_2302__en-us_topic_0000001219149683_i1232081317305">{hiveServiceName}</em><strong id="mrs_01_2302__en-us_topic_0000001219149683_b29919121305">-hive-isCanPrewarm</strong></p>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0581.html">Using Hive</a></div>
|
|
</div>
|
|
</div>
|
|
|