forked from laiweijian4/doc-exports
Wuwan, Qi
050b395397
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com> Co-authored-by: Wuwan, Qi <wuwanqi1@noreply.gitea.eco.tsi-dev.otc-service.com> Co-committed-by: Wuwan, Qi <wuwanqi1@noreply.gitea.eco.tsi-dev.otc-service.com>
154 lines
15 KiB
HTML
154 lines
15 KiB
HTML
<a name="css_01_0187"></a><a name="css_01_0187"></a>
|
|
|
|
<h1 class="topictitle1">Enhanced Cold Data Query Performance</h1>
|
|
<div id="body0000001472612169"><div class="section" id="css_01_0187__en-us_topic_0000001472612169_section1190915124311"><h4 class="sectiontitle">Context</h4><p id="css_01_0187__en-us_topic_0000001472612169_p035931615312">When you query data on the <strong id="css_01_0187__en-us_topic_0000001472612169_b630315206165">Discover</strong> page of Kibana for the first time, all data needs to be obtained from OBS because there is no cache. If a large number of documents are returned, it takes a long time to obtain the corresponding time fields and file metadata from OBS. To accelerate queries the first time they run on the <strong id="css_01_0187__en-us_topic_0000001472612169_b099313528232">Discover</strong> page, you can cache data locally.</p>
|
|
</div>
|
|
<div class="section" id="css_01_0187__en-us_topic_0000001472612169_section1025720591221"><h4 class="sectiontitle">Prerequisites</h4><p id="css_01_0187__en-us_topic_0000001472612169_p159021137235">This feature is available in Elasticsearch clusters of versions 7.6.2 and 7.10.2 and Opensearch clusters created after February 2023.</p>
|
|
</div>
|
|
<div class="section" id="css_01_0187__en-us_topic_0000001472612169_section54701743874"><h4 class="sectiontitle">API for Querying Cold Data from Local Cache</h4><p id="css_01_0187__en-us_topic_0000001472612169_p1391942011133">This API can be used to query the cold data from local cache.</p>
|
|
<p id="css_01_0187__en-us_topic_0000001472612169_p882175712810">Example request:</p>
|
|
<pre class="screen" id="css_01_0187__en-us_topic_0000001472612169_screen153096221116">GET /_frozen_stats/local_cache
|
|
GET /_frozen_stats/local_cache/{nodeId}</pre>
|
|
<p id="css_01_0187__en-us_topic_0000001472612169_p198708572104">Response example:</p>
|
|
<pre class="screen" id="css_01_0187__en-us_topic_0000001472612169_screen85929147148">{
|
|
"_nodes" : {
|
|
"total" : 1,
|
|
"successful" : 1,
|
|
"failed" : 0
|
|
},
|
|
"cluster_name" : "elasticsearch",
|
|
"nodes" : {
|
|
"6by3lPy1R3m55Dcq3liK8Q" : {
|
|
"name" : "node-1",
|
|
"transport_address" : "127.0.0.1:9300",
|
|
"host" : "127.0.0.1",
|
|
"ip" : "127.0.0.1",
|
|
"local_cache" : {
|
|
"get_stats" : {
|
|
"get_total_count" : 562, //Total number of times data was retrieved from the local cold data cache.
|
|
"get_hit_count" : 562, //Total number of hits in the local cold data cache.
|
|
"get_miss_count" : 0, //Total number of local cold data cache misses.
|
|
"get_total_ns" : 43849200, //Total duration for retrieving data from the local cold data cache.
|
|
"get_avg_ns" : 78023 //Average duration for retrieving data from the local cold data cache.
|
|
},
|
|
"load_stats" : {
|
|
"load_count" : 2, //Number of times cold data was loaded from the local cache
|
|
"load_total_ms" : 29, //Total duration for loading cold data from the local cache
|
|
"load_avg_ms" : 14, //Average duration for loading cold data from the local cache
|
|
"load_fail_count" : 0, //Number of failure times for loading cold data from the local cache
|
|
"load_overflow_count" : 0 //Number of times the local cold data cache exceeds the cache pool size.
|
|
},
|
|
"reload_stats" : {
|
|
"reload_count" : 0, //Number of times the local cold data cache was regenerated.
|
|
"reload_total_ms" : 0, //Total duration for regenerating the local cold data cache.
|
|
"reload_avg_ms" : 0, //Average duration for regenerating the local cold data cache.
|
|
"reload_fail_count" : 0 //Number of failures in regenerating the local cold data cache.
|
|
},
|
|
"init_stats" : {
|
|
"init_count" : 0, //Number of times the local cold data cache was initialized.
|
|
"init_total_ms" : 0, //Total duration for initializing the local cold data cache.
|
|
"init_avg_ms" : 0, //Average duration for initializing the local cold data cache.
|
|
"init_fail_count" : 0 //Number of failures in initializing the local cold data cache.
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}</pre>
|
|
<p id="css_01_0187__en-us_topic_0000001472612169_p1239150101915"></p>
|
|
</div>
|
|
<div class="section" id="css_01_0187__en-us_topic_0000001472612169_section688714346202"><h4 class="sectiontitle">Configuring Parameters</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="css_01_0187__en-us_topic_0000001472612169_table1549126112114" frame="border" border="1" rules="all"><thead align="left"><tr id="css_01_0187__en-us_topic_0000001472612169_row1687192652118"><th align="left" class="cellrowborder" valign="top" width="12.88871112888711%" id="mcps1.3.4.2.1.8.1.1"><p id="css_01_0187__en-us_topic_0000001472612169_p7871326202118">Configuration Item</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="5.449455054494551%" id="mcps1.3.4.2.1.8.1.2"><p id="css_01_0187__en-us_topic_0000001472612169_p587152611216">Type</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="5.599440055994401%" id="mcps1.3.4.2.1.8.1.3"><p id="css_01_0187__en-us_topic_0000001472612169_p38732611212">Unit</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="9.339066093390661%" id="mcps1.3.4.2.1.8.1.4"><p id="css_01_0187__en-us_topic_0000001472612169_p08752614216">Value Range</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="6.219378062193781%" id="mcps1.3.4.2.1.8.1.5"><p id="css_01_0187__en-us_topic_0000001472612169_p14871426142111">Scope</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="7.219278072192781%" id="mcps1.3.4.2.1.8.1.6"><p id="css_01_0187__en-us_topic_0000001472612169_p118862652117">Can Be Dynamically Modified</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="53.284671532846716%" id="mcps1.3.4.2.1.8.1.7"><p id="css_01_0187__en-us_topic_0000001472612169_p888626112116">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="css_01_0187__en-us_topic_0000001472612169_row8881326122118"><td class="cellrowborder" valign="top" width="12.88871112888711%" headers="mcps1.3.4.2.1.8.1.1 "><p id="css_01_0187__en-us_topic_0000001472612169_p7887267213">low_cost.local_cache.max.capacity</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="5.449455054494551%" headers="mcps1.3.4.2.1.8.1.2 "><p id="css_01_0187__en-us_topic_0000001472612169_p5888269213">Integer</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="5.599440055994401%" headers="mcps1.3.4.2.1.8.1.3 "><p id="css_01_0187__en-us_topic_0000001472612169_p1388192652114">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.339066093390661%" headers="mcps1.3.4.2.1.8.1.4 "><p id="css_01_0187__en-us_topic_0000001472612169_p118862682120">The value ranges from 10 to 5000. The default value is <strong id="css_01_0187__en-us_topic_0000001472612169_b16429153013387">500</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="6.219378062193781%" headers="mcps1.3.4.2.1.8.1.5 "><p id="css_01_0187__en-us_topic_0000001472612169_p128820261219">node</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.219278072192781%" headers="mcps1.3.4.2.1.8.1.6 "><p id="css_01_0187__en-us_topic_0000001472612169_p888226192111">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="53.284671532846716%" headers="mcps1.3.4.2.1.8.1.7 "><p id="css_01_0187__en-us_topic_0000001472612169_p58882672110">Maximum number of available cold data caches on a node. Each shard corresponds to a cache object.</p>
|
|
<div class="note" id="css_01_0187__en-us_topic_0000001472612169_note125431510514"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="css_01_0187__en-us_topic_0000001472612169_ul970310302573"><li id="css_01_0187__en-us_topic_0000001472612169_li2243153417578">If the heap memory usage remains high, decrease the value.</li><li id="css_01_0187__en-us_topic_0000001472612169_li3704163011570">If the value of <strong id="css_01_0187__en-us_topic_0000001472612169_b93137268390">load_overflow_count</strong> keeps increasing rapidly, increase the value.</li></ul>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="css_01_0187__en-us_topic_0000001472612169_row10881626122117"><td class="cellrowborder" valign="top" width="12.88871112888711%" headers="mcps1.3.4.2.1.8.1.1 "><p id="css_01_0187__en-us_topic_0000001472612169_p158862642110">index.low_cost.local_cache.threshold</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="5.449455054494551%" headers="mcps1.3.4.2.1.8.1.2 "><p id="css_01_0187__en-us_topic_0000001472612169_p17881226102115">Integer</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="5.599440055994401%" headers="mcps1.3.4.2.1.8.1.3 "><p id="css_01_0187__en-us_topic_0000001472612169_p138812612114">%</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.339066093390661%" headers="mcps1.3.4.2.1.8.1.4 "><p id="css_01_0187__en-us_topic_0000001472612169_p1788226132112">The value ranges from 0 to 100. The default value is <strong id="css_01_0187__en-us_topic_0000001472612169_b1556638103918">50</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="6.219378062193781%" headers="mcps1.3.4.2.1.8.1.5 "><p id="css_01_0187__en-us_topic_0000001472612169_p38819266211">index</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.219278072192781%" headers="mcps1.3.4.2.1.8.1.6 "><p id="css_01_0187__en-us_topic_0000001472612169_p188172618215">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="53.284671532846716%" headers="mcps1.3.4.2.1.8.1.7 "><p id="css_01_0187__en-us_topic_0000001472612169_p38852682118">Threshold for enabling the local cache of cold data.</p>
|
|
<div class="note" id="css_01_0187__en-us_topic_0000001472612169_note107343483515"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="css_01_0187__en-us_topic_0000001472612169_ul957011127573"><li id="css_01_0187__en-us_topic_0000001472612169_li417516170578">If the percentage of date fields is less than the value of this parameter, the cold data of the date type will be cached locally. Otherwise, this parameter is not used.</li><li id="css_01_0187__en-us_topic_0000001472612169_li16570412175715">If the date fields of the current index occupy most of the data volume of the current index, you are not advised to use this function.</li></ul>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="css_01_0187__en-us_topic_0000001472612169_row168812617215"><td class="cellrowborder" valign="top" width="12.88871112888711%" headers="mcps1.3.4.2.1.8.1.1 "><p id="css_01_0187__en-us_topic_0000001472612169_p16891826162111">index.low_cost.local_cache.evict_time</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="5.449455054494551%" headers="mcps1.3.4.2.1.8.1.2 "><p id="css_01_0187__en-us_topic_0000001472612169_p5891265214">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="5.599440055994401%" headers="mcps1.3.4.2.1.8.1.3 "><p id="css_01_0187__en-us_topic_0000001472612169_p198912662120">Days</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="9.339066093390661%" headers="mcps1.3.4.2.1.8.1.4 "><p id="css_01_0187__en-us_topic_0000001472612169_p12891926172117">The value ranges from 1d to 365d. The default value is <strong id="css_01_0187__en-us_topic_0000001472612169_b12540163564010">30d</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="6.219378062193781%" headers="mcps1.3.4.2.1.8.1.5 "><p id="css_01_0187__en-us_topic_0000001472612169_p10898263214">index</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="7.219278072192781%" headers="mcps1.3.4.2.1.8.1.6 "><p id="css_01_0187__en-us_topic_0000001472612169_p148952692110">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="53.284671532846716%" headers="mcps1.3.4.2.1.8.1.7 "><p id="css_01_0187__en-us_topic_0000001472612169_p58992611215">Wait time before cold data is deleted from local cache. The value is determined based on <strong id="css_01_0187__en-us_topic_0000001472612169_b14330203610610">index.frozen_date</strong> (time when the freezing is successful).</p>
|
|
<div class="note" id="css_01_0187__en-us_topic_0000001472612169_note166021510115213"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="css_01_0187__en-us_topic_0000001472612169_ul10342139195717"><li id="css_01_0187__en-us_topic_0000001472612169_li12799114219579">For indexes that have been frozen in old clusters and do not have <strong id="css_01_0187__en-us_topic_0000001472612169_b169910370719">index.frozen_date</strong> specified, the value of this parameter is determined based on the index creation time.</li><li id="css_01_0187__en-us_topic_0000001472612169_li13342739175717">You are advised to adjust the deletion time based on the disk usage to avoid high disk usage.</li></ul>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="css_01_0187__en-us_topic_0000001472612169_section321819171616"><h4 class="sectiontitle">Modifying Parameters</h4><ul id="css_01_0187__en-us_topic_0000001472612169_ul1321111971620"><li id="css_01_0187__en-us_topic_0000001472612169_li120101911614">Run the following command to modify <strong id="css_01_0187__en-us_topic_0000001472612169_b1345918514117">low_cost.local_cache.max.capacity</strong>:<pre class="screen" id="css_01_0187__en-us_topic_0000001472612169_screen19201219131612">PUT _cluster/settings
|
|
{
|
|
"persistent": {
|
|
"low_cost.local_cache.max.capacity":1000
|
|
}
|
|
}</pre>
|
|
</li><li id="css_01_0187__en-us_topic_0000001472612169_li192181915164">Run the following command to modify <strong id="css_01_0187__en-us_topic_0000001472612169_b16540102324118">index.low_cost.local_cache.threshold</strong>:<pre class="screen" id="css_01_0187__en-us_topic_0000001472612169_screen182181911615">PUT es_write_pref2-00000000021/_settings
|
|
{
|
|
"index.low_cost.local_cache.threshold":20
|
|
}</pre>
|
|
</li><li id="css_01_0187__en-us_topic_0000001472612169_li202110195164">Run the following command to modify <strong id="css_01_0187__en-us_topic_0000001472612169_b22871734114110">index.low_cost.local_cache.evict_time</strong>:<pre class="screen" id="css_01_0187__en-us_topic_0000001472612169_screen1321131931615">PUT es_write_pref2-00000000021/_settings
|
|
{
|
|
"index.low_cost.local_cache.evict_time":"7d"
|
|
}</pre>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="css_01_0112.html">Storage-Compute Decoupling</a></div>
|
|
</div>
|
|
</div>
|
|
|