forked from docs/doc-exports
Wuwan, Qi
050b395397
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com> Co-authored-by: Wuwan, Qi <wuwanqi1@noreply.gitea.eco.tsi-dev.otc-service.com> Co-committed-by: Wuwan, Qi <wuwanqi1@noreply.gitea.eco.tsi-dev.otc-service.com>
75 lines
9.4 KiB
HTML
75 lines
9.4 KiB
HTML
<a name="css_01_0122"></a><a name="css_01_0122"></a>
|
|
|
|
<h1 class="topictitle1">Cluster Planning for Vector Retrieval</h1>
|
|
<div id="body0000001261909812"><p id="css_01_0122__en-us_topic_0000001261909812_p2804352104215">Off-heap memory is used for index construction and query in vector retrieval. Therefore, the required cluster capacity is related to the index type and off-heap memory size. You can estimate the off-heap memory required by full indexing to select proper cluster specifications.</p>
|
|
<div class="p" id="css_01_0122__en-us_topic_0000001261909812_p1216235123617">There are different methods for estimating the size of off-heap memory required by different types of indexes. The calculation formulas are as follows:<ul id="css_01_0122__en-us_topic_0000001261909812_ul1626234143110"><li id="css_01_0122__en-us_topic_0000001261909812_li122610343315"><strong id="css_01_0122__en-us_topic_0000001261909812_b133801712201417">Graph Index</strong><p id="css_01_0122__en-us_topic_0000001261909812_p176375993914"><span><img class="mathml" id="css_01_0122__en-us_topic_0000001261909812_image116365963919" src="en-us_formulaimage_0000001714802349.png"></span></p>
|
|
<div class="note" id="css_01_0122__en-us_topic_0000001261909812_note264165903915"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="css_01_0122__en-us_topic_0000001261909812_p1164155923917">If you need to update indexes in real time, consider the off-heap memory overhead required for vector index construction and automatic merge. The actual size of required <strong id="css_01_0122__en-us_topic_0000001261909812_b13439033194312">mem_needs</strong> is at least 1.5 to 2 times of the original estimation.</p>
|
|
</div></div>
|
|
</li><li id="css_01_0122__en-us_topic_0000001261909812_li4981114418312"><strong id="css_01_0122__en-us_topic_0000001261909812_b1221195101517">PQ Index</strong><p id="css_01_0122__en-us_topic_0000001261909812_p131124517419"><span><img class="mathml" id="css_01_0122__en-us_topic_0000001261909812_image1253823512515" src="en-us_formulaimage_0000001667002558.png"></span></p>
|
|
</li><li id="css_01_0122__en-us_topic_0000001261909812_li85212188414"><strong id="css_01_0122__en-us_topic_0000001261909812_b88911056101515">FALT and IVF Indexes</strong><p id="css_01_0122__en-us_topic_0000001261909812_p1710782215213"><span><img class="mathml" id="css_01_0122__en-us_topic_0000001261909812_image19891851458" src="en-us_formulaimage_0000001666842858.png"></span></p>
|
|
</li></ul>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="css_01_0122__en-us_topic_0000001261909812_table1875348443" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="css_01_0122__en-us_topic_0000001261909812_row276174134419"><th align="left" class="cellrowborder" valign="top" width="28.89%" id="mcps1.3.2.2.2.3.1.1"><p id="css_01_0122__en-us_topic_0000001261909812_p1876841448">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="71.11%" id="mcps1.3.2.2.2.3.1.2"><p id="css_01_0122__en-us_topic_0000001261909812_p14761464413">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="css_01_0122__en-us_topic_0000001261909812_row107613413445"><td class="cellrowborder" valign="top" width="28.89%" headers="mcps1.3.2.2.2.3.1.1 "><p id="css_01_0122__en-us_topic_0000001261909812_p0761943444">dim</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.11%" headers="mcps1.3.2.2.2.3.1.2 "><p id="css_01_0122__en-us_topic_0000001261909812_p27614484419">Vector dimensions</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="css_01_0122__en-us_topic_0000001261909812_row47616484416"><td class="cellrowborder" valign="top" width="28.89%" headers="mcps1.3.2.2.2.3.1.1 "><p id="css_01_0122__en-us_topic_0000001261909812_p15767474413">neighbors</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.11%" headers="mcps1.3.2.2.2.3.1.2 "><p id="css_01_0122__en-us_topic_0000001261909812_p187644154418">Number of neighbors of a graph node. The default value is <strong id="css_01_0122__en-us_topic_0000001261909812_b159419234163">64</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="css_01_0122__en-us_topic_0000001261909812_row197616434417"><td class="cellrowborder" valign="top" width="28.89%" headers="mcps1.3.2.2.2.3.1.1 "><p id="css_01_0122__en-us_topic_0000001261909812_p6763411446">dim_size</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.11%" headers="mcps1.3.2.2.2.3.1.2 "><p id="css_01_0122__en-us_topic_0000001261909812_p16769444420">Number of bytes required by each dimension. The default value is four bytes in the float type.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="css_01_0122__en-us_topic_0000001261909812_row6764416440"><td class="cellrowborder" valign="top" width="28.89%" headers="mcps1.3.2.2.2.3.1.1 "><p id="css_01_0122__en-us_topic_0000001261909812_p18765444414">num</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.11%" headers="mcps1.3.2.2.2.3.1.2 "><p id="css_01_0122__en-us_topic_0000001261909812_p6761845447">Total number of vectors</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="css_01_0122__en-us_topic_0000001261909812_row97613454416"><td class="cellrowborder" valign="top" width="28.89%" headers="mcps1.3.2.2.2.3.1.1 "><p id="css_01_0122__en-us_topic_0000001261909812_p137611413447">delta</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.11%" headers="mcps1.3.2.2.2.3.1.2 "><p id="css_01_0122__en-us_topic_0000001261909812_p157613464410">Metadata size. This parameter can be left blank.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="css_01_0122__en-us_topic_0000001261909812_row27612424419"><td class="cellrowborder" valign="top" width="28.89%" headers="mcps1.3.2.2.2.3.1.1 "><p id="css_01_0122__en-us_topic_0000001261909812_p076144114413">frag_num</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.11%" headers="mcps1.3.2.2.2.3.1.2 "><p id="css_01_0122__en-us_topic_0000001261909812_p276114154418">Number of vector segments during quantization and coding. If this parameter is not specified when an index is created, the value is determined by the vector dimension <span class="parmname" id="css_01_0122__en-us_topic_0000001261909812_parmname32491715164713"><b>dim</b></span>.</p>
|
|
<pre class="screen" id="css_01_0122__en-us_topic_0000001261909812_screen28239236473">if dim <= 256:
|
|
frag_num = dim / 4
|
|
elif dim <= 512:
|
|
frag_num = dim / 8
|
|
else :
|
|
frag_num = 64</pre>
|
|
</td>
|
|
</tr>
|
|
<tr id="css_01_0122__en-us_topic_0000001261909812_row187684104410"><td class="cellrowborder" valign="top" width="28.89%" headers="mcps1.3.2.2.2.3.1.1 "><p id="css_01_0122__en-us_topic_0000001261909812_p776846446">frag_size</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.11%" headers="mcps1.3.2.2.2.3.1.2 "><p id="css_01_0122__en-us_topic_0000001261909812_p87604154416">Size of the center point during quantization and coding. The default value is <strong id="css_01_0122__en-us_topic_0000001261909812_b9711210192912">1</strong>. If the value of <strong id="css_01_0122__en-us_topic_0000001261909812_b820941782910">frag_num</strong> is greater than <strong id="css_01_0122__en-us_topic_0000001261909812_b13118947192911">256</strong>, the value of <strong id="css_01_0122__en-us_topic_0000001261909812_b798311143011">frag_size</strong> is <strong id="css_01_0122__en-us_topic_0000001261909812_b159601491291">2</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<p id="css_01_0122__en-us_topic_0000001261909812_p19203914115119">These calculation methods can estimate the size of off-heap memory required by a complete vector index. To determine cluster specifications, you also need to consider the heap memory overhead of each node.</p>
|
|
<p id="css_01_0122__en-us_topic_0000001261909812_p53134525012">Heap memory allocation policy: The size of the heap memory of each node is half of the node physical memory, and the maximum size is <strong id="css_01_0122__en-us_topic_0000001261909812_b551817410459">31 GB</strong>.</p>
|
|
<p id="css_01_0122__en-us_topic_0000001261909812_p16313452502">For example, if you create a Graph index for the SIFT10M dataset, set <strong id="css_01_0122__en-us_topic_0000001261909812_b1831116284615">dim</strong> to <strong id="css_01_0122__en-us_topic_0000001261909812_b238319816463">128</strong>, <strong id="css_01_0122__en-us_topic_0000001261909812_b289715131463">dim_size</strong> to <strong id="css_01_0122__en-us_topic_0000001261909812_b1939211614615">4</strong>, <strong id="css_01_0122__en-us_topic_0000001261909812_b9749163117469">neighbors</strong> to default value <strong id="css_01_0122__en-us_topic_0000001261909812_b1734324784619">64</strong>, and <strong id="css_01_0122__en-us_topic_0000001261909812_b14545148124818">num</strong> to <strong id="css_01_0122__en-us_topic_0000001261909812_b2648141513483">10 million</strong>, the off-heap memory required by the Graph index is as follows:</p>
|
|
<p id="css_01_0122__en-us_topic_0000001261909812_p931320513509"><span><img class="mathml" id="css_01_0122__en-us_topic_0000001261909812_image18313115115019" src="en-us_formulaimage_0000001714802345.png"></span></p>
|
|
<p id="css_01_0122__en-us_topic_0000001261909812_p531317519506">Considering the overhead of heap memory, a single server with <span class="parmvalue" id="css_01_0122__en-us_topic_0000001261909812_parmvalue342165318534"><b>8 vCPUs</b></span> and <strong id="css_01_0122__en-us_topic_0000001261909812_b92671130501">16 GB memory</strong> is recommended. If real-time write or update is required, you need to apply for larger memory.</p>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="css_01_0117.html">Vector Retrieval</a></div>
|
|
</div>
|
|
</div>
|
|
|