vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/css/best-practice/css_07_0024.html
Wuwan, Qi f203ab3c98 CSS Bestpractice 20231018 version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Wuwan, Qi <wuwanqi1@noreply.gitea.eco.tsi-dev.otc-service.com>
Co-committed-by: Wuwan, Qi <wuwanqi1@noreply.gitea.eco.tsi-dev.otc-service.com>
2024-02-15 08:56:15 +00:00

201 lines
20 KiB
HTML
Raw Blame History

<a name="css_07_0024"></a><a name="css_07_0024"></a>
<h1 class="topictitle1">Using CSS to Accelerate Database Query and Analysis</h1>
<div id="body0000001307800405"><div class="section" id="css_07_0024__section8905144012112"><h4 class="sectiontitle">Overview</h4><p id="css_07_0024__p116014151047">Elasticsearch is used as a supplement to relational databases, such as MySQL and GaussDB(for MySQL), to improve the full-text search and high-concurrency ad hoc query capabilities of the databases.</p>
<p id="css_07_0024__p16609514951">This chapter describes how to synchronize data from a MySQL database to CSS to accelerate full-text search and ad hoc query and analysis. The following figure shows the solution process.</p>
<div class="fignone" id="css_07_0024__fig581572110813"><span class="figcap"><b>Figure 1 </b>Using CSS to accelerate database query and analysis</span><br><span><img id="css_07_0024__image4815721383" src="en-us_image_0000001299757424.png"></span></div>
<ol id="css_07_0024__ol19292184011152"><li id="css_07_0024__li1292144013155">Service data is stored in the MySQL database.</li><li id="css_07_0024__li994554117159">DRS synchronizes data from MySQL to CSS in real time.</li><li id="css_07_0024__li133726456156">CSS is used for full-text search and data query and analysis.</li></ol>
</div>
<div class="section" id="css_07_0024__section207038451919"><h4 class="sectiontitle">Prerequisites</h4><ul id="css_07_0024__ul163844183187"><li id="css_07_0024__li1138491811812">A CSS cluster and a MySQL database in security mode have been created, and they are in the same VPC and security group.</li><li id="css_07_0024__li8404525121915">Data to be synchronized exists in the MySQL database. This section uses the following table structure and initial data as an example.<ol id="css_07_0024__ol1891874710208"><li id="css_07_0024__li1918124762014">Create a student information table in MySQL.<pre class="screen" id="css_07_0024__screen482456121718">CREATE TABLE `<strong id="css_07_0024__b79761412112117">student</strong>` (
`<strong id="css_07_0024__b1559716158211">dsc</strong>` varchar(100) COLLATE utf8mb4_general_ci DEFAULT NULL,
`<strong id="css_07_0024__b1145872019211">age</strong>` smallint unsigned DEFAULT NULL,
`<strong id="css_07_0024__b189612212110">name</strong>` varchar(32) COLLATE utf8mb4_general_ci NOT NULL,
`<strong id="css_07_0024__b79577247211">id</strong>` int unsigned NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;</pre>
</li><li id="css_07_0024__li791884702014">Insert the initial data of three students into the MySQL database.<pre class="screen" id="css_07_0024__screen58559432209">INSERT INTO student (id,name,age,dsc)
VALUES
('1','Jack Ma Yun','50','Jack Ma Yun is a Chinese business magnate, investor and philanthropist.'),
('2','will smith','22','also known by his stage name the Fresh Prince, is an American actor, rapper, and producer.'),
('3','James Francis Cameron','68','the director of avatar');</pre>
</li></ol>
</li><li id="css_07_0024__li13230110132410">Indexes have been created in the CSS cluster and match the table indexes in the MySQL database.<div class="p" id="css_07_0024__p1435516110246"><a name="css_07_0024__li13230110132410"></a><a name="li13230110132410"></a>The following is an example of the indexes in the cluster in this chapter:<pre class="screen" id="css_07_0024__screen1682155611718">PUT <strong id="css_07_0024__b1468793118574">student</strong>
{
"settings": {
"number_of_replicas": 0,
"number_of_shards": 3
},
"mappings": {
"properties": {
"<strong id="css_07_0024__b133143359574">id</strong>": {
"type": "keyword"
},
"<strong id="css_07_0024__b15313103985712">name</strong>": {
"type": "short"
},
"<strong id="css_07_0024__b5461114218579">age</strong>": {
"type": "short"
},
"<strong id="css_07_0024__b3689174425714">desc</strong>": {
"type": "text"
}
}
}
}</pre>
</div>
<p id="css_07_0024__p1635719177268">Configure <strong id="css_07_0024__b14833174114589">number_of_shards</strong> and <strong id="css_07_0024__b1362904365810">number_of_replicas</strong> as needed.</p>
</li></ul>
</div>
<div class="section" id="css_07_0024__section1959845416112"><h4 class="sectiontitle">Procedure</h4><ol id="css_07_0024__ol1555440132714"><li id="css_07_0024__li55715301810"><span>Use DRS to synchronize MySQL data to CSS in real time. For details, see .</span><p><div class="p" id="css_07_0024__p297614301582">In this example, configure the parameters by following the suggestions in <a href="#css_07_0024__table131609582113">Table 1</a>.
<div class="tablenoborder"><a name="css_07_0024__table131609582113"></a><a name="table131609582113"></a><table cellpadding="4" cellspacing="0" summary="" id="css_07_0024__table131609582113" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Synchronization parameters</caption><thead align="left"><tr id="css_07_0024__row1516035891120"><th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.3.2.1.2.1.2.2.4.1.1"><p id="css_07_0024__p884810122135">Module</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="25%" id="mcps1.3.3.2.1.2.1.2.2.4.1.2"><p id="css_07_0024__p4160158181116">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50%" id="mcps1.3.3.2.1.2.1.2.2.4.1.3"><p id="css_07_0024__p716095851118">Suggestion</p>
</th>
</tr>
</thead>
<tbody><tr id="css_07_0024__row116017585119"><td class="cellrowborder" rowspan="3" valign="top" width="25%" headers="mcps1.3.3.2.1.2.1.2.2.4.1.1 "><p id="css_07_0024__p138481212171311"><strong id="css_07_0024__b7722654134717">Create Synchronization Instance</strong> &gt; <strong id="css_07_0024__b718825612474">Synchronize Instance Details</strong></p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.1.2.1.2.2.4.1.2 "><p id="css_07_0024__p13160165831113"><strong id="css_07_0024__b175175282471">Network Type</strong></p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.2.1.2.2.4.1.3 "><p id="css_07_0024__p9160115814116">Select <strong id="css_07_0024__b143178431771739">VPC</strong>.</p>
</td>
</tr>
<tr id="css_07_0024__row9160195817114"><td class="cellrowborder" valign="top" headers="mcps1.3.3.2.1.2.1.2.2.4.1.1 "><p id="css_07_0024__p1160758121117"><strong id="css_07_0024__b1244364734716">Source DB Instance</strong></p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.2.1.2.1.2.2.4.1.2 "><p id="css_07_0024__p61601358191111">Select the RDS for MySQL instance to be synchronized, that is, the MySQL database that stores service data.</p>
</td>
</tr>
<tr id="css_07_0024__row01607589113"><td class="cellrowborder" valign="top" headers="mcps1.3.3.2.1.2.1.2.2.4.1.1 "><p id="css_07_0024__p1816035811115"><strong id="css_07_0024__b8371134814716">Synchronization Instance Subnet</strong></p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.2.1.2.1.2.2.4.1.2 "><p id="css_07_0024__p216085820112">Select the subnet where the synchronization instance is located. You are advised to select the subnet where the database instance and the CSS cluster are located.</p>
</td>
</tr>
<tr id="css_07_0024__row4160195818118"><td class="cellrowborder" rowspan="4" valign="top" width="25%" headers="mcps1.3.3.2.1.2.1.2.2.4.1.1 "><p id="css_07_0024__p3848312121312"><strong id="css_07_0024__b171317034813">Configure Source and Destination Databases</strong> &gt; <strong id="css_07_0024__b8116416485">Destination Database</strong></p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.1.2.1.2.2.4.1.2 "><p id="css_07_0024__p191601758121118"><strong id="css_07_0024__b496821554612">VPC</strong> and <strong id="css_07_0024__b172492166">Subnet</strong></p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.2.1.2.2.4.1.3 "><p id="css_07_0024__p3160145811115">Select the VPC and subnet of the CSS cluster.</p>
</td>
</tr>
<tr id="css_07_0024__row982919191295"><td class="cellrowborder" valign="top" headers="mcps1.3.3.2.1.2.1.2.2.4.1.1 "><p id="css_07_0024__p7829141922917"><strong id="css_07_0024__b127878139464">IP Address or Domain Name</strong></p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.2.1.2.1.2.2.4.1.2 "><p id="css_07_0024__p1683021915296">Enter the IP address of the CSS cluster. For details, see <a href="#css_07_0024__li1999495913506">Obtaining the IP address of a CSS cluster</a>.</p>
</td>
</tr>
<tr id="css_07_0024__row209351620102919"><td class="cellrowborder" valign="top" headers="mcps1.3.3.2.1.2.1.2.2.4.1.1 "><p id="css_07_0024__p12935142082919"><strong id="css_07_0024__b07471548124515">Database Username</strong> and <strong id="css_07_0024__b430718478459">Database Password</strong></p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.2.1.2.1.2.2.4.1.2 "><p id="css_07_0024__p1993510209291">Enter the administrator username (<strong id="css_07_0024__b5331234192419">admin</strong>) and password of the CSS cluster.</p>
</td>
</tr>
<tr id="css_07_0024__row930315216295"><td class="cellrowborder" valign="top" headers="mcps1.3.3.2.1.2.1.2.2.4.1.1 "><p id="css_07_0024__p1303122120295"><strong id="css_07_0024__b717692024714">Encryption Certificate</strong></p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.2.1.2.1.2.2.4.1.2 "><p id="css_07_0024__p17303142112915">Select the security certificate of the CSS cluster. If <strong id="css_07_0024__b1742012134494">SSL Connection</strong> is not enabled, you do not need to select any certificate. For details, see <a href="#css_07_0024__li78671114175115">Obtaining the security certificate of a CSS cluster</a>.</p>
</td>
</tr>
<tr id="css_07_0024__row216035820111"><td class="cellrowborder" rowspan="3" valign="top" width="25%" headers="mcps1.3.3.2.1.2.1.2.2.4.1.1 "><p id="css_07_0024__p184851231313"><strong id="css_07_0024__b4657105614910">Set Synchronization Task</strong></p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.1.2.1.2.2.4.1.2 "><p id="css_07_0024__p51601581118"><strong id="css_07_0024__b1434312504">Flow Control</strong></p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.2.1.2.2.4.1.3 "><p id="css_07_0024__p4161165818110">Select <strong id="css_07_0024__b330781814506">No</strong>.</p>
</td>
</tr>
<tr id="css_07_0024__row1116175811116"><td class="cellrowborder" valign="top" headers="mcps1.3.3.2.1.2.1.2.2.4.1.1 "><p id="css_07_0024__p1161185851118"><strong id="css_07_0024__b79461320195011">Synchronization Object Type</strong></p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.2.1.2.1.2.2.4.1.2 "><p id="css_07_0024__p191611258131114">Deselect <strong id="css_07_0024__b0188193415018">Table structure</strong>, because the indexes matching MySQL tables have been created in the CSS cluster.</p>
</td>
</tr>
<tr id="css_07_0024__row8161125851115"><td class="cellrowborder" valign="top" headers="mcps1.3.3.2.1.2.1.2.2.4.1.1 "><p id="css_07_0024__p31611587111"><strong id="css_07_0024__b15864185415502">Synchronization Object</strong></p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.2.1.2.1.2.2.4.1.2 "><p id="css_07_0024__p1916125881115">Select <strong id="css_07_0024__b59699144215">Tables</strong>. Select the database and table name corresponding to CSS.</p>
<div class="note" id="css_07_0024__note185881251171413"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="css_07_0024__p1588151131412">Ensure the type name in the configuration item is the same as the index name, that is, <strong id="css_07_0024__b14312352714">_doc</strong>.</p>
</div></div>
</td>
</tr>
<tr id="css_07_0024__row16792814114"><td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.1.2.1.2.2.4.1.1 "><p id="css_07_0024__p27931111314"><strong id="css_07_0024__b1775383314211">Process Data</strong></p>
</td>
<td class="cellrowborder" valign="top" width="25%" headers="mcps1.3.3.2.1.2.1.2.2.4.1.2 "><p id="css_07_0024__p5793711111">-</p>
</td>
<td class="cellrowborder" valign="top" width="50%" headers="mcps1.3.3.2.1.2.1.2.2.4.1.3 "><p id="css_07_0024__p1793513117">Click <strong id="css_07_0024__b2098710391215">Next</strong>.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<p id="css_07_0024__p14841144172216">After the synchronization task is started, wait until the <strong id="css_07_0024__b4919164214616">Status</strong> of the task changes from <strong id="css_07_0024__b1520210501615">Full</strong> synchronization to <strong id="css_07_0024__b1325711531966">Incremental</strong>, indicating real-time synchronization has started.</p>
</p></li><li id="css_07_0024__li165555404278"><span>Check the synchronization status of the database.</span><p><ol type="a" id="css_07_0024__ol1064943616367"><li id="css_07_0024__li1564953610360">Verify full data synchronization.<p id="css_07_0024__p3649736193617"><a name="css_07_0024__li1564953610360"></a><a name="li1564953610360"></a>Run the following command in Kibana of CSS to check whether full data has been synchronized to CSS:</p>
<pre class="screen" id="css_07_0024__screen1064963653615">GET student/_search</pre>
</li><li id="css_07_0024__li964933623618">Insert new data in the source cluster and check whether the data is synchronized to CSS.<div class="p" id="css_07_0024__p106491836153616"><a name="css_07_0024__li964933623618"></a><a name="li964933623618"></a>For example, insert a record whose <span class="parmname" id="css_07_0024__parmname161141524444"><b>id</b></span> is <span class="parmvalue" id="css_07_0024__parmvalue3309418445"><b>4</b></span> in the source cluster.<pre class="screen" id="css_07_0024__screen4649236113615">INSERT INTO student (id,name,age,dsc)
VALUES
('4','Bill Gates','50','Gates III is an American business magnate, software developer, investor, author, and philanthropist.')</pre>
</div>
<p id="css_07_0024__p1064919363362">Run the following command in Kibana of CSS to check whether new data is synchronized to CSS:</p>
<pre class="screen" id="css_07_0024__screen126491836143618">GET student/_search</pre>
</li><li id="css_07_0024__li5649436113611">Update data in the source cluster and check whether the data is synchronized to CSS.<div class="p" id="css_07_0024__p064933617368"><a name="css_07_0024__li5649436113611"></a><a name="li5649436113611"></a>For example, in the record whose <strong id="css_07_0024__b123851582910">id</strong> is <strong id="css_07_0024__b4626166112919">4</strong>, change the value of <strong id="css_07_0024__b69494002918">age</strong> from <strong id="css_07_0024__b12883316142914">50</strong> to <strong id="css_07_0024__b379571772918">55</strong>.<pre class="screen" id="css_07_0024__screen2649236193618">UPDATE student set age='55' WHERE id=4;</pre>
</div>
<p id="css_07_0024__p13649183643620">Run the following command in Kibana of CSS to check whether the data is updated in CSS:</p>
<pre class="screen" id="css_07_0024__screen12649123693614">GET student/_search</pre>
</li><li id="css_07_0024__li196491536123610">Delete data from the source cluster and check whether the data is deleted synchronously from CSS.<div class="p" id="css_07_0024__p1064933615363"><a name="css_07_0024__li196491536123610"></a><a name="li196491536123610"></a>For example, delete the record whose <span class="parmname" id="css_07_0024__parmname10634162223015"><b>id</b></span> is <span class="parmvalue" id="css_07_0024__parmvalue146349229307"><b>4</b></span>.<pre class="screen" id="css_07_0024__screen964963615366">DELETE FROM student WHERE id=4;</pre>
</div>
<p id="css_07_0024__p1064993643611">Run the following command in Kibana of CSS to check whether the data is deleted synchronously from CSS:</p>
<pre class="screen" id="css_07_0024__screen264923611363">GET student/_search</pre>
</li></ol>
</p></li><li id="css_07_0024__li18759135563315"><span>Verify the full-text search capability of the database.</span><p><p id="css_07_0024__p8117219591">For example, run the following command to query the data that contains <strong id="css_07_0024__b9912161412311">avatar</strong> in <strong id="css_07_0024__b4417191653110">dsc</strong> in CSS:</p>
<pre class="screen" id="css_07_0024__screen27361981907">GET student/_search
{
"query": {
"match": {
"dsc": "avatar"
}
}
}</pre>
</p></li><li id="css_07_0024__li1664132516342"><span>Verify the ad hoc query capability of the database.</span><p><p id="css_07_0024__p171274321902">For example, query <strong id="css_07_0024__b045717133316">philanthropist</strong> whose age is greater than <strong id="css_07_0024__b788216963310">40</strong> in CSS.</p>
<pre class="screen" id="css_07_0024__screen14689511910">GET student/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"dsc": "philanthropist"
}
},
{
"range": {
"age": {
"gte": 40
}
}
}
]
}
}
}</pre>
</p></li><li id="css_07_0024__li63971147205815"><span>Verify the statistical analysis capability of the database.</span><p><p id="css_07_0024__p1739734717584">For example, use CSS to collect statistics on the age distributions of all users.</p>
<pre class="screen" id="css_07_0024__screen6397204765812">GET student/_search
{
"size": 0,
"query": {
"match_all": {}
},
"aggs": {
"age_count": {
"terms": {
"field": "age",
"size": 10
}
}
}
}</pre>
</p></li></ol>
</div>
<div class="section" id="css_07_0024__section16716164414411"><h4 class="sectiontitle">Other Operations</h4><ul id="css_07_0024__ul699415917506"><li id="css_07_0024__li1999495913506"><a name="css_07_0024__li1999495913506"></a><a name="li1999495913506"></a><strong id="css_07_0024__b106016141242">Obtaining the IP address of a CSS cluster</strong><ol id="css_07_0024__ol65311228104710"><li id="css_07_0024__li17866174620423">In the navigation pane on the left, choose <span class="parmname" id="css_07_0024__parmname138661463421"><b>Clusters</b></span>.</li><li id="css_07_0024__li14265565421">In the cluster list, locate a cluster, and obtain the IP address of the CSS cluster from the <strong id="css_07_0024__b66161642377">Private Network Address</strong> column. Generally, the IP address format is <em id="css_07_0024__i18649471783">&lt;host&gt;</em><strong id="css_07_0024__b2579681816">:</strong><em id="css_07_0024__i137378108811">&lt;port&gt;</em> or <em id="css_07_0024__i1112271513813">&lt;host&gt;</em><strong id="css_07_0024__b899317159817">:</strong><em id="css_07_0024__i063311171484">&lt;port&gt;</em><strong id="css_07_0024__b1247320181687">,</strong><em id="css_07_0024__i24512201183">&lt;host&gt;</em><strong id="css_07_0024__b142171821087">:</strong><em id="css_07_0024__i151942231816">&lt;port&gt;</em>.<p id="css_07_0024__p192655612424">If the cluster has only one node, the IP address and port number of only one node are displayed, for example, <strong id="css_07_0024__b169617452337">10.62.179.32:9200</strong>. If the cluster has multiple nodes, the IP addresses and port numbers of all nodes are displayed, for example, <strong id="css_07_0024__b18438550203310">10.62.179.32:9200,10.62.179.33:9200</strong>.</p>
</li></ol>
</li><li id="css_07_0024__li78671114175115"><a name="css_07_0024__li78671114175115"></a><a name="li78671114175115"></a><strong id="css_07_0024__b01265711255">Obtaining the security certificate of a CSS cluster</strong><ol id="css_07_0024__ol950510531194"><li id="css_07_0024__en-us_topic_0000001111110356_li1798555810535">Log in to the CSS management console.</li><li id="css_07_0024__en-us_topic_0000001111110356_li146461036135810">In the navigation pane, choose <strong id="css_07_0024__en-us_topic_0000001111110356_b1361463413337">Clusters</strong>. The cluster list is displayed.</li><li id="css_07_0024__en-us_topic_0000001111110356_li19708195345420">Click the name of a cluster to go to the cluster details page.</li><li id="css_07_0024__en-us_topic_0000001111110356_li426111575415">On the <strong id="css_07_0024__en-us_topic_0000001111110356_b52752041164216">Configuration</strong> page, click <strong id="css_07_0024__en-us_topic_0000001111110356_b117539447428">Download Certificate</strong> next to <strong id="css_07_0024__en-us_topic_0000001111110356_b0503981813">HTTPS Access</strong>.</li></ol>
</li></ul>
</div>
</div>
View Git Blame Copy Permalink