vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0387.html
Su, Xiaomeng 04d4597cf3 dli_sqlreference_0511_version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2023-11-02 14:34:08 +00:00

237 lines
27 KiB
HTML
Raw Blame History

<a name="dli_08_0387"></a><a name="dli_08_0387"></a>
<h1 class="topictitle1">MySQL CDC Source Table</h1>
<div id="body8662426"><div class="section" id="dli_08_0387__en-us_topic_0000001262815682_section11231720174018"><h4 class="sectiontitle">Function</h4><p id="dli_08_0387__en-us_topic_0000001262815682_p12954023204019">The MySQL CDC source table, that is, the MySQL streaming source table, reads all historical data in the database first and then smoothly switches data read to the Binlog to ensure data integrity.</p>
</div>
<div class="section" id="dli_08_0387__en-us_topic_0000001262815682_section153771755204816"><h4 class="sectiontitle">Prerequisites</h4><ul id="dli_08_0387__en-us_topic_0000001262815682_ul1094501134920"><li id="dli_08_0387__en-us_topic_0000001262815682_li1585110112719">MySQL CDC requires MySQL 5.7 or 8.0.<em id="dli_08_0387__en-us_topic_0000001262815682_i1752523643418">x</em>.</li><li id="dli_08_0387__en-us_topic_0000001262815682_li1817918345355">An enhanced datasource connection has been created for DLI to connect to the MySQL database, so that you can configure security group rules as required.
</li><li id="dli_08_0387__en-us_topic_0000001262815682_li5583230184910">Binlog is enabled for MySQL, and <strong id="dli_08_0387__en-us_topic_0000001262815682_b341825920437">binlog_row_image</strong> is set to <strong id="dli_08_0387__en-us_topic_0000001262815682_b115431166444">FULL</strong>.</li><li id="dli_08_0387__en-us_topic_0000001262815682_li106194349204">A MySQL user has been created and granted the <strong id="dli_08_0387__en-us_topic_0000001262815682_b13369181720442">SELECT</strong>, <strong id="dli_08_0387__en-us_topic_0000001262815682_b1869092034419">SHOW DATABASES</strong>, <strong id="dli_08_0387__en-us_topic_0000001262815682_b1643924114417">REPLICATION SLAVE</strong>, and <strong id="dli_08_0387__en-us_topic_0000001262815682_b79138278445">REPLICATION CLIENT</strong> permissions.</li></ul>
</div>
<div class="section" id="dli_08_0387__en-us_topic_0000001262815682_section178291144918"><h4 class="sectiontitle">Precautions</h4><ul id="dli_08_0387__en-us_topic_0000001262815682_ul13107843165817"><li id="dli_08_0387__en-us_topic_0000001262815682_li377716472316">When creating a Flink OpenSource SQL job, you need to set <strong id="dli_08_0387__en-us_topic_0000001262815682_b154171236144415">Flink Version</strong> to <strong id="dli_08_0387__en-us_topic_0000001262815682_b1141733674419">1.12</strong> on the <strong id="dli_08_0387__en-us_topic_0000001262815682_b11418153644412">Running Parameters</strong> tab of the job editing page, select <strong id="dli_08_0387__en-us_topic_0000001262815682_b134181536154419">Save Job Log</strong>, and set the OBS bucket for saving job logs.</li><li id="dli_08_0387__en-us_topic_0000001262815682_li16650612171616">Each client that synchronizes database data has a unique ID, that is, the server ID. You are advised to configure a unique server ID for each MySQL CDC job in the same database.<div class="p" id="dli_08_0387__en-us_topic_0000001262815682_p629715133166"><a name="dli_08_0387__en-us_topic_0000001262815682_li16650612171616"></a><a name="en-us_topic_0000001262815682_li16650612171616"></a>Main reasons are as follows:<ul id="dli_08_0387__en-us_topic_0000001262815682_ul15843142016157"><li id="dli_08_0387__en-us_topic_0000001262815682_li4300121211156">The MySQL server maintains the network connection and Binlog location based on the ID. Therefore, if a large number of clients with the same server ID connect to the MySQL server, the CPU usage of the MySQL server may increase sharply, affecting the stability of online services.</li><li id="dli_08_0387__en-us_topic_0000001262815682_li49641923153">If multiple jobs share the same server ID, Binlog locations will be disordered, making data read inaccurate. Therefore, you are advised to configure different server IDs for each MySQL CDC job.</li></ul>
</div>
</li></ul>
<ul id="dli_08_0387__en-us_topic_0000001262815682_ul14851049162012"><li id="dli_08_0387__en-us_topic_0000001262815682_li4851649102016">Watermarks cannot be defined for MySQL CDC source tables. For details about window aggregation, see <a href="#dli_08_0387__en-us_topic_0000001262815682_section09412772314">FAQ</a>.</li><li id="dli_08_0387__en-us_topic_0000001262815682_li1554215505510">If you connect to a sink source that supports upsert, such as GaussDB(DWS) and MySQL, you need to define the primary key in the statement for creating the sink table. For details, see the printSink table creation statement in <a href="#dli_08_0387__en-us_topic_0000001262815682_section2787228135313">Example</a>.</li></ul>
</div>
<div class="section" id="dli_08_0387__en-us_topic_0000001262815682_section1332752476"><h4 class="sectiontitle">Syntax</h4><pre class="screen" id="dli_08_0387__en-us_topic_0000001262815682_screen1996932684815">create table mySqlCdcSource (
attr_name attr_type
(',' attr_name attr_type)*
(','PRIMARY KEY (attr_name, ...) NOT ENFORCED)
)
with (
'connector' = 'mysql-cdc',
'hostname' = '<em id="dli_08_0387__en-us_topic_0000001262815682_i9899205202018">mysqlHostname</em>',
'username' = '<em id="dli_08_0387__en-us_topic_0000001262815682_i16692205614208">mysql</em><em id="dli_08_0387__en-us_topic_0000001262815682_i4215102142114">Username</em>',
'password' = '<em id="dli_08_0387__en-us_topic_0000001262815682_i16620711122116">mysql</em><em id="dli_08_0387__en-us_topic_0000001262815682_i7743151782119">Password</em>',
'database-name' = '<em id="dli_08_0387__en-us_topic_0000001262815682_i1518061412217">mysql</em><em id="dli_08_0387__en-us_topic_0000001262815682_i81921720172118">DatabaseName</em>',
'table-name' = '<em id="dli_08_0387__en-us_topic_0000001262815682_i16457111517215">mysql</em><em id="dli_08_0387__en-us_topic_0000001262815682_i62705226211">TableName</em>'
);</pre>
</div>
<div class="section" id="dli_08_0387__en-us_topic_0000001262815682_section53607322481"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0387__en-us_topic_0000001262815682_table849825425517" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dli_08_0387__en-us_topic_0000001262815682_row5498185405511"><th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.5.2.2.6.1.1"><p id="dli_08_0387__en-us_topic_0000001262815682_p349812548553">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="7.86%" id="mcps1.3.5.2.2.6.1.2"><p id="dli_08_0387__en-us_topic_0000001262815682_p144981154105515">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="8.38%" id="mcps1.3.5.2.2.6.1.3"><p id="dli_08_0387__en-us_topic_0000001262815682_p1749810542553">Default Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="6.49%" id="mcps1.3.5.2.2.6.1.4"><p id="dli_08_0387__en-us_topic_0000001262815682_p14981154165516">Data Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="57.269999999999996%" id="mcps1.3.5.2.2.6.1.5"><p id="dli_08_0387__en-us_topic_0000001262815682_p34982544552">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0387__en-us_topic_0000001262815682_row19498105445515"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0387__en-us_topic_0000001262815682_p049865435510">connector</p>
</td>
<td class="cellrowborder" valign="top" width="7.86%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0387__en-us_topic_0000001262815682_p19498125414555">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="8.38%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0387__en-us_topic_0000001262815682_p449875418553">None</p>
</td>
<td class="cellrowborder" valign="top" width="6.49%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0387__en-us_topic_0000001262815682_p14983548556">String</p>
</td>
<td class="cellrowborder" valign="top" width="57.269999999999996%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0387__en-us_topic_0000001262815682_p1526613419233">Connector to be used. Set this parameter to <strong id="dli_08_0387__en-us_topic_0000001262815682_b31876288538">mysql-cdc</strong>.</p>
</td>
</tr>
<tr id="dli_08_0387__en-us_topic_0000001262815682_row5498145455517"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0387__en-us_topic_0000001262815682_p84983549553">hostname</p>
</td>
<td class="cellrowborder" valign="top" width="7.86%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0387__en-us_topic_0000001262815682_p849815425518">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="8.38%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0387__en-us_topic_0000001262815682_p7441133135910">None</p>
</td>
<td class="cellrowborder" valign="top" width="6.49%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0387__en-us_topic_0000001262815682_p1637124413594">String</p>
</td>
<td class="cellrowborder" valign="top" width="57.269999999999996%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0387__en-us_topic_0000001262815682_p1049817540551">IP address or hostname of the MySQL database.</p>
</td>
</tr>
<tr id="dli_08_0387__en-us_topic_0000001262815682_row19498205445512"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0387__en-us_topic_0000001262815682_p84986542551">username</p>
</td>
<td class="cellrowborder" valign="top" width="7.86%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0387__en-us_topic_0000001262815682_p10498125413552">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="8.38%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0387__en-us_topic_0000001262815682_p149711833205911">None</p>
</td>
<td class="cellrowborder" valign="top" width="6.49%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0387__en-us_topic_0000001262815682_p779394425912">String</p>
</td>
<td class="cellrowborder" valign="top" width="57.269999999999996%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0387__en-us_topic_0000001262815682_p1349895413553">Username of the MySQL database.</p>
</td>
</tr>
<tr id="dli_08_0387__en-us_topic_0000001262815682_row1249820545550"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0387__en-us_topic_0000001262815682_p4498254185515">password</p>
</td>
<td class="cellrowborder" valign="top" width="7.86%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0387__en-us_topic_0000001262815682_p5498135420559">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="8.38%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0387__en-us_topic_0000001262815682_p1431154391910">None</p>
</td>
<td class="cellrowborder" valign="top" width="6.49%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0387__en-us_topic_0000001262815682_p13503164512597">String</p>
</td>
<td class="cellrowborder" valign="top" width="57.269999999999996%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0387__en-us_topic_0000001262815682_p1049845455517">Password of the MySQL database.</p>
</td>
</tr>
<tr id="dli_08_0387__en-us_topic_0000001262815682_row1749835425514"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0387__en-us_topic_0000001262815682_p134981454165518">database-name</p>
</td>
<td class="cellrowborder" valign="top" width="7.86%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0387__en-us_topic_0000001262815682_p74981554135512">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="8.38%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0387__en-us_topic_0000001262815682_p85511935165915">None</p>
</td>
<td class="cellrowborder" valign="top" width="6.49%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0387__en-us_topic_0000001262815682_p9250124620591">String</p>
</td>
<td class="cellrowborder" valign="top" width="57.269999999999996%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0387__en-us_topic_0000001262815682_p1349845475516">Name of the database to connect.</p>
<p id="dli_08_0387__en-us_topic_0000001262815682_p971482505">The database name supports regular expressions to read data from multiple databases. For example, <strong id="dli_08_0387__en-us_topic_0000001262815682_b85772405554">flink(.)*</strong> indicates all database names starting with <strong id="dli_08_0387__en-us_topic_0000001262815682_b88144552555">flink</strong>.</p>
</td>
</tr>
<tr id="dli_08_0387__en-us_topic_0000001262815682_row649895420554"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0387__en-us_topic_0000001262815682_p154981054105512">table-name</p>
</td>
<td class="cellrowborder" valign="top" width="7.86%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0387__en-us_topic_0000001262815682_p249835435515">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="8.38%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0387__en-us_topic_0000001262815682_p1321233675918">None</p>
</td>
<td class="cellrowborder" valign="top" width="6.49%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0387__en-us_topic_0000001262815682_p1521647155913">String</p>
</td>
<td class="cellrowborder" valign="top" width="57.269999999999996%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0387__en-us_topic_0000001262815682_p124987542557">Name of the table to read data from.</p>
<p id="dli_08_0387__en-us_topic_0000001262815682_p157221090010">The table name supports regular expressions to read data from multiple tables. For example, <strong id="dli_08_0387__en-us_topic_0000001262815682_b550104314565">cdc_order(.)*</strong> indicates all table names starting with <strong id="dli_08_0387__en-us_topic_0000001262815682_b97311733578">cdc_order</strong>.</p>
</td>
</tr>
<tr id="dli_08_0387__en-us_topic_0000001262815682_row0498454205517"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0387__en-us_topic_0000001262815682_p54987546551">port</p>
</td>
<td class="cellrowborder" valign="top" width="7.86%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0387__en-us_topic_0000001262815682_p44983541557">No</p>
</td>
<td class="cellrowborder" valign="top" width="8.38%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0387__en-us_topic_0000001262815682_p549875485510">3306</p>
</td>
<td class="cellrowborder" valign="top" width="6.49%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0387__en-us_topic_0000001262815682_p4498135475510">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="57.269999999999996%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0387__en-us_topic_0000001262815682_p12498195405519">Port number of the MySQL database.</p>
</td>
</tr>
<tr id="dli_08_0387__en-us_topic_0000001262815682_row1149825405513"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0387__en-us_topic_0000001262815682_p649945412556">server-id</p>
</td>
<td class="cellrowborder" valign="top" width="7.86%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0387__en-us_topic_0000001262815682_p049975412553">No</p>
</td>
<td class="cellrowborder" valign="top" width="8.38%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0387__en-us_topic_0000001262815682_p11499105413557">A random value from 5400 to 6000</p>
</td>
<td class="cellrowborder" valign="top" width="6.49%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0387__en-us_topic_0000001262815682_p1069411548597">String</p>
</td>
<td class="cellrowborder" valign="top" width="57.269999999999996%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0387__en-us_topic_0000001262815682_p161214277291">A numeric ID of the database client, which must be globally unique in the MySQL cluster. You are advised to set a unique ID for each job in the same database.</p>
<p id="dli_08_0387__en-us_topic_0000001262815682_p3194135610274">By default, a random value ranging from 5400 to 6400 is generated.</p>
</td>
</tr>
<tr id="dli_08_0387__en-us_topic_0000001262815682_row1749918543558"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0387__en-us_topic_0000001262815682_p1549975419557">scan.startup.mode</p>
</td>
<td class="cellrowborder" valign="top" width="7.86%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0387__en-us_topic_0000001262815682_p19499145435512">No</p>
</td>
<td class="cellrowborder" valign="top" width="8.38%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0387__en-us_topic_0000001262815682_p11499454105513">initial</p>
</td>
<td class="cellrowborder" valign="top" width="6.49%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0387__en-us_topic_0000001262815682_p118561016107">String</p>
</td>
<td class="cellrowborder" valign="top" width="57.269999999999996%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0387__en-us_topic_0000001262815682_p049945425517">Startup mode for consuming data.</p>
<ul id="dli_08_0387__en-us_topic_0000001262815682_ul12323714301"><li id="dli_08_0387__en-us_topic_0000001262815682_li2321713018"><strong id="dli_08_0387__en-us_topic_0000001262815682_b2073912101461">initial</strong> (default): In the first startup, the database scans all historical data and then reads the latest Binlog data.</li><li id="dli_08_0387__en-us_topic_0000001262815682_li9321793013"><strong id="dli_08_0387__en-us_topic_0000001262815682_b1499710526715">latest-offset</strong>: In the first startup, the database reads data directly from the end of the Binlog (the latest Binlog) instead of scanning all historical data. That is, it reads only the latest changes after the connector is started.</li></ul>
</td>
</tr>
<tr id="dli_08_0387__en-us_topic_0000001262815682_row6499154185515"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0387__en-us_topic_0000001262815682_p1549917548554">server-time-zone</p>
</td>
<td class="cellrowborder" valign="top" width="7.86%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0387__en-us_topic_0000001262815682_p449975465513">No</p>
</td>
<td class="cellrowborder" valign="top" width="8.38%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0387__en-us_topic_0000001262815682_p1499165455510">None</p>
</td>
<td class="cellrowborder" valign="top" width="6.49%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0387__en-us_topic_0000001262815682_p1877118178011">String</p>
</td>
<td class="cellrowborder" valign="top" width="57.269999999999996%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0387__en-us_topic_0000001262815682_p15583531203013">Time zone of the session used by the database.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0387__en-us_topic_0000001262815682_section2787228135313"><a name="dli_08_0387__en-us_topic_0000001262815682_section2787228135313"></a><a name="en-us_topic_0000001262815682_section2787228135313"></a><h4 class="sectiontitle">Example</h4><p id="dli_08_0387__en-us_topic_0000001262815682_p1251263213252">In this example, MySQL-CDC is used to read data from RDS for MySQL in real time and write the data to the Print result table. The procedure is as follows (MySQL 5.7.32 is used in this example):</p>
<ol id="dli_08_0387__en-us_topic_0000001262815682_ol5185446103217"><li id="dli_08_0387__en-us_topic_0000001262815682_li6127741153217">Create an enhanced datasource connection in the VPC and subnet where MySQL locates, and bind the connection to the required Flink elastic resource pool.</li><li id="dli_08_0387__en-us_topic_0000001262815682_li1233002211514">Set MySQL security groups and add inbound rules to allow access from the Flink queue. Test the connectivity using the MySQL address. If the connection is successful, the datasource is bound to the queue. Otherwise, the binding fails.</li><li id="dli_08_0387__en-us_topic_0000001262815682_li4185164615322">Create a table named <strong id="dli_08_0387__en-us_topic_0000001262815682_b36517166128">cdc_order</strong> in database <strong id="dli_08_0387__en-us_topic_0000001262815682_b99243323123">flink</strong> of the MySQL database.<pre class="screen" id="dli_08_0387__en-us_topic_0000001262815682_screen9883103014217">CREATE TABLE `flink`.`cdc_order` (
`order_id` VARCHAR(32) NOT NULL,
`order_channel` VARCHAR(32) NULL,
`order_time` VARCHAR(32) NULL,
`pay_amount` DOUBLE NULL,
`real_pay` DOUBLE NULL,
`pay_time` VARCHAR(32) NULL,
`user_id` VARCHAR(32) NULL,
`user_name` VARCHAR(32) NULL,
`area_id` VARCHAR(32) NULL,
PRIMARY KEY (`order_id`)
) ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8mb4
COLLATE = utf8mb4_general_ci;</pre>
</li><li id="dli_08_0387__en-us_topic_0000001262815682_li133558543514">Create a Flink OpenSource SQL job. Enter the following job script and submit the job.<div class="p" id="dli_08_0387__en-us_topic_0000001262815682_p145941442173519"><a name="dli_08_0387__en-us_topic_0000001262815682_li133558543514"></a><a name="en-us_topic_0000001262815682_li133558543514"></a>When you create a job, set <strong id="dli_08_0387__en-us_topic_0000001262815682_b1960931691314">Flink Version</strong> to <strong id="dli_08_0387__en-us_topic_0000001262815682_b17609131611134">1.12</strong> on the <strong id="dli_08_0387__en-us_topic_0000001262815682_b1361011611135">Running Parameters</strong> tab. Select <strong id="dli_08_0387__en-us_topic_0000001262815682_b9610131617138">Save Job Log</strong>, and specify the OBS bucket for saving job logs. <strong id="dli_08_0387__en-us_topic_0000001262815682_b130132020131">Change the values of the parameters in bold as needed in the following script.</strong><pre class="screen" id="dli_08_0387__en-us_topic_0000001262815682_screen2639193553516">create table mysqlCdcSource(
order_id string,
order_channel string,
order_time string,
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id STRING
) with (
'connector' = 'mysql-cdc',
'hostname' = '<em id="dli_08_0387__en-us_topic_0000001262815682_i92917211378"><strong id="dli_08_0387__en-us_topic_0000001262815682_b2145153618377">mysqlHostname</strong></em>',
'username' = '<em id="dli_08_0387__en-us_topic_0000001262815682_i172911921153711"><strong id="dli_08_0387__en-us_topic_0000001262815682_b39821238163714">mysql</strong></em><em id="dli_08_0387__en-us_topic_0000001262815682_i20291102133718"><strong id="dli_08_0387__en-us_topic_0000001262815682_b1298363883717">Username</strong></em>',
'password' = '<em id="dli_08_0387__en-us_topic_0000001262815682_i22911021123715"><strong id="dli_08_0387__en-us_topic_0000001262815682_b1486894053719">mysql</strong></em><em id="dli_08_0387__en-us_topic_0000001262815682_i429162114377"><strong id="dli_08_0387__en-us_topic_0000001262815682_b886834018372">Password</strong></em>',
'database-name' = '<em id="dli_08_0387__en-us_topic_0000001262815682_i172912213371"><strong id="dli_08_0387__en-us_topic_0000001262815682_b15822184214378">mysql</strong></em><em id="dli_08_0387__en-us_topic_0000001262815682_i82911621193714"><strong id="dli_08_0387__en-us_topic_0000001262815682_b1822114214376">DatabaseName</strong></em>',
'table-name' = '<em id="dli_08_0387__en-us_topic_0000001262815682_i152917210374"><strong id="dli_08_0387__en-us_topic_0000001262815682_b343720458375">mysql</strong></em><em id="dli_08_0387__en-us_topic_0000001262815682_i15291142118372"><strong id="dli_08_0387__en-us_topic_0000001262815682_b24371745133713">TableName</strong></em>'
);
create table printSink(
order_id string,
order_channel string,
order_time string,
pay_amount double,
real_pay double,
pay_time string,
user_id string,
user_name string,
area_id STRING,
primary key(order_id) not enforced
) with (
'connector' = 'print'
);
insert into printSink select * from mysqlCdcSource;</pre>
</div>
</li><li id="dli_08_0387__en-us_topic_0000001262815682_li18185194619329">Insert test data in MySQL.<pre class="screen" id="dli_08_0387__en-us_topic_0000001262815682_screen11849141112415">insert into cdc_order values
('202103241000000001','webShop','2021-03-24 10:00:00','100.00','100.00','2021-03-24 10:02:03','0001','Alice','330106'),
('202103241606060001','appShop','2021-03-24 16:06:06','200.00','180.00','2021-03-24 16:10:06','0001','Alice','330106');
delete from cdc_order where order_channel = 'webShop';
insert into cdc_order values('202103251202020001','miniAppShop','2021-03-25 12:02:02','60.00','60.00','2021-03-25 12:03:00','0002','Bob','330110');</pre>
</li><li id="dli_08_0387__en-us_topic_0000001262815682_li10185946183216">Perform the following operations to view the data result in the <strong id="dli_08_0387__en-us_topic_0000001262815682_b181351269146">taskmanager.out</strong> file:<ol type="a" id="dli_08_0387__en-us_topic_0000001262815682_ol864115198285"><li id="dli_08_0387__en-us_topic_0000001262815682_li10901621122819">Log in to the DLI console. In the navigation pane, choose <strong id="dli_08_0387__en-us_topic_0000001262815682_b466037131410">Job Management</strong> &gt; <strong id="dli_08_0387__en-us_topic_0000001262815682_b1866015781414">Flink Jobs</strong>.</li><li id="dli_08_0387__en-us_topic_0000001262815682_li1912163912282">Click the name of the corresponding Flink job, choose <strong id="dli_08_0387__en-us_topic_0000001262815682_b193768991410">Run Log</strong>, click <strong id="dli_08_0387__en-us_topic_0000001262815682_b73761493143">OBS Bucket</strong>, and locate the folder of the log you want to view according to the date.</li><li id="dli_08_0387__en-us_topic_0000001262815682_li0641191914285">Go to the folder of the date, find the folder whose name contains <strong id="dli_08_0387__en-us_topic_0000001262815682_b02651811161416">taskmanager</strong>, download the <strong id="dli_08_0387__en-us_topic_0000001262815682_b142651311201416">taskmanager.out</strong> file, and view result logs.</li></ol>
<p id="dli_08_0387__en-us_topic_0000001262815682_p1282118785912">The data result is as follows:</p>
<pre class="screen" id="dli_08_0387__en-us_topic_0000001262815682_screen17252518182418">+I(202103241000000001,webShop,2021-03-2410:00:00,100.0,100.0,2021-03-2410:02:03,0001,Alice,330106)
+I(202103241606060001,appShop,2021-03-2416:06:06,200.0,180.0,2021-03-2416:10:06,0001,Alice,330106)
-D(202103241000000001,webShop,2021-03-2410:00:00,100.0,100.0,2021-03-2410:02:03,0001,Alice,330106)
+I(202103251202020001,miniAppShop,2021-03-2512:02:02,60.0,60.0,2021-03-2512:03:00,0002,Bob,330110)</pre>
</li></ol>
</div>
<div class="section" id="dli_08_0387__en-us_topic_0000001262815682_section09412772314"><a name="dli_08_0387__en-us_topic_0000001262815682_section09412772314"></a><a name="en-us_topic_0000001262815682_section09412772314"></a><h4 class="sectiontitle">FAQ</h4><p id="dli_08_0387__en-us_topic_0000001262815682_p83831313395">Q: How do I perform window aggregation if the MySQL CDC source table does not support definition of watermarks?</p>
<p id="dli_08_0387__en-us_topic_0000001262815682_p33213195213">A: You can use the non-window aggregation method. That is, convert the time field into a window value, and then use <strong id="dli_08_0387__en-us_topic_0000001262815682_b144034619155">GROUP BY</strong> to perform aggregation based on the window value.</p>
<p id="dli_08_0387__en-us_topic_0000001262815682_p4218115919240">For example, you can use the following script to collect statistics on the number of orders per minute (<strong id="dli_08_0387__en-us_topic_0000001262815682_b766081710204">order_time</strong> indicates the order time, in the string format):</p>
<pre class="screen" id="dli_08_0387__en-us_topic_0000001262815682_screen16550125612260">insert into printSink select DATE_FORMAT(order_time, 'yyyy-MM-dd HH:mm'), count(*) from mysqlCdcSource group by DATE_FORMAT(order_time, 'yyyy-MM-dd HH:mm');</pre>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0381.html">Creating Source Tables</a></div>
</div>
</div>
View Git Blame Copy Permalink