forked from docs/doc-exports
Su, Xiaomeng
04d4597cf3
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
160 lines
15 KiB
HTML
160 lines
15 KiB
HTML
<a name="dli_08_0412"></a><a name="dli_08_0412"></a>
|
|
|
|
<h1 class="topictitle1">Debezium</h1>
|
|
<div id="body8662426"><div class="section" id="dli_08_0412__en-us_topic_0000001310215797_section9536315265"><h4 class="sectiontitle">Function</h4><p id="dli_08_0412__en-us_topic_0000001310215797_p198981723152716">Debezium is a Changelog Data Capture (CDC) tool that can stream changes in real-time from other databases into Kafka. Debezium provides a unified format schema for changelog and supports to serialize messages using JSON.</p>
|
|
<p id="dli_08_0412__en-us_topic_0000001310215797_p18898182310271">Flink supports to interpret Debezium JSON and Avro messages as INSERT/UPDATE/DELETE messages into Flink SQL system. This is useful in many cases to leverage this feature, such as:</p>
|
|
<ul id="dli_08_0412__en-us_topic_0000001310215797_ul78987231278"><li id="dli_08_0412__en-us_topic_0000001310215797_li789842318272">synchronizing incremental data from databases to other systems</li><li id="dli_08_0412__en-us_topic_0000001310215797_li14898172315279">Auditing logs</li><li id="dli_08_0412__en-us_topic_0000001310215797_li2898162382712">Real-time materialized view on databases</li><li id="dli_08_0412__en-us_topic_0000001310215797_li17898192317271">Temporal join changing history of a database table, etc.</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_08_0412__en-us_topic_0000001310215797_section139091342182616"><h4 class="sectiontitle">Parameters</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0412__en-us_topic_0000001310215797_table1035231193019" frame="border" border="1" rules="all"><caption><b>Table 1 </b></caption><thead align="left"><tr id="dli_08_0412__en-us_topic_0000001310215797_row53583111302"><th align="left" class="cellrowborder" valign="top" width="19.36%" id="mcps1.3.2.2.2.6.1.1"><p id="dli_08_0412__en-us_topic_0000001310215797_p10359314305">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="10.33%" id="mcps1.3.2.2.2.6.1.2"><p id="dli_08_0412__en-us_topic_0000001310215797_p23573111308">Mandatory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="10.530000000000001%" id="mcps1.3.2.2.2.6.1.3"><p id="dli_08_0412__en-us_topic_0000001310215797_p193513119305">Default Value</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="10%" id="mcps1.3.2.2.2.6.1.4"><p id="dli_08_0412__en-us_topic_0000001310215797_p1535123143012">Mandatory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="49.78%" id="mcps1.3.2.2.2.6.1.5"><p id="dli_08_0412__en-us_topic_0000001310215797_p153563115305">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_08_0412__en-us_topic_0000001310215797_row637153110300"><td class="cellrowborder" valign="top" width="19.36%" headers="mcps1.3.2.2.2.6.1.1 "><p id="dli_08_0412__en-us_topic_0000001310215797_p47931610316">format</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.33%" headers="mcps1.3.2.2.2.6.1.2 "><p id="dli_08_0412__en-us_topic_0000001310215797_p47931653111">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.530000000000001%" headers="mcps1.3.2.2.2.6.1.3 "><p id="dli_08_0412__en-us_topic_0000001310215797_p1279131610311">None</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10%" headers="mcps1.3.2.2.2.6.1.4 "><p id="dli_08_0412__en-us_topic_0000001310215797_p167915168318">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.78%" headers="mcps1.3.2.2.2.6.1.5 "><p id="dli_08_0412__en-us_topic_0000001310215797_p7791916153116">Format to be used. In this example.Set this parameter to <strong id="dli_08_0412__en-us_topic_0000001310215797_b6442031114">debezium-json</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0412__en-us_topic_0000001310215797_row193716316300"><td class="cellrowborder" valign="top" width="19.36%" headers="mcps1.3.2.2.2.6.1.1 "><p id="dli_08_0412__en-us_topic_0000001310215797_p197917163312">debezium-json.schema-include</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.33%" headers="mcps1.3.2.2.2.6.1.2 "><p id="dli_08_0412__en-us_topic_0000001310215797_p18019165318">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.530000000000001%" headers="mcps1.3.2.2.2.6.1.3 "><p id="dli_08_0412__en-us_topic_0000001310215797_p12801716163112">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10%" headers="mcps1.3.2.2.2.6.1.4 "><p id="dli_08_0412__en-us_topic_0000001310215797_p148018167310">Boolean</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.78%" headers="mcps1.3.2.2.2.6.1.5 "><p id="dli_08_0412__en-us_topic_0000001310215797_p118061610319">Whether the Debezium JSON messages contain the schema. When setting up Debezium Kafka Connect, enable the Kafka configuration <strong id="dli_08_0412__en-us_topic_0000001310215797_b11957134101814">value.converter.schemas.enable</strong> to include the schema in the message.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0412__en-us_topic_0000001310215797_row737113193015"><td class="cellrowborder" valign="top" width="19.36%" headers="mcps1.3.2.2.2.6.1.1 "><p id="dli_08_0412__en-us_topic_0000001310215797_p1380816133113">debezium-json.ignore-parse-errors</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.33%" headers="mcps1.3.2.2.2.6.1.2 "><p id="dli_08_0412__en-us_topic_0000001310215797_p180171653110">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.530000000000001%" headers="mcps1.3.2.2.2.6.1.3 "><p id="dli_08_0412__en-us_topic_0000001310215797_p18807164314">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10%" headers="mcps1.3.2.2.2.6.1.4 "><p id="dli_08_0412__en-us_topic_0000001310215797_p380191623118">Boolean</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.78%" headers="mcps1.3.2.2.2.6.1.5 "><p id="dli_08_0412__en-us_topic_0000001310215797_p178016166318">Whether fields and rows with parse errors will be skipped or failed. The default value is <strong id="dli_08_0412__en-us_topic_0000001310215797_b92001959201817">false</strong>, indicating that an error will be thrown. Fields are set to null in case of errors.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0412__en-us_topic_0000001310215797_row137193119302"><td class="cellrowborder" valign="top" width="19.36%" headers="mcps1.3.2.2.2.6.1.1 "><p id="dli_08_0412__en-us_topic_0000001310215797_p118021663115">debezium-json.timestamp-format.standard</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.33%" headers="mcps1.3.2.2.2.6.1.2 "><p id="dli_08_0412__en-us_topic_0000001310215797_p780416113118">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.530000000000001%" headers="mcps1.3.2.2.2.6.1.3 "><p id="dli_08_0412__en-us_topic_0000001310215797_p145917357318">'SQL'</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10%" headers="mcps1.3.2.2.2.6.1.4 "><p id="dli_08_0412__en-us_topic_0000001310215797_p188031693112">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.78%" headers="mcps1.3.2.2.2.6.1.5 "><p id="dli_08_0412__en-us_topic_0000001310215797_p4804161319">Input and output timestamp formats. Currently supported values are <strong id="dli_08_0412__en-us_topic_0000001310215797_b8898618154512">SQL</strong> and <strong id="dli_08_0412__en-us_topic_0000001310215797_b28152184516">ISO-8601</strong>.</p>
|
|
<ul id="dli_08_0412__en-us_topic_0000001310215797_ul180111610318"><li id="dli_08_0412__en-us_topic_0000001310215797_li1980316183115"><strong id="dli_08_0412__en-us_topic_0000001310215797_b13659216465">SQL</strong> will parse input timestamp in "yyyy-MM-dd HH:mm:ss.s{precision}" format, for example <strong id="dli_08_0412__en-us_topic_0000001310215797_b458389174615">2020-12-30 12:13:14.123</strong> and output timestamp in the same format.</li><li id="dli_08_0412__en-us_topic_0000001310215797_li148031613310"><strong id="dli_08_0412__en-us_topic_0000001310215797_b13797191264610">ISO-8601</strong> will parse input timestamp in "yyyy-MM-ddTHH:mm:ss.s{precision}" format, for example <strong id="dli_08_0412__en-us_topic_0000001310215797_b1441514433466">2020-12-30T12:13:14.123</strong> and output timestamp in the same format.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0412__en-us_topic_0000001310215797_row7371331123018"><td class="cellrowborder" valign="top" width="19.36%" headers="mcps1.3.2.2.2.6.1.1 "><p id="dli_08_0412__en-us_topic_0000001310215797_p68021611319">debezium-json.map-null-key.mode</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.33%" headers="mcps1.3.2.2.2.6.1.2 "><p id="dli_08_0412__en-us_topic_0000001310215797_p188012162312">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.530000000000001%" headers="mcps1.3.2.2.2.6.1.3 "><p id="dli_08_0412__en-us_topic_0000001310215797_p854182293112">'FAIL'</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10%" headers="mcps1.3.2.2.2.6.1.4 "><p id="dli_08_0412__en-us_topic_0000001310215797_p180316193117">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.78%" headers="mcps1.3.2.2.2.6.1.5 "><p id="dli_08_0412__en-us_topic_0000001310215797_p1380816153118">Handling mode when serializing null keys for map data. Available values are as follows:</p>
|
|
<ul id="dli_08_0412__en-us_topic_0000001310215797_ul18801161314"><li id="dli_08_0412__en-us_topic_0000001310215797_li118051612311"><strong id="dli_08_0412__en-us_topic_0000001310215797_b19997124812463">FAIL</strong> will throw exception when encountering map value with null key.</li><li id="dli_08_0412__en-us_topic_0000001310215797_li68013168312"><strong id="dli_08_0412__en-us_topic_0000001310215797_b5444550184613">DROP</strong> will drop null key entries for map data.</li><li id="dli_08_0412__en-us_topic_0000001310215797_li14800166315"><strong id="dli_08_0412__en-us_topic_0000001310215797_b12100529461">LITERAL</strong> replaces the empty key value in the map with a string constant. The string literal is defined by <strong id="dli_08_0412__en-us_topic_0000001310215797_b6554165717465">debezium-json.map-null-key.literal</strong> option.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0412__en-us_topic_0000001310215797_row19372031113018"><td class="cellrowborder" valign="top" width="19.36%" headers="mcps1.3.2.2.2.6.1.1 "><p id="dli_08_0412__en-us_topic_0000001310215797_p280616133115">debezium-json.map-null-key.literal</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.33%" headers="mcps1.3.2.2.2.6.1.2 "><p id="dli_08_0412__en-us_topic_0000001310215797_p08091619310">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10.530000000000001%" headers="mcps1.3.2.2.2.6.1.3 "><p id="dli_08_0412__en-us_topic_0000001310215797_p15801416123111">'null'</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="10%" headers="mcps1.3.2.2.2.6.1.4 "><p id="dli_08_0412__en-us_topic_0000001310215797_p1180316153115">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.78%" headers="mcps1.3.2.2.2.6.1.5 "><p id="dli_08_0412__en-us_topic_0000001310215797_p128010161317">String literal to replace null key when <strong id="dli_08_0412__en-us_topic_0000001310215797_b17367819112119">debezium-json.map-null-key.mode</strong> is <strong id="dli_08_0412__en-us_topic_0000001310215797_b537418198219">LITERAL</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="dli_08_0412__en-us_topic_0000001310215797_section193656483268"><h4 class="sectiontitle">Supported Connectors</h4><ul id="dli_08_0412__en-us_topic_0000001310215797_ul1118274610361"><li id="dli_08_0412__en-us_topic_0000001310215797_li13182154663618">Kafka</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_08_0412__en-us_topic_0000001310215797_section241815222713"><h4 class="sectiontitle">Example</h4><p id="dli_08_0412__en-us_topic_0000001310215797_p15881132116016">Use Kafka to send data and output the data to print.</p>
|
|
<ol id="dli_08_0412__en-us_topic_0000001310215797_ol192013231710"><li id="dli_08_0412__en-us_topic_0000001310215797_li04031578234"><span>Create a datasource connection for the communication with the VPC and subnet where Kafka locates and bind the connection to the queue. Set a security group and inbound rule to allow access of the queue and test the connectivity of the queue using the Kafka IP address. For example, locate a general-purpose queue where the job runs and choose <strong id="dli_08_0412__en-us_topic_0000001310215797_b13234164520212">More</strong> > <strong id="dli_08_0412__en-us_topic_0000001310215797_b9235164552110">Test Address Connectivity</strong> in the <strong id="dli_08_0412__en-us_topic_0000001310215797_b13235174592112">Operation</strong> column. If the connection is successful, the datasource is bound to the queue. Otherwise, the binding fails.</span></li><li id="dli_08_0412__en-us_topic_0000001310215797_li18401164741714"><span>Create a Flink OpenSource SQL job. Copy the following statement and submit the job:</span><p><pre class="screen" id="dli_08_0412__en-us_topic_0000001310215797_screen6126183681810">create table kafkaSource(
|
|
id BIGINT,
|
|
name STRING,
|
|
description STRING,
|
|
weight DECIMAL(10, 2)
|
|
) with (
|
|
'connector' = 'kafka',
|
|
'topic' = '<yourTopic>',
|
|
'properties.group.id' = '<yourGroupId>',
|
|
'properties.bootstrap.servers' = '<yourKafkaAddress>:<yourKafkaPort>',
|
|
'scan.startup.mode' = 'latest-offset',
|
|
'format' = 'debezium-json'
|
|
);
|
|
create table printSink(
|
|
id BIGINT,
|
|
name STRING,
|
|
description STRING,
|
|
weight DECIMAL(10, 2)
|
|
) with (
|
|
'connector' = 'print'
|
|
);
|
|
insert into printSink select * from kafkaSource;</pre>
|
|
</p></li><li id="dli_08_0412__en-us_topic_0000001310215797_li1285084111812"><span>Insert the following data to the corresponding topic in Kafka:</span><p><pre class="screen" id="dli_08_0412__en-us_topic_0000001310215797_screen1525315566188">{
|
|
"before": {
|
|
"id": 111,
|
|
"name": "scooter",
|
|
"description": "Big 2-wheel scooter",
|
|
"weight": 5.18
|
|
},
|
|
"after": {
|
|
"id": 111,
|
|
"name": "scooter",
|
|
"description": "Big 2-wheel scooter",
|
|
"weight": 5.15
|
|
},
|
|
"source": {
|
|
"version": "0.9.5.Final",
|
|
"connector": "mysql",
|
|
"name": "fullfillment",
|
|
"server_id" :1,
|
|
"ts_sec": 1629607909,
|
|
"gtid": "mysql-bin.000001",
|
|
"pos": 2238,"row": 0,
|
|
"snapshot": false,
|
|
"thread": 7,
|
|
"db": "inventory",
|
|
"table": "test",
|
|
"query": null},
|
|
"op": "u",
|
|
"ts_ms": 1589362330904,
|
|
"transaction": null
|
|
}</pre>
|
|
</p></li><li id="dli_08_0412__en-us_topic_0000001310215797_li4353143193117"><span>View the output through either of the following methods:</span><p><ul id="dli_08_0412__en-us_topic_0000001310215797_ul173531443113114"><li id="dli_08_0412__en-us_topic_0000001310215797_li735314312316">Method 1: Locate the job and click <strong id="dli_08_0412__en-us_topic_0000001310215797_b16649190202220">More</strong> > <strong id="dli_08_0412__en-us_topic_0000001310215797_b126500032210">FlinkUI</strong>. Choose <strong id="dli_08_0412__en-us_topic_0000001310215797_b26501506223">Task Managers</strong> > <strong id="dli_08_0412__en-us_topic_0000001310215797_b1650180202220">Stdout</strong>.</li><li id="dli_08_0412__en-us_topic_0000001310215797_li33538437315">Method 2: If you allow DLI to save job logs in OBS, view the output in the <strong id="dli_08_0412__en-us_topic_0000001310215797_b640811282218">taskmanager.out</strong> file.</li></ul>
|
|
<pre class="screen" id="dli_08_0412__en-us_topic_0000001310215797_screen184391521191911">-U(111,scooter,Big2-wheel scooter,5.18)
|
|
+U(111,scooter,Big2-wheel scooter,5.15)</pre>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0407.html">Format</a></div>
|
|
</div>
|
|
</div>
|
|
|