doc-exports/docs/dli/sqlreference/dli_08_0414.html

<a name="dli_08_0414"></a><a name="dli_08_0414"></a>

<h1 class="topictitle1">Maxwell</h1>
<div id="body8662426"><div class="section" id="dli_08_0414__en-us_topic_0000001262655742_section14475651181811"><h4 class="sectiontitle">Function</h4><p id="dli_08_0414__en-us_topic_0000001262655742_p18251753182412">Flink supports to interpret Maxwell JSON messages as INSERT/UPDATE/DELETE messages into Flink SQL system. This is useful in many cases to leverage this feature, </p>
</div>
<p id="dli_08_0414__en-us_topic_0000001262655742_p8060118">such as:</p>
<ul id="dli_08_0414__en-us_topic_0000001262655742_ul3512142202111"><li id="dli_08_0414__en-us_topic_0000001262655742_li1251213420218">Synchronizing incremental data from databases to other systems</li><li id="dli_08_0414__en-us_topic_0000001262655742_li1962814913223">Auditing logs</li><li id="dli_08_0414__en-us_topic_0000001262655742_li9368520122211">Real-time materialized views on databases</li><li id="dli_08_0414__en-us_topic_0000001262655742_li463062914229">Temporal join changing history of a database table and so on</li></ul>
<p id="dli_08_0414__en-us_topic_0000001262655742_p2093610424228">Flink also supports to encode the INSERT/UPDATE/DELETE messages in Flink SQL as Maxwell JSON messages, and emit to external systems like Kafka. However, currently Flink cannot combine UPDATE_BEFORE and UPDATE_AFTER into a single UPDATE message. Therefore, Flink encodes UPDATE_BEFORE and UDPATE_AFTER as DELETE and INSERT Maxwell messages.</p>
<div class="section" id="dli_08_0414__en-us_topic_0000001262655742_section146400512257"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0414__en-us_topic_0000001262655742_table19357124612615" frame="border" border="1" rules="all"><caption><b>Table 1 </b></caption><thead align="left"><tr id="dli_08_0414__en-us_topic_0000001262655742_row535894682615"><th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.5.2.2.6.1.1"><p id="dli_08_0414__en-us_topic_0000001262655742_p935817463265">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="7.89%" id="mcps1.3.5.2.2.6.1.2"><p id="dli_08_0414__en-us_topic_0000001262655742_p10358124617269">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="9.01%" id="mcps1.3.5.2.2.6.1.3"><p id="dli_08_0414__en-us_topic_0000001262655742_p14358184611265">Default Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="10.32%" id="mcps1.3.5.2.2.6.1.4"><p id="dli_08_0414__en-us_topic_0000001262655742_p5358144612264">Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="52.78%" id="mcps1.3.5.2.2.6.1.5"><p id="dli_08_0414__en-us_topic_0000001262655742_p1235816469264">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0414__en-us_topic_0000001262655742_row1635874616261"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0414__en-us_topic_0000001262655742_p935874612610">format</p>
</td>
<td class="cellrowborder" valign="top" width="7.89%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0414__en-us_topic_0000001262655742_p6358546192616">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="9.01%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0414__en-us_topic_0000001262655742_p435810461269">None</p>
</td>
<td class="cellrowborder" valign="top" width="10.32%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0414__en-us_topic_0000001262655742_p435819465261">String</p>
</td>
<td class="cellrowborder" valign="top" width="52.78%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0414__en-us_topic_0000001262655742_p5139513182815">Format to be used. Set this parameter to <strong id="dli_08_0414__en-us_topic_0000001262655742_b105341632123818">maxwell-json</strong>.</p>
</td>
</tr>
<tr id="dli_08_0414__en-us_topic_0000001262655742_row735804619266"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0414__en-us_topic_0000001262655742_p7358124672619">maxwell-json.ignore-parse-errors</p>
</td>
<td class="cellrowborder" valign="top" width="7.89%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0414__en-us_topic_0000001262655742_p18358146132614">No</p>
</td>
<td class="cellrowborder" valign="top" width="9.01%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0414__en-us_topic_0000001262655742_p13358114617268">false</p>
</td>
<td class="cellrowborder" valign="top" width="10.32%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0414__en-us_topic_0000001262655742_p1135854632616">Boolean</p>
</td>
<td class="cellrowborder" valign="top" width="52.78%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0414__en-us_topic_0000001262655742_p19358946172617">Whether fields and rows with parse errors will be skipped or failed. Fields are set to null in case of errors.</p>
</td>
</tr>
<tr id="dli_08_0414__en-us_topic_0000001262655742_row53581246162612"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0414__en-us_topic_0000001262655742_p9358546122617">maxwell-json.timestamp-format.standard</p>
</td>
<td class="cellrowborder" valign="top" width="7.89%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0414__en-us_topic_0000001262655742_p193581046162612">No</p>
</td>
<td class="cellrowborder" valign="top" width="9.01%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0414__en-us_topic_0000001262655742_p0358946102616">'SQL'</p>
</td>
<td class="cellrowborder" valign="top" width="10.32%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0414__en-us_topic_0000001262655742_p173581346172613">String</p>
</td>
<td class="cellrowborder" valign="top" width="52.78%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0414__en-us_topic_0000001262655742_p1730217154519">Input and output timestamp formats. Currently supported values are <strong id="dli_08_0414__en-us_topic_0000001262655742_b1030277114511">SQL</strong> and <strong id="dli_08_0414__en-us_topic_0000001262655742_b183028784520">ISO-8601</strong>: </p>
<p id="dli_08_0414__en-us_topic_0000001262655742_p583116954515"><strong id="dli_08_0414__en-us_topic_0000001262655742_b12830109124514">SQL</strong> will parse input timestamp in "yyyy-MM-dd HH:mm:ss.s{precision}" format, for example, <strong id="dli_08_0414__en-us_topic_0000001262655742_b9830791459">2020-12-30 12:13:14.123</strong> and output timestamp in the same format.</p>
<p id="dli_08_0414__en-us_topic_0000001262655742_p1435884614265"><strong id="dli_08_0414__en-us_topic_0000001262655742_b017285513410">ISO-8601</strong> will parse input timestamp in "yyyy-MM-ddTHH:mm:ss.s{precision}" format, for example <strong id="dli_08_0414__en-us_topic_0000001262655742_b47818813429">2020-12-30T12:13:14.123</strong> and output timestamp in the same format.</p>
</td>
</tr>
<tr id="dli_08_0414__en-us_topic_0000001262655742_row143588467263"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0414__en-us_topic_0000001262655742_p1835817462261">maxwell-json.map-null-key.mode</p>
</td>
<td class="cellrowborder" valign="top" width="7.89%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0414__en-us_topic_0000001262655742_p1359194682613">No</p>
</td>
<td class="cellrowborder" valign="top" width="9.01%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0414__en-us_topic_0000001262655742_p5359746172610">'FAIL'</p>
</td>
<td class="cellrowborder" valign="top" width="10.32%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0414__en-us_topic_0000001262655742_p63591346172618">String</p>
</td>
<td class="cellrowborder" valign="top" width="52.78%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0414__en-us_topic_0000001262655742_p14622023114510">Handling mode when serializing null keys for map data. Currently supported values are 'FAIL', 'DROP' and 'LITERAL': </p>
<p id="dli_08_0414__en-us_topic_0000001262655742_p20478624194518"><strong id="dli_08_0414__en-us_topic_0000001262655742_b194781224184520">FAIL</strong> will throw exception when encountering map with null key.</p>
<p id="dli_08_0414__en-us_topic_0000001262655742_p35914263458"><strong id="dli_08_0414__en-us_topic_0000001262655742_b1759182644520">DROP</strong> will drop null key entries for map data.</p>
<p id="dli_08_0414__en-us_topic_0000001262655742_p7729836193912"><strong id="dli_08_0414__en-us_topic_0000001262655742_b359425734317">LITERAL</strong> will replace null key with string literal. The string literal is defined by <strong id="dli_08_0414__en-us_topic_0000001262655742_b7115513134410">maxwell-json.map-null-key.literal</strong> option.</p>
</td>
</tr>
<tr id="dli_08_0414__en-us_topic_0000001262655742_row2035964612618"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0414__en-us_topic_0000001262655742_p15359134615268">maxwell-json.map-null-key.literal</p>
</td>
<td class="cellrowborder" valign="top" width="7.89%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0414__en-us_topic_0000001262655742_p7359144613263">No</p>
</td>
<td class="cellrowborder" valign="top" width="9.01%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0414__en-us_topic_0000001262655742_p1135974610263">'null'</p>
</td>
<td class="cellrowborder" valign="top" width="10.32%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0414__en-us_topic_0000001262655742_p1935911462265">String</p>
</td>
<td class="cellrowborder" valign="top" width="52.78%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0414__en-us_topic_0000001262655742_p93591846102612">String literal to replace null key when <strong id="dli_08_0414__en-us_topic_0000001262655742_b219511904610">maxwell-json.map-null-key.mode</strong> is <strong id="dli_08_0414__en-us_topic_0000001262655742_b16796131094615">LITERAL</strong>.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0414__en-us_topic_0000001262655742_section813135324020"><h4 class="sectiontitle">Supported Connectors</h4><ul id="dli_08_0414__en-us_topic_0000001262655742_ul4477192194417"><li id="dli_08_0414__en-us_topic_0000001262655742_li247702194418">Kafka</li></ul>
</div>
<div class="section" id="dli_08_0414__en-us_topic_0000001262655742_section9958949174411"><h4 class="sectiontitle">Example</h4><p id="dli_08_0414__en-us_topic_0000001262655742_p15881132116016">Use Kafka to send data and output the data to print.</p>
<ol id="dli_08_0414__en-us_topic_0000001262655742_ol8219141522813"><li id="dli_08_0414__en-us_topic_0000001262655742_li04031578234"><span>Create a datasource connection for the communication with the VPC and subnet where Kafka locates and bind the connection to the queue. Set a security group and inbound rule to allow access of the queue and test the connectivity of the queue using the Kafka IP address. For example, locate a general-purpose queue where the job runs and choose <strong id="dli_08_0414__en-us_topic_0000001262655742_b10422184516463">More</strong> &gt; <strong id="dli_08_0414__en-us_topic_0000001262655742_b0423164513462">Test Address Connectivity</strong> in the <strong id="dli_08_0414__en-us_topic_0000001262655742_b19423945104610">Operation</strong> column. If the connection is successful, the datasource is bound to the queue. Otherwise, the binding fails.</span></li><li id="dli_08_0414__en-us_topic_0000001262655742_li8898724152820"><span>Create a Flink OpenSource SQL job and select Flink 1.12. Copy the following statement and submit the job:</span><p><pre class="screen" id="dli_08_0414__en-us_topic_0000001262655742_screen13511156292">create table kafkaSource(
  id bigint,
  name string,
  description string,
  weight DECIMAL(10, 2)  
  ) with (
    'connector' = 'kafka',
    'topic' = '&lt;yourTopic&gt;',
    'properties.group.id' = '&lt;yourGroupId&gt;',
    'properties.bootstrap.servers' = '&lt;yourKafkaAddress1&gt;:&lt;yourKafkaPort&gt;,&lt;yourKafkaAddress2&gt;:&lt;yourKafkaPort&gt;',
    'scan.startup.mode' = 'latest-offset',
    'format' = 'maxwell-json'
);
create table printSink(
  id bigint,
  name string,
  description string,
  weight DECIMAL(10, 2)
   ) with (
     'connector' = 'print'
   );
insert into printSink select * from kafkaSource;</pre>
</p></li><li id="dli_08_0414__en-us_topic_0000001262655742_li1467893242816"><span>Insert the following data to the corresponding topic in Kafka:</span><p><pre class="screen" id="dli_08_0414__en-us_topic_0000001262655742_screen91771126153016">{
   "database":"test",
   "table":"e",
   "type":"insert",
   "ts":1477053217,
   "xid":23396,
   "commit":true,
   "position":"master.000006:800911",
   "server_id":23042,
   "thread_id":108,
   "primary_key": [1, "2016-10-21 05:33:37.523000"],
   "primary_key_columns": ["id", "c"],
   "data":{
     "id":111,
     "name":"scooter",
     "description":"Big 2-wheel scooter",
     "weight":5.15
   },
   "old":{
     "weight":5.18
   }
}</pre>
</p></li><li id="dli_08_0414__en-us_topic_0000001262655742_li4353143193117"><span>View the output through either of the following methods:</span><p><ul id="dli_08_0414__en-us_topic_0000001262655742_ul173531443113114"><li id="dli_08_0414__en-us_topic_0000001262655742_li735314312316">Method 1: Locate the job and click <strong id="dli_08_0414__en-us_topic_0000001262655742_b1459260104713">More</strong> &gt; <strong id="dli_08_0414__en-us_topic_0000001262655742_b45921602475">FlinkUI</strong>. Choose <strong id="dli_08_0414__en-us_topic_0000001262655742_b559250144710">Task Managers</strong> &gt; <strong id="dli_08_0414__en-us_topic_0000001262655742_b759230204712">Stdout</strong>.</li><li id="dli_08_0414__en-us_topic_0000001262655742_li33538437315">Method 2: If you allow DLI to save job logs in OBS, view the output in the <strong id="dli_08_0414__en-us_topic_0000001262655742_b20999191124715">taskmanager.out</strong> file.</li></ul>
<pre class="screen" id="dli_08_0414__en-us_topic_0000001262655742_screen1268774014308">+I(111,scooter,Big 2-wheel scooter,5.15)</pre>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0407.html">Format</a></div>
</div>
</div>