vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0235.html
Su, Xiaomeng 76a5b1ee83 dli_sqlreference_20240227 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-03-27 22:02:33 +00:00

388 lines
40 KiB
HTML
Raw Blame History

<a name="dli_08_0235"></a><a name="dli_08_0235"></a>
<h1 class="topictitle1">DIS Source Stream</h1>
<div id="body1574393062459"><div class="section" id="dli_08_0235__en-us_topic_0111501791_section10885512285"><h4 class="sectiontitle">Function</h4><p id="dli_08_0235__en-us_topic_0111501791_p994529151211">Create a source stream to read data from DIS. DIS accesses user data and Flink job reads data from the DIS stream as input data for jobs. Flink jobs can quickly remove data from producers using DIS source sources for continuous processing. Flink jobs are applicable to scenarios where data outside the cloud service is imported to the cloud service for filtering, real-time analysis, monitoring reports, and dumping.</p>
<p id="dli_08_0235__en-us_topic_0111501791_p675193523118">DIS addresses the challenge of transmitting data outside cloud services to cloud services. DIS builds data intake streams for custom applications capable of processing or analyzing streaming data. DIS continuously captures, transmits, and stores terabytes of data from hundreds of thousands of sources every hour, such as logs, Internet of Things (IoT) data, social media feeds, website clickstreams, and location-tracking events. For more information about DIS, see the <em id="dli_08_0235__i1551785682011">Data Ingestion Service User Guide</em>.</p>
</div>
<div class="section" id="dli_08_0235__en-us_topic_0111501791_section197181028112213"><h4 class="sectiontitle">Syntax</h4><pre class="screen" id="dli_08_0235__screen13771820174811">CREATE SOURCE STREAM stream_id (attr_name attr_type (',' attr_name attr_type)* )
WITH (
type = "dis",
region = "",
channel = "",
partition_count = "",
encode = "",
field_delimiter = "",
offset= "");</pre>
</div>
<div class="section" id="dli_08_0235__section14382135111170"><h4 class="sectiontitle">Keywords</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0235__en-us_topic_0111501791_table2060911914818" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Keywords</caption><thead align="left"><tr id="dli_08_0235__en-us_topic_0111501791_row1960910913480"><th align="left" class="cellrowborder" valign="top" width="15.212121212121213%" id="mcps1.3.3.2.2.4.1.1"><p id="dli_08_0235__en-us_topic_0111501791_p176092984816">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="9.090909090909092%" id="mcps1.3.3.2.2.4.1.2"><p id="dli_08_0235__en-us_topic_0111501791_p03846300421">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="75.6969696969697%" id="mcps1.3.3.2.2.4.1.3"><p id="dli_08_0235__en-us_topic_0111501791_p136107924817">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0235__en-us_topic_0111501791_row2061019914488"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p17610179154820">type</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p6385193044210">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p4610209124812">Data source type. <span class="parmvalue" id="dli_08_0235__en-us_topic_0111501791_parmvalue12345115174717"><b>dis</b></span> indicates that the data source is DIS.</p>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row166109917484"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p5610096487">region</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p53856301429">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p206108919489">Region where DIS for storing the data is located.</p>
</td>
</tr>
<tr id="dli_08_0235__row1152541173820"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__p111537416387">ak</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__p161531941193814">No</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__p141531741143816">Access Key ID (AK). </p>
</td>
</tr>
<tr id="dli_08_0235__row2089817363388"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__p58994360384">sk</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__p20899163615388">No</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__p1989953683812">Specifies the secret access key used together with the ID of the access key. </p>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row461018984810"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p06101291481">channel</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p4385230164216">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p46108913484">Name of the DIS stream where data is located.</p>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row1761020910489"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p0610498487">partition_count</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p23851530174214">No</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p76101297484">Number of partitions of the DIS stream where data is located. This parameter and <strong id="dli_08_0235__en-us_topic_0111501791_b842352706224256">partition_range</strong> cannot be configured at the same time. If this parameter is not specified, data of all partitions is read by default.</p>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row63401157191116"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p133421757181115">partition_range</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p143421657201111">No</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p434225741113">Range of partitions of a DIS stream, data in which is ingested by the DLI job. This parameter and <strong id="dli_08_0235__b1215054141">partition_count</strong> cannot be configured at the same time. If this parameter is not specified, data of all partitions is read by default.</p>
<p id="dli_08_0235__en-us_topic_0111501791_p1698251245">If you set this parameter to <strong id="dli_08_0235__b152952014161716">[0:2]</strong>, data will be read from partitions 1, 2, and 3.</p>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row186106919480"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p176109994813">encode</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p338511305428">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p1061013918482">Data encoding format. The value can be <span class="parmvalue" id="dli_08_0235__parmvalue4944185943416"><b>csv</b></span>, <span class="parmvalue" id="dli_08_0235__parmvalue1494518591342"><b>json</b></span>, <span class="parmvalue" id="dli_08_0235__parmvalue2945559103413"><b>xml</b></span>, <span class="parmvalue" id="dli_08_0235__parmvalue129451359123419"><b>email</b></span>, <span class="parmvalue" id="dli_08_0235__parmvalue13945145917342"><b>blob</b></span>, or <span class="parmvalue" id="dli_08_0235__parmvalue59461859113417"><b>user_defined</b></span>.</p>
<ul id="dli_08_0235__ul206191537152217"><li id="dli_08_0235__en-us_topic_0111501791_li5347114174217"><span class="parmname" id="dli_08_0235__parmname16759101357"><b>field_delimiter</b></span> must be specified if this parameter is set to <span class="parmvalue" id="dli_08_0235__parmvalue20763102357"><b>csv</b></span>.</li><li id="dli_08_0235__en-us_topic_0111501791_li734784174215"><span class="parmname" id="dli_08_0235__parmname94361517113518"><b>json_config</b></span> must be specified if this parameter is set to <span class="parmvalue" id="dli_08_0235__parmvalue84365175353"><b>json</b></span>.</li><li id="dli_08_0235__en-us_topic_0111501791_li43714439293"><span class="parmname" id="dli_08_0235__parmname114912289358"><b>xml_config</b></span> must be specified if this parameter is set to <span class="parmvalue" id="dli_08_0235__parmvalue649928103512"><b>xml</b></span>.</li><li id="dli_08_0235__en-us_topic_0111501791_li469471595115"><span class="parmvalue" id="dli_08_0235__en-us_topic_0111501791_parmvalue1046213020017"><b>email_key</b></span> must be specified if this parameter is set to <span class="parmvalue" id="dli_08_0235__en-us_topic_0111501791_parmvalue2628438809"><b>email</b></span>.</li><li id="dli_08_0235__en-us_topic_0111501791_li1612685512223">If this parameter is set to <span class="parmvalue" id="dli_08_0235__en-us_topic_0111501791_parmvalue1595337957224737"><b>blob</b></span>, the received data is not parsed, only one stream attribute exists, and the data format is ARRAY[TINYINT].</li><li id="dli_08_0235__li36208378225"><span class="parmname" id="dli_08_0235__parmname64141013103612"><b>encode_class_name</b></span> and <span class="parmname" id="dli_08_0235__parmname134151913143616"><b>encode_class_parameter</b></span> must be specified if this parameter is set to <span class="parmvalue" id="dli_08_0235__parmvalue6415161318363"><b>user_defined</b></span>.</li></ul>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row169031311992"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p1511954297">field_delimiter</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p31211343917">No</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p48351921162817">Attribute delimiter. This parameter is mandatory only when the CSV encoding format is used. You can set this parameter, for example, to a comma (,).</p>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row571913182263"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p14728918132620">quote</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p8730131818266">No</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p397121416310">Quoted symbol in a data format. The attribute delimiters between two quoted symbols are treated as common characters.</p>
<ul id="dli_08_0235__en-us_topic_0111501791_ul18631537631"><li id="dli_08_0235__en-us_topic_0111501791_li10631173715314">If double quotation marks are used as the quoted symbol, set this parameter to <strong id="dli_08_0235__b178155813920">\u005c\u0022</strong> for character conversion.</li><li id="dli_08_0235__en-us_topic_0111501791_li1963112378318">If a single quotation mark is used as the quoted symbol, set this parameter to a single quotation mark (').</li></ul>
<div class="note" id="dli_08_0235__en-us_topic_0111501791_note1377013361642"><span class="notetitle"> NOTE: </span><div class="notebody"><ul id="dli_08_0235__ul48541722101518"><li id="dli_08_0235__li18854122212157">Currently, only the CSV format is supported.</li><li id="dli_08_0235__li1685582212154">After this parameter is specified, ensure that each field does not contain quoted symbols or contains an even number of quoted symbols. Otherwise, parsing will fail.</li></ul>
</div></div>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row7708105292812"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p870925212283">json_config</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p170985214289">No</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p12709115262819">When the encoding format is JSON, you need to use this parameter to specify the mapping between JSON fields and stream definition fields. The format is <strong id="dli_08_0235__b861023515395">field1=data_json.field1; field2=data_json.field2; field3=$</strong>, where <strong id="dli_08_0235__b12614153913912">field3=$</strong> indicates that the content of field3 is the entire JSON string.</p>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row17654112817306"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p1654192815302">xml_config</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p1065452814308">No</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p1265542873013">If <strong id="dli_08_0235__en-us_topic_0111501791_b842352706173044">encode</strong> is set to <strong id="dli_08_0235__en-us_topic_0111501791_b842352706173051">xml</strong>, you need to set this parameter to specify the mapping between the xml field and the stream definition field. An example of the format is as follows: field1=data_xml.field1; field2=data_xml.field2.</p>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row629181185312"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p1129113135319">email_key</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p129117125312">No</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p3291181125318">If <strong id="dli_08_0235__en-us_topic_0111501791_b1222662276173415">encode</strong> is set to <strong id="dli_08_0235__en-us_topic_0111501791_b1180834731173415">email</strong>, you need to set the parameter to specify the information to be extracted. You need to list the key values that correspond to stream definition fields. Multiple key values are separated by commas (,), for example, "Message-ID, Date, Subject, body". There is no keyword in the email body and DLI specifies "body" as the keyword.</p>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row5559511111618"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p1588112121610">encode_class_name</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p2881182101617">No</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p68811521201617">If <strong id="dli_08_0235__b9392928104012">encode</strong> is set to <strong id="dli_08_0235__b113949287402">user_defined</strong>, you need to set this parameter to the name of the user-defined decoding class (including the complete package path). The class must inherit the <strong id="dli_08_0235__b2709756194915">DeserializationSchema</strong> class.</p>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row1175141714169"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p8881321191617">encode_class_parameter</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p1988120218166">No</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p198821021151620">If <strong id="dli_08_0235__b1950419421403">encode</strong> is set to <strong id="dli_08_0235__b1450613429408">user_defined</strong>, you can set this parameter to specify the input parameter of the user-defined decoding class. Only one parameter of the string type is supported.</p>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row16811151817209"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p18811201817205">offset</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p18813101872016">No</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><ul id="dli_08_0235__en-us_topic_0111501791_ul347994119719"><li id="dli_08_0235__en-us_topic_0111501791_li13479154120713">If data is imported to the DIS stream after the job is started, this parameter will become invalid.</li><li id="dli_08_0235__en-us_topic_0111501791_li9533366261">If the job is started after data is imported to the DIS stream, you can set the parameter as required.<p id="dli_08_0235__en-us_topic_0111501791_p12812718297"><a name="dli_08_0235__en-us_topic_0111501791_li9533366261"></a><a name="en-us_topic_0111501791_li9533366261"></a>For example, if <strong id="dli_08_0235__en-us_topic_0111501791_b84235270620494">offset</strong> is set to <strong id="dli_08_0235__en-us_topic_0111501791_b842352706204912">100</strong>, DLI starts from the 100th data record in DIS.</p>
</li></ul>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row6461151175414"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p215955720543">start_time</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p71621457175417">No</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p4372123095514">Start time for reading DIS data.</p>
<ul id="dli_08_0235__en-us_topic_0111501791_ul176281319155713"><li id="dli_08_0235__en-us_topic_0111501791_li11628121917578">If this parameter is specified, DLI reads data read from the specified time. The format is <strong id="dli_08_0235__b1795761294018">yyyy-MM-dd HH:mm:ss</strong>.</li><li id="dli_08_0235__en-us_topic_0111501791_li136283195573">If neither <strong id="dli_08_0235__en-us_topic_0111501791_b2082045314257">start_time</strong> nor <strong id="dli_08_0235__en-us_topic_0111501791_b1766471215266">offset</strong> is specified, DLI reads the latest data.</li><li id="dli_08_0235__en-us_topic_0111501791_li1962811191571">If <strong id="dli_08_0235__en-us_topic_0111501791_b133001627162619">start_time</strong> is not specified but <strong id="dli_08_0235__en-us_topic_0111501791_b1106533162612">offset</strong> is specified, DLI reads data from the data record specified by <strong id="dli_08_0235__en-us_topic_0111501791_b1252952672717">offset</strong>.</li></ul>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row28521818301"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p845713281261">enable_checkpoint</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p385161823019">No</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p2851180306">Whether to enable the checkpoint function. The value can be <strong id="dli_08_0235__b17835656132611">true</strong> (enabled) or <strong id="dli_08_0235__b851651152710">false</strong> (disabled). The default value is <strong id="dli_08_0235__b388813362714">false</strong>.</p>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row61176420305"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p1711744211300">checkpoint_app_name</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p1011714215304">No</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p4117442143010">ID of a DIS consumer. If a DIS stream is consumed by different jobs, you need to configure the consumer ID for each job to avoid checkpoint confusion.</p>
</td>
</tr>
<tr id="dli_08_0235__en-us_topic_0111501791_row836713863014"><td class="cellrowborder" valign="top" width="15.212121212121213%" headers="mcps1.3.3.2.2.4.1.1 "><p id="dli_08_0235__en-us_topic_0111501791_p536753823013">checkpoint_interval</p>
</td>
<td class="cellrowborder" valign="top" width="9.090909090909092%" headers="mcps1.3.3.2.2.4.1.2 "><p id="dli_08_0235__en-us_topic_0111501791_p123671938183012">No</p>
</td>
<td class="cellrowborder" valign="top" width="75.6969696969697%" headers="mcps1.3.3.2.2.4.1.3 "><p id="dli_08_0235__en-us_topic_0111501791_p1367173816309">Interval of checkpoint operations on the DIS source operator. The value is in the unit of seconds. The default value is <strong id="dli_08_0235__en-us_topic_0111501791_b15500435182212">60</strong>.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0235__section239404119184"><h4 class="sectiontitle">Precautions</h4><p id="dli_08_0235__p14212154655419">When creating a source stream, you can specify a time model for subsequent calculation. Currently, DLI supports two time models: Processing Time and Event Time. For details about the syntax, see <a href="dli_08_0107.html">Configuring Time Models</a>.</p>
</div>
<div class="section" id="dli_08_0235__section12681313198"><h4 class="sectiontitle">Example</h4><ul id="dli_08_0235__en-us_topic_0111501791_ul1146811614302"><li id="dli_08_0235__en-us_topic_0111501791_li144683168304">In CSV encoding format, DLI reads data from the DIS stream and records it as codes in CSV format. The codes are separated by commas (,).<div class="codecoloring" codetype="Sql" id="dli_08_0235__en-us_topic_0111501791_screen26644102712"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">SOURCE</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">car_infos</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="n">car_id</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_owner</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_age</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span>
<span class="w"> </span><span class="n">average_speed</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span>
<span class="w"> </span><span class="n">total_miles</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_timestamp</span><span class="w"> </span><span class="n">LONG</span>
<span class="p">)</span>
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;dis&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;xxx&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">channel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;dliinput&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">encode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;csv&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">field_delimiter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;,&quot;</span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_08_0235__en-us_topic_0111501791_li638772317301">In JSON encoding format, DLI reads data from the DIS stream and records it as codes in JSON format. For example, {"car":{"car_id":"ZJA710XC", "car_owner":"coco", "car_age":5, "average_speed":80, "total_miles":15000, "car_timestamp":1526438880}}<div class="codecoloring" codetype="Sql" id="dli_08_0235__en-us_topic_0111501791_screen1640454511298"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">SOURCE</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">car_infos</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="n">car_id</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_owner</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_age</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span>
<span class="w"> </span><span class="n">average_speed</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span>
<span class="w"> </span><span class="n">total_miles</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span>
<span class="w"> </span><span class="n">car_timestamp</span><span class="w"> </span><span class="n">LONG</span>
<span class="p">)</span>
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;dis&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;xxx&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">channel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;dliinput&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">encode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;json&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">json_config</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;car_id=car.car_id;car_owner =car.car_owner;car_age=car.car_age;average_speed =car.average_speed ;total_miles=car.total_miles;&quot;</span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_08_0235__en-us_topic_0111501791_li14994115743520">In XML encoding format, DLI reads data from the DIS stream and records it as codes in XML format.<div class="codecoloring" codetype="Sql" id="dli_08_0235__en-us_topic_0111501791_screen599416578359"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">SOURCE</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">person_infos</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="n">pid</span><span class="w"> </span><span class="nb">BIGINT</span><span class="p">,</span>
<span class="w"> </span><span class="n">pname</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">page</span><span class="w"> </span><span class="nb">int</span><span class="p">,</span>
<span class="w"> </span><span class="n">plocation</span><span class="w"> </span><span class="n">STRING</span><span class="p">,</span>
<span class="w"> </span><span class="n">pbir</span><span class="w"> </span><span class="nb">DATE</span><span class="p">,</span>
<span class="w"> </span><span class="n">phealthy</span><span class="w"> </span><span class="nb">BOOLEAN</span><span class="p">,</span>
<span class="w"> </span><span class="n">pgrade</span><span class="w"> </span><span class="nb">ARRAY</span><span class="p">[</span><span class="n">STRING</span><span class="p">]</span>
<span class="p">)</span>
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;dis&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;xxx&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">channel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;dis-dli-input&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">encode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;xml&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">field_delimiter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;,&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">xml_config</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;pid=person.pid;page=person.page;pname=person.pname;plocation=person.plocation;pbir=person.pbir;pgrade=person.pgrade;phealthy=person.phealthy&quot;</span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
<div class="p" id="dli_08_0235__en-us_topic_0111501791_p29951857153511">An example of XML data is as follows:<div class="codecoloring" codetype="Xml" id="dli_08_0235__en-us_topic_0111501791_screen799515753520"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span></pre></div></td><td class="code"><div><pre><span></span><span class="cp">&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot;?&gt;</span>
<span class="nt">&lt;root&gt;</span>
<span class="w"> </span><span class="nt">&lt;person&gt;</span>
<span class="w"> </span><span class="nt">&lt;pid&gt;</span>362305199010025042<span class="nt">&lt;/pid&gt;</span>
<span class="w"> </span><span class="nt">&lt;pname&gt;</span>xiaoming<span class="nt">&lt;/pname&gt;</span>
<span class="w"> </span><span class="nt">&lt;page&gt;</span>28<span class="nt">&lt;/page&gt;</span>
<span class="w"> </span><span class="nt">&lt;plocation&gt;</span>xxx<span class="nt">&lt;/plocation&gt;</span>
<span class="w"> </span><span class="nt">&lt;pbir&gt;</span>1990-10-02<span class="nt">&lt;/pbir&gt;</span>
<span class="w"> </span><span class="nt">&lt;phealthy&gt;</span>true<span class="nt">&lt;/phealthy&gt;</span>
<span class="w"> </span><span class="nt">&lt;pgrade&gt;</span>[A,B,C]<span class="nt">&lt;/pgrade&gt;</span>
<span class="w"> </span><span class="nt">&lt;/person&gt;</span>
<span class="nt">&lt;/root&gt;</span>
</pre></div></td></tr></table></div>
</div>
</div>
</li><li id="dli_08_0235__en-us_topic_0111501791_li966662832714">In EMAIL encoding format, DLI reads data from the DIS stream and records it as a complete Email.<div class="codecoloring" codetype="Sql" id="dli_08_0235__en-us_topic_0111501791_screen820211022313"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">CREATE</span><span class="w"> </span><span class="k">SOURCE</span><span class="w"> </span><span class="n">STREAM</span><span class="w"> </span><span class="n">email_infos</span><span class="w"> </span><span class="p">(</span><span class="w"> </span>
<span class="w"> </span><span class="n">Event_ID</span><span class="w"> </span><span class="n">String</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">Event_Time</span><span class="w"> </span><span class="nb">Date</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">Subject</span><span class="w"> </span><span class="n">String</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">From_Email</span><span class="w"> </span><span class="n">String</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">To_EMAIL</span><span class="w"> </span><span class="n">String</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">CC_EMAIL</span><span class="w"> </span><span class="nb">Array</span><span class="p">[</span><span class="n">String</span><span class="p">],</span><span class="w"> </span>
<span class="w"> </span><span class="n">BCC_EMAIL</span><span class="w"> </span><span class="n">String</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">MessageBody</span><span class="w"> </span><span class="n">String</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">Mime_Version</span><span class="w"> </span><span class="n">String</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">Content_Type</span><span class="w"> </span><span class="n">String</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">charset</span><span class="w"> </span><span class="n">String</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">Content_Transfer_Encoding</span><span class="w"> </span><span class="n">String</span>
<span class="p">)</span>
<span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span><span class="w"> </span>
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;dis&quot;</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">region</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;xxx&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="n">channel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;dliinput&quot;</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">encode</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;email&quot;</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">email_key</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ss">&quot;Message-ID, Date, Subject, From, To, CC, BCC, Body, Mime-Version, Content-Type, charset, Content_Transfer_Encoding&quot;</span><span class="w"> </span>
<span class="p">);</span>
</pre></div></td></tr></table></div>
</div>
<p id="dli_08_0235__en-us_topic_0111501791_p10303175412407">An example of email data is as follows:</p>
<pre class="screen" id="dli_08_0235__en-us_topic_0111501791_screen10972135518425">Message-ID: &lt;200906291839032504254@sample.com&gt;
Date: Fri, 11 May 2001 09:54:00 -0700 (PDT)
From: zhangsan@sample.com
To: lisi@sample.com, wangwu@sample.com
Subject: "Hello World"
Cc: lilei@sample.com, hanmei@sample.com
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Bcc: jack@sample.com, lily@sample.com
X-From: Zhang San
X-To: Li Si, Wang Wu
X-cc: Li Lei, Han Mei
X-bcc:
X-Folder: \Li_Si_June2001\Notes Folders\Notes inbox
X-Origin: Lucy
X-FileName: sample.nsf
Dear Associate / Analyst Committee:
Hello World!
Thank you,
Associate / Analyst Program
zhangsan</pre>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0234.html">Creating a Source Stream</a></div>
</div>
</div>
View Git Blame Copy Permalink