doc-exports/docs/dli/sqlreference/dli_08_0415.html

<a name="dli_08_0415"></a><a name="dli_08_0415"></a>

<h1 class="topictitle1">Raw</h1>
<div id="body8662426"><div class="section" id="dli_08_0415__en-us_topic_0000001262495798_section21131854311"><h4 class="sectiontitle">Function</h4><p id="dli_08_0415__en-us_topic_0000001262495798_p1106133684412">The raw format allows you to read and write raw (byte based) values as a single column.</p>
<p id="dli_08_0415__en-us_topic_0000001262495798_p151061836104415">Note: This format encodes null values as <strong id="dli_08_0415__en-us_topic_0000001262495798_b417623604814">null</strong> of the <strong id="dli_08_0415__en-us_topic_0000001262495798_b2597102514815">byte[]</strong> type. This may have limitation when used in <strong id="dli_08_0415__en-us_topic_0000001262495798_b12560183065017">upsert-kafka</strong>, because <strong id="dli_08_0415__en-us_topic_0000001262495798_b56212033125012">upsert-kafka</strong> treats null values as a tombstone message (DELETE on the key). Therefore, we recommend avoiding using <strong id="dli_08_0415__en-us_topic_0000001262495798_b10791740125014">upsert-kafka</strong> connector and the <strong id="dli_08_0415__en-us_topic_0000001262495798_b1517704945018">raw</strong> format as a <strong id="dli_08_0415__en-us_topic_0000001262495798_b182375218500">value.format</strong> if the field can have a null value.</p>
<p id="dli_08_0415__en-us_topic_0000001262495798_p1110623620443">The raw format connector is built-in, no additional dependencies are required.</p>
</div>
<div class="section" id="dli_08_0415__en-us_topic_0000001262495798_section861233454311"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0415__en-us_topic_0000001262495798_table9325840115315" frame="border" border="1" rules="all"><caption><b>Table 1 </b></caption><thead align="left"><tr id="dli_08_0415__en-us_topic_0000001262495798_row1732694010539"><th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.2.2.2.6.1.1"><p id="dli_08_0415__en-us_topic_0000001262495798_p93267409538">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.2.2.2.6.1.2"><p id="dli_08_0415__en-us_topic_0000001262495798_p173261640155311">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.2.2.2.6.1.3"><p id="dli_08_0415__en-us_topic_0000001262495798_p10341171813543">Default Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.2.2.2.6.1.4"><p id="dli_08_0415__en-us_topic_0000001262495798_p1032617403532">Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="20%" id="mcps1.3.2.2.2.6.1.5"><p id="dli_08_0415__en-us_topic_0000001262495798_p053368175414">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0415__en-us_topic_0000001262495798_row123261240135316"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.2.2.2.6.1.1 "><p id="dli_08_0415__en-us_topic_0000001262495798_p1660815540549">format</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.2.2.2.6.1.2 "><p id="dli_08_0415__en-us_topic_0000001262495798_p7608854165417">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.2.2.2.6.1.3 "><p id="dli_08_0415__en-us_topic_0000001262495798_p2608954125415">None</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.2.2.2.6.1.4 "><p id="dli_08_0415__en-us_topic_0000001262495798_p86085545542">String</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.2.2.2.6.1.5 "><p id="dli_08_0415__en-us_topic_0000001262495798_p156081554105413">Format to be used. Set this parameter to <strong id="dli_08_0415__en-us_topic_0000001262495798_b194972128524">raw</strong>.</p>
</td>
</tr>
<tr id="dli_08_0415__en-us_topic_0000001262495798_row14326114075311"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.2.2.2.6.1.1 "><p id="dli_08_0415__en-us_topic_0000001262495798_p1608195435416">raw.charset</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.2.2.2.6.1.2 "><p id="dli_08_0415__en-us_topic_0000001262495798_p660985455411">No</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.2.2.2.6.1.3 "><p id="dli_08_0415__en-us_topic_0000001262495798_p1160905485412">UTF-8</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.2.2.2.6.1.4 "><p id="dli_08_0415__en-us_topic_0000001262495798_p176091354145410">String</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.2.2.2.6.1.5 "><p id="dli_08_0415__en-us_topic_0000001262495798_p126091954205414">Charset to encode the text string.</p>
</td>
</tr>
<tr id="dli_08_0415__en-us_topic_0000001262495798_row13326194014531"><td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.2.2.2.6.1.1 "><p id="dli_08_0415__en-us_topic_0000001262495798_p060985485415">raw.endianness</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.2.2.2.6.1.2 "><p id="dli_08_0415__en-us_topic_0000001262495798_p17609254165418">No</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.2.2.2.6.1.3 "><p id="dli_08_0415__en-us_topic_0000001262495798_p460995413547">big-endian</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.2.2.2.6.1.4 "><p id="dli_08_0415__en-us_topic_0000001262495798_p20609175415544">String</p>
</td>
<td class="cellrowborder" valign="top" width="20%" headers="mcps1.3.2.2.2.6.1.5 "><p id="dli_08_0415__en-us_topic_0000001262495798_p19609145415543">Endianness to encode the bytes of numeric value. Valid values are <strong id="dli_08_0415__en-us_topic_0000001262495798_b191534417545">big-endian</strong> and <strong id="dli_08_0415__en-us_topic_0000001262495798_b1350245419535">little-endian</strong>. You can search for endianness for more details.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0415__en-us_topic_0000001262495798_section5642124014435"><h4 class="sectiontitle">Supported Connectors</h4><ul id="dli_08_0415__en-us_topic_0000001262495798_ul11011025135618"><li id="dli_08_0415__en-us_topic_0000001262495798_li20101625145614">Kafka</li><li id="dli_08_0415__en-us_topic_0000001262495798_li1525193016569">UpsertKafka</li></ul>
</div>
<div class="section" id="dli_08_0415__en-us_topic_0000001262495798_section2958175584318"><h4 class="sectiontitle">Example</h4><p id="dli_08_0415__en-us_topic_0000001262495798_p1375334113113">Use Kafka to send data and output the data to print.</p>
<ol id="dli_08_0415__en-us_topic_0000001262495798_ol14231744133117"><li id="dli_08_0415__en-us_topic_0000001262495798_li04031578234"><span>Create a datasource connection for the communication with the VPC and subnet where Kafka locates and bind the connection to the queue. Set a security group and inbound rule to allow access of the queue and test the connectivity of the queue using the Kafka IP address. For example, locate a general-purpose queue where the job runs and choose <strong id="dli_08_0415__en-us_topic_0000001262495798_b18680102314542">More</strong> &gt; <strong id="dli_08_0415__en-us_topic_0000001262495798_b3681923115416">Test Address Connectivity</strong> in the <strong id="dli_08_0415__en-us_topic_0000001262495798_b66811123105411">Operation</strong> column. If the connection is successful, the datasource is bound to the queue. Otherwise, the binding fails.</span></li><li id="dli_08_0415__en-us_topic_0000001262495798_li1599913011242"><span>Create a Flink OpenSource SQL job and select Flink 1.12. Copy the following statement and submit the job:</span><p><pre class="screen" id="dli_08_0415__en-us_topic_0000001262495798_screen525011213328">create table kafkaSource(
  log string
  ) with (
    'connector' = 'kafka',
    'topic' = '&lt;yourTopic&gt;',
    'properties.group.id' = '&lt;yourGroupId&gt;',
    'properties.bootstrap.servers' = '&lt;yourKafkaAddress&gt;:&lt;yourKafkaPort&gt;',
    'scan.startup.mode' = 'latest-offset',
    'format' = 'raw'
);
create table printSink(
  log string
   ) with (
     'connector' = 'print'
   );
insert into printSink select * from kafkaSource;</pre>
</p></li><li id="dli_08_0415__en-us_topic_0000001262495798_li18839185317311"><span>Insert the following data to the corresponding topic in Kafka:</span><p><pre class="screen" id="dli_08_0415__en-us_topic_0000001262495798_screen56532035203216">47.29.201.179 - - [28/Feb/2019:13:17:10 +0000] "GET /?p=1 HTTP/2.0" 200 5316 "https://domain.com/?p=1" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75"</pre>
</p></li><li id="dli_08_0415__en-us_topic_0000001262495798_li4353143193117"><span>View the output through either of the following methods:</span><p><ul id="dli_08_0415__en-us_topic_0000001262495798_ul173531443113114"><li id="dli_08_0415__en-us_topic_0000001262495798_li735314312316">Method 1: Locate the job and click <strong id="dli_08_0415__en-us_topic_0000001262495798_b16891337145418">More</strong> &gt; <strong id="dli_08_0415__en-us_topic_0000001262495798_b1289837115411">FlinkUI</strong>. Choose <strong id="dli_08_0415__en-us_topic_0000001262495798_b790237195415">Task Managers</strong> &gt; <strong id="dli_08_0415__en-us_topic_0000001262495798_b89020377541">Stdout</strong>.</li><li id="dli_08_0415__en-us_topic_0000001262495798_li22773498322">Method 2: If you allow DLI to save job logs in OBS, view the output in the <strong id="dli_08_0415__en-us_topic_0000001262495798_b1977833915413">taskmanager.out</strong> file.</li></ul>
<pre class="screen" id="dli_08_0415__en-us_topic_0000001262495798_screen13510115814327">+I(47.29.201.179 - - [28/Feb/2019:13:17:10 +0000] "GET /?p=1 HTTP/2.0"2005316"https://domain.com/?p=1"
"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36" "2.75")</pre>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0407.html">Format</a></div>
</div>
</div>