vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/sqlreference/dli_08_0382.html
Su, Xiaomeng 04d4597cf3 dli_sqlreference_0511_version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2023-11-02 14:34:08 +00:00

164 lines
20 KiB
HTML
Raw Blame History

<a name="dli_08_0382"></a><a name="dli_08_0382"></a>
<h1 class="topictitle1">DataGen Source Table</h1>
<div id="body8662426"><div class="section" id="dli_08_0382__en-us_topic_0000001262655726_section1598811329411"><h4 class="sectiontitle">Function</h4><p id="dli_08_0382__en-us_topic_0000001262655726_p138701039965">DataGen is used to generate random data for debugging and testing.</p>
</div>
<div class="section" id="dli_08_0382__en-us_topic_0000001262655726_section1085044117520"><h4 class="sectiontitle">Prerequisites</h4><p id="dli_08_0382__en-us_topic_0000001262655726_p131812023485">None</p>
</div>
<div class="section" id="dli_08_0382__en-us_topic_0000001262655726_section17666546857"><h4 class="sectiontitle">Precautions</h4><ul id="dli_08_0382__en-us_topic_0000001262655726_ul144771616134614"><li id="dli_08_0382__en-us_topic_0000001262655726_li164771916194619">When you create a DataGen table, the table field type cannot be Array, Map, or Row. You can use <strong id="dli_08_0382__en-us_topic_0000001262655726_b382012118440">COMPUTED COLUMN</strong> in <a href="dli_08_0375.html#dli_08_0375">CREATE TABLE</a> to construct similar functions.</li><li id="dli_08_0382__en-us_topic_0000001262655726_li195721518124612">When creating a Flink OpenSource SQL job, you need to set <strong id="dli_08_0382__en-us_topic_0000001262655726_b9773122519124">Flink Version</strong> to <strong id="dli_08_0382__en-us_topic_0000001262655726_b1268223117123">1.12</strong> on the <strong id="dli_08_0382__en-us_topic_0000001262655726_b9489153810121">Running Parameters</strong> tab of the job editing page, select <strong id="dli_08_0382__en-us_topic_0000001262655726_b1836714323156">Save Job Log</strong>, and set the OBS bucket for saving job logs.</li></ul>
</div>
<div class="section" id="dli_08_0382__en-us_topic_0000001262655726_section69469501551"><h4 class="sectiontitle">Syntax</h4><pre class="screen" id="dli_08_0382__en-us_topic_0000001262655726_screen19281731397">create table dataGenSource(
attr_name attr_type
(',' attr_name attr_type)*
(',' WATERMARK FOR rowtime_column_name AS watermark-strategy_expression)
)
with (
'connector' = 'datagen'
);</pre>
</div>
<div class="section" id="dli_08_0382__en-us_topic_0000001262655726_section0475313610"><h4 class="sectiontitle">Parameters</h4>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0382__en-us_topic_0000001262655726_table517231215112" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="dli_08_0382__en-us_topic_0000001262655726_row6172712121117"><th align="left" class="cellrowborder" valign="top" width="11.72%" id="mcps1.3.5.2.2.6.1.1"><p id="dli_08_0382__en-us_topic_0000001262655726_p1417216126113">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="9.53%" id="mcps1.3.5.2.2.6.1.2"><p id="dli_08_0382__en-us_topic_0000001262655726_p161724126119">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="16.32%" id="mcps1.3.5.2.2.6.1.3"><p id="dli_08_0382__en-us_topic_0000001262655726_p16172151215110">Default Value</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="11.53%" id="mcps1.3.5.2.2.6.1.4"><p id="dli_08_0382__en-us_topic_0000001262655726_p19172512121111">Data Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="50.9%" id="mcps1.3.5.2.2.6.1.5"><p id="dli_08_0382__en-us_topic_0000001262655726_p19172101221114">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="dli_08_0382__en-us_topic_0000001262655726_row8172141214114"><td class="cellrowborder" valign="top" width="11.72%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0382__en-us_topic_0000001262655726_p1017211124118">connector</p>
</td>
<td class="cellrowborder" valign="top" width="9.53%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0382__en-us_topic_0000001262655726_p13172912101115">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="16.32%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0382__en-us_topic_0000001262655726_p168024243501">None</p>
</td>
<td class="cellrowborder" valign="top" width="11.53%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0382__en-us_topic_0000001262655726_p141721112151113">String</p>
</td>
<td class="cellrowborder" valign="top" width="50.9%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0382__en-us_topic_0000001262655726_p15172161213119">Connector to be used. Set this parameter to <strong id="dli_08_0382__en-us_topic_0000001262655726_b1126530121711">datagen</strong>.</p>
</td>
</tr>
<tr id="dli_08_0382__en-us_topic_0000001262655726_row1172131216111"><td class="cellrowborder" valign="top" width="11.72%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0382__en-us_topic_0000001262655726_p8172161218116">rows-per-second</p>
</td>
<td class="cellrowborder" valign="top" width="9.53%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0382__en-us_topic_0000001262655726_p131721127112">No</p>
</td>
<td class="cellrowborder" valign="top" width="16.32%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0382__en-us_topic_0000001262655726_p111721412111110">10000</p>
</td>
<td class="cellrowborder" valign="top" width="11.53%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0382__en-us_topic_0000001262655726_p117211126113">Long</p>
</td>
<td class="cellrowborder" valign="top" width="50.9%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0382__en-us_topic_0000001262655726_p121721112101118">Number of rows generated per second, which is used to control the emit rate.</p>
</td>
</tr>
<tr id="dli_08_0382__en-us_topic_0000001262655726_row8172812171117"><td class="cellrowborder" valign="top" width="11.72%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0382__en-us_topic_0000001262655726_p15172101291119">fields.#.kind</p>
</td>
<td class="cellrowborder" valign="top" width="9.53%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0382__en-us_topic_0000001262655726_p181721712171120">No</p>
</td>
<td class="cellrowborder" valign="top" width="16.32%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0382__en-us_topic_0000001262655726_p01721212141119">random</p>
</td>
<td class="cellrowborder" valign="top" width="11.53%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0382__en-us_topic_0000001262655726_p41721712111118">String</p>
</td>
<td class="cellrowborder" valign="top" width="50.9%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0382__en-us_topic_0000001262655726_p31971475539">Generator of the <strong id="dli_08_0382__en-us_topic_0000001262655726_b69991458201912">#</strong> field. The <strong id="dli_08_0382__en-us_topic_0000001262655726_b1233125211207">#</strong> field must be an actual field in the DataGen table. Replace <strong id="dli_08_0382__en-us_topic_0000001262655726_b1035261913210">#</strong> with the corresponding field name. The meanings of the <strong id="dli_08_0382__en-us_topic_0000001262655726_b195721048112119">#</strong> field for other parameters are the same.</p>
<p id="dli_08_0382__en-us_topic_0000001262655726_p8543610141814">The value can be <strong id="dli_08_0382__en-us_topic_0000001262655726_b177305447228">sequence</strong> or <strong id="dli_08_0382__en-us_topic_0000001262655726_b6726144518221">random</strong>.</p>
<ul id="dli_08_0382__en-us_topic_0000001262655726_ul11237181812018"><li id="dli_08_0382__en-us_topic_0000001262655726_li129801523117"><strong id="dli_08_0382__en-us_topic_0000001262655726_b896257102210">random</strong> is the default generator. You can use the <strong id="dli_08_0382__en-us_topic_0000001262655726_b1591186112312">fields.#.max</strong> and <strong id="dli_08_0382__en-us_topic_0000001262655726_b114359818238">fields.#.min</strong> parameters to specify the maximum and minimum values that are randomly generated.<p id="dli_08_0382__en-us_topic_0000001262655726_p749135319115">If the specified field type is char, varchar, or string, you can also use the <strong id="dli_08_0382__en-us_topic_0000001262655726_b65361741132310">fields.#.length</strong> field to specify the length. A random generator is an unbounded generator.</p>
</li><li id="dli_08_0382__en-us_topic_0000001262655726_li525092319018">Sequence generator. You can use <strong id="dli_08_0382__en-us_topic_0000001262655726_b4544198163610">fields.#.start</strong> and <strong id="dli_08_0382__en-us_topic_0000001262655726_b20121181113617">fields.#.end</strong> to specify the start and end values of a sequence. A sequence generator is a bounded generator. When the sequence number reaches the end value, the reading ends.</li></ul>
</td>
</tr>
<tr id="dli_08_0382__en-us_topic_0000001262655726_row1717213122117"><td class="cellrowborder" valign="top" width="11.72%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0382__en-us_topic_0000001262655726_p151728122110">fields.#.min</p>
</td>
<td class="cellrowborder" valign="top" width="9.53%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0382__en-us_topic_0000001262655726_p71721912141111">No</p>
</td>
<td class="cellrowborder" valign="top" width="16.32%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0382__en-us_topic_0000001262655726_p17172191251117">Minimum value of the field type specified by <strong id="dli_08_0382__en-us_topic_0000001262655726_b981275153820">#</strong></p>
</td>
<td class="cellrowborder" valign="top" width="11.53%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0382__en-us_topic_0000001262655726_p1617221213117">Field type specified by <strong id="dli_08_0382__en-us_topic_0000001262655726_b62611521143818">#</strong></p>
</td>
<td class="cellrowborder" valign="top" width="50.9%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0382__en-us_topic_0000001262655726_p420473212613">This parameter is valid only when <strong id="dli_08_0382__en-us_topic_0000001262655726_b791833012388">fields.#.kind</strong> is set to <strong id="dli_08_0382__en-us_topic_0000001262655726_b314033243817">random</strong>.</p>
<p id="dli_08_0382__en-us_topic_0000001262655726_p13172712121120">Minimum value of the random generator. It applies only to numeric field types specified by <strong id="dli_08_0382__en-us_topic_0000001262655726_b143631623113910">#</strong>.</p>
</td>
</tr>
<tr id="dli_08_0382__en-us_topic_0000001262655726_row8172121251110"><td class="cellrowborder" valign="top" width="11.72%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0382__en-us_topic_0000001262655726_p17172212181117">fields.#.max</p>
</td>
<td class="cellrowborder" valign="top" width="9.53%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0382__en-us_topic_0000001262655726_p117201215119">No</p>
</td>
<td class="cellrowborder" valign="top" width="16.32%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0382__en-us_topic_0000001262655726_p1985120175617">Maximum value of the field type specified by <strong id="dli_08_0382__en-us_topic_0000001262655726_b1742554717395">#</strong></p>
</td>
<td class="cellrowborder" valign="top" width="11.53%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0382__en-us_topic_0000001262655726_p1661330654">Field type specified by <strong id="dli_08_0382__en-us_topic_0000001262655726_b12496115615391">#</strong></p>
</td>
<td class="cellrowborder" valign="top" width="50.9%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0382__en-us_topic_0000001262655726_p368316402811">This parameter is valid only when <strong id="dli_08_0382__en-us_topic_0000001262655726_b080512217409">fields.#.kind</strong> is set to <strong id="dli_08_0382__en-us_topic_0000001262655726_b1796435184015">random</strong>.</p>
<p id="dli_08_0382__en-us_topic_0000001262655726_p10172912121116">Maximum value of the random generator. It applies only to numeric field types specified by <strong id="dli_08_0382__en-us_topic_0000001262655726_b6366144612438">#</strong>.</p>
</td>
</tr>
<tr id="dli_08_0382__en-us_topic_0000001262655726_row417211219118"><td class="cellrowborder" valign="top" width="11.72%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0382__en-us_topic_0000001262655726_p20172141217114">fields.#.length</p>
</td>
<td class="cellrowborder" valign="top" width="9.53%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0382__en-us_topic_0000001262655726_p131729128113">No</p>
</td>
<td class="cellrowborder" valign="top" width="16.32%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0382__en-us_topic_0000001262655726_p217281291119">100</p>
</td>
<td class="cellrowborder" valign="top" width="11.53%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0382__en-us_topic_0000001262655726_p12172912181112">Integer</p>
</td>
<td class="cellrowborder" valign="top" width="50.9%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0382__en-us_topic_0000001262655726_p36511201197">This parameter is valid only when <strong id="dli_08_0382__en-us_topic_0000001262655726_b1587083314012">fields.#.kind</strong> is set to <strong id="dli_08_0382__en-us_topic_0000001262655726_b18877183384013">random</strong>.</p>
<p id="dli_08_0382__en-us_topic_0000001262655726_p171727125117">Length of the characters generated by the random generator. It applies only to char, varchar, and string types specified by <strong id="dli_08_0382__en-us_topic_0000001262655726_b1187822512415">#</strong>.</p>
</td>
</tr>
<tr id="dli_08_0382__en-us_topic_0000001262655726_row15172121213111"><td class="cellrowborder" valign="top" width="11.72%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0382__en-us_topic_0000001262655726_p10172181231115">fields.#.start</p>
</td>
<td class="cellrowborder" valign="top" width="9.53%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0382__en-us_topic_0000001262655726_p1717281271114">No</p>
</td>
<td class="cellrowborder" valign="top" width="16.32%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0382__en-us_topic_0000001262655726_p217221281110">None</p>
</td>
<td class="cellrowborder" valign="top" width="11.53%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0382__en-us_topic_0000001262655726_p3432185819915">Field type specified by <strong id="dli_08_0382__en-us_topic_0000001262655726_b51494004414">#</strong></p>
</td>
<td class="cellrowborder" valign="top" width="50.9%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0382__en-us_topic_0000001262655726_p1067273311910">This parameter is valid only when <strong id="dli_08_0382__en-us_topic_0000001262655726_b16133144320448">fields.#.kind</strong> is set to <strong id="dli_08_0382__en-us_topic_0000001262655726_b151351543114419">sequence</strong>.</p>
<p id="dli_08_0382__en-us_topic_0000001262655726_p1917201261115">Start value of a sequence generator.</p>
</td>
</tr>
<tr id="dli_08_0382__en-us_topic_0000001262655726_row17172812201110"><td class="cellrowborder" valign="top" width="11.72%" headers="mcps1.3.5.2.2.6.1.1 "><p id="dli_08_0382__en-us_topic_0000001262655726_p9172712161114">fields.#.end</p>
</td>
<td class="cellrowborder" valign="top" width="9.53%" headers="mcps1.3.5.2.2.6.1.2 "><p id="dli_08_0382__en-us_topic_0000001262655726_p2172171271120">No</p>
</td>
<td class="cellrowborder" valign="top" width="16.32%" headers="mcps1.3.5.2.2.6.1.3 "><p id="dli_08_0382__en-us_topic_0000001262655726_p12172171241117">None</p>
</td>
<td class="cellrowborder" valign="top" width="11.53%" headers="mcps1.3.5.2.2.6.1.4 "><p id="dli_08_0382__en-us_topic_0000001262655726_p817291271110">Field type specified by <strong id="dli_08_0382__en-us_topic_0000001262655726_b38422524517">#</strong></p>
</td>
<td class="cellrowborder" valign="top" width="50.9%" headers="mcps1.3.5.2.2.6.1.5 "><p id="dli_08_0382__en-us_topic_0000001262655726_p172726421499">This parameter is valid only when <strong id="dli_08_0382__en-us_topic_0000001262655726_b177520854513">fields.#.kind</strong> is set to <strong id="dli_08_0382__en-us_topic_0000001262655726_b14777148164510">sequence</strong>.</p>
<p id="dli_08_0382__en-us_topic_0000001262655726_p017251217118">End value of a sequence generator.</p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="dli_08_0382__en-us_topic_0000001262655726_section1572208365"><h4 class="sectiontitle">Example</h4><p id="dli_08_0382__en-us_topic_0000001262655726_p133815267248">Create a Flink OpenSource SQL job. Run the following script to generate random data through the DataGen table and output the data to the Print result table.</p>
<p id="dli_08_0382__en-us_topic_0000001262655726_p4737175412255">When you create a job, set <strong id="dli_08_0382__en-us_topic_0000001262655726_b1530313195113">Flink Version</strong> to <strong id="dli_08_0382__en-us_topic_0000001262655726_b35306133513">1.12</strong> on the <strong id="dli_08_0382__en-us_topic_0000001262655726_b1753010135518">Running Parameters</strong> tab. Select <strong id="dli_08_0382__en-us_topic_0000001262655726_b953171318511">Save Job Log</strong>, and specify the OBS bucket for saving job logs.</p>
<pre class="screen" id="dli_08_0382__en-us_topic_0000001262655726_screen1990817895210">create table dataGenSOurce(
user_id string,
amount int
) with (
'connector' = 'datagen',
'rows-per-second' = '1', --Generates a piece of data per second.
'fields.user_id.kind' = 'random', --Specifies a random generator for the user_id field.
'fields.user_id.length' = '3' --Limits the length of user_id to 3.
);
create table printSink(
user_id string,
amount int
) with (
'connector' = 'print'
);
insert into printSink select * from dataGenSOurce;</pre>
<p id="dli_08_0382__en-us_topic_0000001262655726_p23018165418">After the job is submitted, the job status changes to <strong id="dli_08_0382__en-us_topic_0000001262655726_b18603103795317">Running</strong>. You can perform the following operations to view the output result:</p>
<ul id="dli_08_0382__en-us_topic_0000001262655726_ul133191514102210"><li id="dli_08_0382__en-us_topic_0000001262655726_li11970649381">Method 1:<ol id="dli_08_0382__en-us_topic_0000001262655726_ol4711275385"><li id="dli_08_0382__en-us_topic_0000001262655726_li5612141010388">Log in to the DLI console. In the navigation pane, choose <strong id="dli_08_0382__en-us_topic_0000001262655726_b62357286562">Job Management</strong> &gt; <strong id="dli_08_0382__en-us_topic_0000001262655726_b524292818566">Flink Jobs</strong>.</li><li id="dli_08_0382__en-us_topic_0000001262655726_li117110711383">Locate the row that contains the target Flink job, and choose <strong id="dli_08_0382__en-us_topic_0000001262655726_b15206134195616">More</strong> &gt; <strong id="dli_08_0382__en-us_topic_0000001262655726_b82076485615">FlinkUI</strong> in the <strong id="dli_08_0382__en-us_topic_0000001262655726_b15207164175618">Operation</strong> column.</li><li id="dli_08_0382__en-us_topic_0000001262655726_li07833554385">On the Flink UI, choose <strong id="dli_08_0382__en-us_topic_0000001262655726_b103945373566">Task Managers</strong>, click the task name, and select <strong id="dli_08_0382__en-us_topic_0000001262655726_b2400193705619">Stdout</strong> to view job logs.</li></ol>
</li><li id="dli_08_0382__en-us_topic_0000001262655726_li341910155285">Method 2: If you select <strong id="dli_08_0382__en-us_topic_0000001262655726_b1451949195611">Save Job Log</strong> on the <strong id="dli_08_0382__en-us_topic_0000001262655726_b1345449145610">Running Parameters</strong> tab before submitting the job, perform the following operations:<ol id="dli_08_0382__en-us_topic_0000001262655726_ol864115198285"><li id="dli_08_0382__en-us_topic_0000001262655726_li10901621122819">Log in to the DLI console. In the navigation pane, choose <strong id="dli_08_0382__en-us_topic_0000001262655726_b88821120578">Job Management</strong> &gt; <strong id="dli_08_0382__en-us_topic_0000001262655726_b9883191185714">Flink Jobs</strong>.</li><li id="dli_08_0382__en-us_topic_0000001262655726_li1912163912282">Click the name of the corresponding Flink job, choose <strong id="dli_08_0382__en-us_topic_0000001262655726_b14371410155715">Run Log</strong>, click <strong id="dli_08_0382__en-us_topic_0000001262655726_b1937791075715">OBS Bucket</strong>, and locate the folder of the log you want to view according to the date.</li><li id="dli_08_0382__en-us_topic_0000001262655726_li0641191914285">Go to the folder of the date, find the folder whose name contains <strong id="dli_08_0382__en-us_topic_0000001262655726_b1065712249575">taskmanager</strong>, download the <strong id="dli_08_0382__en-us_topic_0000001262655726_b3658162475717">taskmanager.out</strong> file, and view result logs.</li></ol>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0381.html">Creating Source Tables</a></div>
</div>
</div>
View Git Blame Copy Permalink