forked from docs/doc-exports
Xiong, Chen Xiao
f2733178cc
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com> Co-authored-by: Xiong, Chen Xiao <chenxiaoxiong@huawei.com> Co-committed-by: Xiong, Chen Xiao <chenxiaoxiong@huawei.com>
117 lines
12 KiB
HTML
117 lines
12 KiB
HTML
<a name="dataartsstudio_02_0304"></a><a name="dataartsstudio_02_0304"></a>
|
|
|
|
<h1 class="topictitle1">To Elasticsearch/Cloud Search Service</h1>
|
|
<div id="body8662426"><div class="section" id="dataartsstudio_02_0304__en-us_topic_0108272837_section33401108172339"><h4 class="sectiontitle">Sample JSON File</h4><pre class="screen" id="dataartsstudio_02_0304__en-us_topic_0108272837_screen6699861812014">"to-config-values": {
|
|
"configs": [
|
|
{
|
|
"inputs": [
|
|
{
|
|
"name": "toJobConfig.index",
|
|
"value": "cdm"
|
|
},
|
|
{
|
|
"name": "toJobConfig.type",
|
|
"value": "type1"
|
|
},
|
|
{
|
|
"name": "toJobConfig.shouldClearType",
|
|
"value": "false"
|
|
},
|
|
{
|
|
"name": "toJobConfig.pipeLine",
|
|
"value": "es_03"
|
|
}
|
|
],
|
|
"name": "toJobConfig"
|
|
}
|
|
]
|
|
}</pre>
|
|
</div>
|
|
<div class="section" id="dataartsstudio_02_0304__en-us_topic_0108272837_section60129242115832"><h4 class="sectiontitle">Parameter Description</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dataartsstudio_02_0304__en-us_topic_0108272837_table6307873415412" frame="border" border="1" rules="all"><thead align="left"><tr id="dataartsstudio_02_0304__en-us_topic_0108272837_row2882542015412"><th align="left" class="cellrowborder" valign="top" width="22.657734226577343%" id="mcps1.3.2.2.1.5.1.1"><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p5315765115412">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20.157984201579843%" id="mcps1.3.2.2.1.5.1.2"><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p1080249515412">Mandatory</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="16.2983701629837%" id="mcps1.3.2.2.1.5.1.3"><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p258693615412">Type</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="40.885911408859116%" id="mcps1.3.2.2.1.5.1.4"><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p821526915412">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dataartsstudio_02_0304__en-us_topic_0108272837_row3532522715412"><td class="cellrowborder" valign="top" width="22.657734226577343%" headers="mcps1.3.2.2.1.5.1.1 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p31423427144217">toJobConfig.index</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.157984201579843%" headers="mcps1.3.2.2.1.5.1.2 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p62269650144217">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="16.2983701629837%" headers="mcps1.3.2.2.1.5.1.3 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p10676914144217">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40.885911408859116%" headers="mcps1.3.2.2.1.5.1.4 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p59523723144217">Index of the written data, which is similar to the database name in the relational database</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_02_0304__en-us_topic_0108272837_row2888033144824"><td class="cellrowborder" valign="top" width="22.657734226577343%" headers="mcps1.3.2.2.1.5.1.1 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p32604129144824">toJobConfig.type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.157984201579843%" headers="mcps1.3.2.2.1.5.1.2 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p23688773144824">Yes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="16.2983701629837%" headers="mcps1.3.2.2.1.5.1.3 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p39742454144824">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40.885911408859116%" headers="mcps1.3.2.2.1.5.1.4 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p65022171144824">Type of the written data, which is similar to the table name in the relational database</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_02_0304__en-us_topic_0108272837_row56184745141343"><td class="cellrowborder" valign="top" width="22.657734226577343%" headers="mcps1.3.2.2.1.5.1.1 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p54670509141343">toJobConfig.shouldClearType</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.157984201579843%" headers="mcps1.3.2.2.1.5.1.2 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p66235072141343">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="16.2983701629837%" headers="mcps1.3.2.2.1.5.1.3 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p63440653141343">Boolean</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40.885911408859116%" headers="mcps1.3.2.2.1.5.1.4 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p38419258141343">Whether to clear data before import</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_02_0304__en-us_topic_0108272837_row130765815412"><td class="cellrowborder" valign="top" width="22.657734226577343%" headers="mcps1.3.2.2.1.5.1.1 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p40468004144217">toJobConfig.primaryKey</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.157984201579843%" headers="mcps1.3.2.2.1.5.1.2 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p56682864144217">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="16.2983701629837%" headers="mcps1.3.2.2.1.5.1.3 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p27909297144217">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40.885911408859116%" headers="mcps1.3.2.2.1.5.1.4 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p18776402152853">Primary key or unique index</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_02_0304__en-us_topic_0108272837_row5037735615412"><td class="cellrowborder" valign="top" width="22.657734226577343%" headers="mcps1.3.2.2.1.5.1.1 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p52978520144217">toJobConfig.columnList</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.157984201579843%" headers="mcps1.3.2.2.1.5.1.2 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p63401693144217">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="16.2983701629837%" headers="mcps1.3.2.2.1.5.1.3 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p35263486144217">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40.885911408859116%" headers="mcps1.3.2.2.1.5.1.4 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p37770149144217">List of fields to be written. Use <span class="parmvalue" id="dataartsstudio_02_0304__en-us_topic_0108272837_parmvalue361045875114827"><b>&</b></span> to separate field names. For example, <span class="parmvalue" id="dataartsstudio_02_0304__en-us_topic_0108272837_parmvalue1597360760113647"><b>id&gid&name</b></span>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_02_0304__en-us_topic_0108272837_row663119528426"><td class="cellrowborder" valign="top" width="22.657734226577343%" headers="mcps1.3.2.2.1.5.1.1 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p26311852144211">toJobConfig.pipeLine</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.157984201579843%" headers="mcps1.3.2.2.1.5.1.2 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p17631195244219">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="16.2983701629837%" headers="mcps1.3.2.2.1.5.1.3 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p66311752114215">String</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40.885911408859116%" headers="mcps1.3.2.2.1.5.1.4 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p8755914553">This parameter is available only after a pipeline ID is created in Kibana. It is used to convert the data format using the data transformation pipeline of Cloud Search Service or Elasticsearch after data is transferred to Cloud Search Service or Elasticsearch.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dataartsstudio_02_0304__en-us_topic_0108272837_row186852037173515"><td class="cellrowborder" valign="top" width="22.657734226577343%" headers="mcps1.3.2.2.1.5.1.1 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p1668573773512">toJobConfig.createIndexStrategy</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.157984201579843%" headers="mcps1.3.2.2.1.5.1.2 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p468615374355">No</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="16.2983701629837%" headers="mcps1.3.2.2.1.5.1.3 "><p id="dataartsstudio_02_0304__en-us_topic_0108272837_p1168663717353">Enumeration</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40.885911408859116%" headers="mcps1.3.2.2.1.5.1.4 "><div class="p" id="dataartsstudio_02_0304__en-us_topic_0108272837_p53511015420">For streaming jobs that continuously write data to Elasticsearch, CDM periodically creates indexes and writes data to the indexes, which helps you delete expired data. The indexes can be created based on the following periods:<ul id="dataartsstudio_02_0304__en-us_topic_0108272837_ul112848823812"><li id="dataartsstudio_02_0304__en-us_topic_0108272837_li1228468113814"><strong id="dataartsstudio_02_0304__en-us_topic_0108272837_b1371824210493">EveryHour</strong>: CDM creates indexes on the hour. The new indexes are named in the format of <em id="dataartsstudio_02_0304__en-us_topic_0108272837_i106418295516">Index name+Year+Month+Day+Hour</em>, for example, <span class="parmvalue" id="dataartsstudio_02_0304__en-us_topic_0108272837_parmvalue18674544175115"><b>index2018121709</b></span>.</li><li id="dataartsstudio_02_0304__en-us_topic_0108272837_li163712311494"><strong id="dataartsstudio_02_0304__en-us_topic_0108272837_b1069011219522">EveryDay</strong>: CDM creates indexes at 00:00 every day. The new indexes are named in the format of <em id="dataartsstudio_02_0304__en-us_topic_0108272837_i8722175855212">Index name+Year+Month+Day</em>, for example, <span class="parmvalue" id="dataartsstudio_02_0304__en-us_topic_0108272837_parmvalue13941086536"><b>index20181217</b></span>.</li><li id="dataartsstudio_02_0304__en-us_topic_0108272837_li118381542104917"><strong id="dataartsstudio_02_0304__en-us_topic_0108272837_b086371675318">EveryWeek</strong>: CDM creates indexes at 00:00 every Monday. The new indexes are named in the format of <em id="dataartsstudio_02_0304__en-us_topic_0108272837_i128491904543">Index name+Year+Week</em>, for example, <span class="parmvalue" id="dataartsstudio_02_0304__en-us_topic_0108272837_parmvalue1713012126548"><b>index201842</b></span>.</li><li id="dataartsstudio_02_0304__en-us_topic_0108272837_li879918301535"><strong id="dataartsstudio_02_0304__en-us_topic_0108272837_b44741014165410">EveryMonth</strong>: CDM creates indexes at 00:00 on the first day of each month. The new indexes are named in the format of <em id="dataartsstudio_02_0304__en-us_topic_0108272837_i0474514105412">Index name+Year+Month</em>, for example, <span class="parmvalue" id="dataartsstudio_02_0304__en-us_topic_0108272837_parmvalue43841656105818"><b>index201812</b></span>.</li></ul>
|
|
</div>
|
|
<p id="dataartsstudio_02_0304__en-us_topic_0108272837_p837316411590">When extracting data from a file, you must configure a single extractor, which means setting <span class="parmname" id="dataartsstudio_02_0304__en-us_topic_0108272837_parmname1018351410576"><b>Concurrent Extractors</b></span> to <strong id="dataartsstudio_02_0304__en-us_topic_0108272837_b3167163375720">1</strong>. Otherwise, this parameter is invalid.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dataartsstudio_02_0295.html">Destination Job Parameters</a></div>
|
|
</div>
|
|
</div>
|
|
|