vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dataartsstudio/umn/dataartsstudio_01_0059.html
Xiong, Chen Xiao f3ee29a3dd DataArts UMN 20230329 version 
Reviewed-by: Rechenburg, Matthias <matthias.rechenburg@t-systems.com>
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Xiong, Chen Xiao <chenxiaoxiong@huawei.com>
Co-committed-by: Xiong, Chen Xiao <chenxiaoxiong@huawei.com>
2023-05-11 17:09:25 +00:00

13 KiB
Raw Permalink Blame History

From Elasticsearch or CSS

If the source link of a job is the Link to Elasticsearch/CSS, configure the source job parameters based on Table 1.

Table 1 Job parameters when Elasticsearch or CSS is the source

Parameter

Description

Example Value

Index

Elasticsearch index, which is similar to the name of a relational database. The index name can contain only lowercase letters.

index

Type

Elasticsearch type, which is similar to the table name of a relational database. The type name can contain only lowercase letters.

type

Split Nested Field

(Optional) Whether to split the JSON content of the nested fields. For example, a:{ b:{ c:1, d:{ e:2, f:3 } } } can be split into a.b.c, a.b.d.e, and a.b.d.f.

No

Filter Conditions
(Optional) CDM migrates only the data that meets the filter conditions.
  • Currently, only the query string (q syntax) of Elasticsearch can be used to filter source data. The q syntax is used in the following way:
    • In exact match, the column:data format is used to match and filter data. column indicates the field name, and data indicates the query condition, for example, last_name:Smith.

      In addition, if data is a string containing spaces, it must be enclosed in double quotation marks. If column is not specified, all fields will be matched by data.

    • Multiple query conditions can be combined with connection words. The format is column1:data1 AND column2:data2. The connection words can be AND, OR, or NOT. They must be in uppercase, and there must be a space before and after each connection word.

      Example: last_name:Smith AND last_name:John

    • In range matching, you can directly use a condition expression to filter data. The expression is in column:>data format. The operator can be >, >=, <, or <=.

      An example is time:>=1636905600000 AND time:<1637078400000. It can also be used together with a macro variable of date and time, for example, createTime:>=${timestamp(dateformat(yyyyMMdd,-1,DAY))} AND createTime:< ${timestamp(dateformat(yyyyMMdd))}.

    • In range matching, you can also use the range syntax to filter data. The format is column:{data1 TO data2}. { and } indicate that a value is not included. [ and ] indicate that a value is included. TO must be capitalized, and there must be a space before and after it. * indicates all data.

      For example, time:{1636992000000 TO *] filters out all the data greater than 1636992000000 in the time field. It can also be used together with a macro variable of date and time, for example, createTime:[${timestamp(dateformat(yyyyMMdd,-1,DAY))} TO ${timestamp(dateformat(yyyyMMdd))}}.

  • Source data cannot be filtered using the query domain-specific language (DSL) of Elasticsearch.

last_name:Smith

Extract Meta-field

Whether to extract index meta-fields. For example, _index, _type, _id, and _score.

Yes

On the Map Field page, you can set custom fields for the source and destination.

Figure 1 Setting custom fields
Parent topic: Source Job Parameters