forked from laiweijian4/doc-exports
Zheng, Xiu
0c90df93b1
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Zheng, Xiu <zhengxiu@huawei.com> Co-committed-by: Zheng, Xiu <zhengxiu@huawei.com>
192 lines
33 KiB
HTML
192 lines
33 KiB
HTML
<a name="css_01_0048"></a><a name="css_01_0048"></a>
|
|
|
|
<h1 class="topictitle1">Using Logstash to Import Data to Elasticsearch</h1>
|
|
<div id="body0000001334778789"><p id="css_01_0048__p6252043134616">You can use Logstash to collect data and migrate collected data to Elasticsearch in <span id="css_01_0048__text171881117164312">CSS</span>. This method helps you effectively obtain and manage data through Elasticsearch. Data files can be in the JSON or CSV format.</p>
|
|
<p id="css_01_0048__p2714111201213">Logstash is an open-source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to Elasticsearch. For details about Logstash, visit the following website: <a href="https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html" target="_blank" rel="noopener noreferrer">https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html</a></p>
|
|
<p id="css_01_0048__p651723317178">The following two scenarios are involved depending on the Logstash deployment:</p>
|
|
<ul id="css_01_0048__ul11548165271713"><li id="css_01_0048__li16548852101714"><a href="#css_01_0048__section072813417814">Importing Data When Logstash Is Deployed on the External Network</a></li><li id="css_01_0048__li0548652141717"><a href="#css_01_0048__section1098217174335">Importing Data When Logstash Is Deployed on an ECS</a></li></ul>
|
|
<div class="section" id="css_01_0048__section371994174412"><h4 class="sectiontitle">Prerequisites</h4><ul id="css_01_0048__ul15513934114514"><li id="css_01_0048__li851317349456">To facilitate operations, you are advised to deploy Logstash on a host that runs the Linux operating system (OS).</li><li id="css_01_0048__li105075367453">To download Logstash, visit the following website: <a href="https://www.elastic.co/downloads/logstash-oss" target="_blank" rel="noopener noreferrer">https://www.elastic.co/downloads/logstash-oss</a><div class="note" id="css_01_0048__note17978173617489"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="css_01_0048__p14791434135311">Logstash requires an OSS version same as the CSS version.</p>
|
|
</div></div>
|
|
</li><li id="css_01_0048__li16377162604714">After installing Logstash, perform the following steps to import data. For details about how to install Logstash, visit the following website: <a href="https://www.elastic.co/guide/en/logstash/current/installing-logstash.html" target="_blank" rel="noopener noreferrer">https://www.elastic.co/guide/en/logstash/current/installing-logstash.html</a></li><li id="css_01_0048__li9641131917016">The JDK must be installed before Logstash is installed. In Linux OS, you can run the <strong id="css_01_0048__b20393185013114">yum -y install java-1.8.0</strong> command to install JDK 1.8.0. In Windows OS, you can download the required JDK version from the <a href="https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html" target="_blank" rel="noopener noreferrer">official website of JDK</a>, and install it by following the installation guide.</li><li id="css_01_0048__li1995041711115">In the <a href="#css_01_0048__section1098217174335">Importing Data When Logstash Is Deployed on an ECS</a> scenario, ensure that the ECS and the Elasticsearch cluster to which data is imported reside in the same VPC.</li></ul>
|
|
</div>
|
|
<div class="section" id="css_01_0048__section072813417814"><a name="css_01_0048__section072813417814"></a><a name="section072813417814"></a><h4 class="sectiontitle">Importing Data When Logstash Is Deployed on the External Network</h4><p id="css_01_0048__p4281513171111"><a href="#css_01_0048__fig471717481106">Figure 1</a> illustrates how data is imported when Logstash is deployed on an external network.</p>
|
|
<div class="fignone" id="css_01_0048__fig471717481106"><a name="css_01_0048__fig471717481106"></a><a name="fig471717481106"></a><span class="figcap"><b>Figure 1 </b>Importing data when Logstash is deployed on an external network</span><br><span><img class="vsd" id="css_01_0048__image11185034163311" src="en-us_image_0000001554897285.png"></span></div>
|
|
<p id="css_01_0048__p12185123416332"></p>
|
|
<ol id="css_01_0048__ol20180835132314"><li id="css_01_0048__li1648853125014"><a name="css_01_0048__li1648853125014"></a><a name="li1648853125014"></a>Create a jump host and configure it as follows:<ul id="css_01_0048__ul2807152315574"><li id="css_01_0048__li18807132310572">The jump host is an ECS running the Linux OS and has been bound with an EIP.</li><li id="css_01_0048__li1692633435810">The jump host resides in the same VPC as the CSS cluster.</li><li id="css_01_0048__li129110535017">SSH local port forwarding is configured for the jump host to forward requests from a chosen local port to port <strong id="css_01_0048__b181343814612">9200</strong> on one node of the CSS cluster.</li><li id="css_01_0048__li1277830155915">Refer to <a href="https://man.openbsd.org/ssh.1#L" target="_blank" rel="noopener noreferrer">SSH documentation</a> for the local port forwarding configuration.</li></ul>
|
|
</li><li id="css_01_0048__li19164183513237">Use PuTTY to log in to the created jump host with the EIP.</li><li id="css_01_0048__li1316473511236">Run the following command to perform port mapping and transfer the request sent to the port on the jump host to the target cluster:<pre class="screen" id="css_01_0048__screen716453518231">ssh -g -L <em id="css_01_0048__i31643353232"><Local port of the jump host</em>:<em id="css_01_0048__i716443572314">Private network address and port number of a node></em> -N -f root@<em id="css_01_0048__i216416353238"><Private IP address of the jump host</em>></pre>
|
|
<div class="note" id="css_01_0048__note101641635112310"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="css_01_0048__ul3164163513237"><li id="css_01_0048__li1216415353237">In the preceding command, <em id="css_01_0048__i1216433515231"><<em id="css_01_0048__i315714325494">Local port of the jump host</em>></em> refers to the port obtained in <a href="#css_01_0048__li1648853125014">1</a>.</li><li id="css_01_0048__li516410352234">In the preceding command, <em id="css_01_0048__i142757577509"><Private network address and port number of a node></em> refers to the private network address and port number of a node in the cluster. If the node is faulty, the command execution will fail. If the cluster contains multiple nodes, you can replace the value of <em id="css_01_0048__i16164119103712"><</em><em id="css_01_0048__i51641396379">private network address and port number of a node></em> with the private network address and port number of any available node in the cluster. If the cluster contains only one node, restore the node and execute the command again.</li><li id="css_01_0048__li91643359239">Replace <Private IP address of the <em id="css_01_0048__i1616463510233"><em id="css_01_0048__i146539612506">jump host</em></em>> in the preceding command with the IP address (with <span class="parmvalue" id="css_01_0048__parmvalue31641635122313"><b>Private IP</b></span>) of the created jump host in the <span class="parmname" id="css_01_0048__parmname8164735202318"><b>IP Address</b></span> column in the ECS list on the ECS management console.</li></ul>
|
|
</div></div>
|
|
<p id="css_01_0048__p1416483562310">For example, port <strong id="css_01_0048__b6116104104118">9200</strong> on the jump host is assigned external network access permissions, the private network address and port number of the node are <strong id="css_01_0048__b11808313212">192.168.0.81</strong> and <strong id="css_01_0048__b172891341022">9200</strong>, respectively, and the private IP address of the jump host is <strong id="css_01_0048__b185281761121">192.168.0.227</strong>. You need to run the following command to perform port mapping:</p>
|
|
<pre class="screen" id="css_01_0048__screen7164123516239">ssh -g -L 9200:192.168.0.81:9200 -N -f root@192.168.0.227</pre>
|
|
</li><li id="css_01_0048__li5164153542312"><a name="css_01_0048__li5164153542312"></a><a name="li5164153542312"></a>Log in to the server where Logstash is deployed and store the data files to be imported on the server.<p id="css_01_0048__p9164235142316"><a name="css_01_0048__li5164153542312"></a><a name="li5164153542312"></a>For example, data file <span class="filepath" id="css_01_0048__filepath142871815163915"><b>access_20181029_log</b></span> needs to be imported, the file storage path is <span class="filepath" id="css_01_0048__filepath1428781583910"><b>/tmp/access_log/</b></span>, and the data file includes the following data:</p>
|
|
<div class="note" id="css_01_0048__note12987155053817"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="css_01_0048__p19881050183816">Create the <strong id="css_01_0048__b143944294322">access_log</strong> folder if it does not exist.</p>
|
|
</div></div>
|
|
<div class="p" id="css_01_0048__p154861346103811"><pre class="screen" id="css_01_0048__screen916412354232">| All | Heap used for segments | | 18.6403 | MB |
|
|
| All | Heap used for doc values | | 0.119289 | MB |
|
|
| All | Heap used for terms | | 17.4095 | MB |
|
|
| All | Heap used for norms | | 0.0767822 | MB |
|
|
| All | Heap used for points | | 0.225246 | MB |
|
|
| All | Heap used for stored fields | | 0.809448 | MB |
|
|
| All | Segment count | | 101 | |
|
|
| All | Min Throughput | index-append | 66232.6 | docs/s |
|
|
| All | Median Throughput | index-append | 66735.3 | docs/s |
|
|
| All | Max Throughput | index-append | 67745.6 | docs/s |
|
|
| All | 50th percentile latency | index-append | 510.261 | ms |</pre>
|
|
</div>
|
|
</li><li id="css_01_0048__li416420352232">In the server where Logstash is deployed, run the following command to create configuration file <strong id="css_01_0048__b20766742183011">logstash-simple.conf</strong> in the Logstash installation directory:<pre class="screen" id="css_01_0048__screen8164235182310">cd /<em id="css_01_0048__i1317714259514"><Logstash installation directory></em>/
|
|
vi logstash-simple.conf</pre>
|
|
</li><li id="css_01_0048__li416415355230">Input the following content in <strong id="css_01_0048__b9720810494">logstash-simple.conf</strong>:<pre class="screen" id="css_01_0048__screen151641235102317">input {
|
|
<em id="css_01_0048__i1594111511224">Location of data</em>
|
|
}
|
|
filter {
|
|
<em id="css_01_0048__i10594815152216">Related data processing</em>
|
|
}
|
|
output {
|
|
elasticsearch {
|
|
hosts => "<EIP of the <em id="css_01_0048__i3542420651"><em id="css_01_0048__i2541152016516">jump host</em></em>>:<em id="css_01_0048__i11542620659"><Number of the port assigned external network access permissions on the jump host></em>"
|
|
(Optional) If communication encryption has been enabled on the cluster, you need to add the following configuration:
|
|
ssl => true
|
|
ssl_certificate_verification => false
|
|
}
|
|
}</pre>
|
|
<ul id="css_01_0048__ul1416414353230"><li id="css_01_0048__li1216411359233">The <strong id="css_01_0048__b071245572520">input</strong> parameter indicates the data source. Set this parameter based on the actual conditions. For details about the <strong id="css_01_0048__b11998827110">input</strong> parameter and parameter usage, visit the following website: <a href="https://www.elastic.co/guide/en/logstash/current/input-plugins.html" target="_blank" rel="noopener noreferrer">https://www.elastic.co/guide/en/logstash/current/input-plugins.html</a></li><li id="css_01_0048__li9164735172310">The <strong id="css_01_0048__b16659135673710">filter</strong> parameter specifies the mode in which data is processed. For example, extract and process logs to convert unstructured information into structured information. For details about the <strong id="css_01_0048__b3326196113">filter</strong> parameter and parameter usage, visit the following website: <a href="https://www.elastic.co/guide/en/logstash/current/filter-plugins.html" target="_blank" rel="noopener noreferrer">https://www.elastic.co/guide/en/logstash/current/filter-plugins.html</a></li><li id="css_01_0048__li121641735122315">The <strong id="css_01_0048__b8423527062042">output</strong> parameter indicates the destination address of the data. For details about the <strong id="css_01_0048__b15520928389">output</strong> parameter and parameter usage, visit <a href="https://www.elastic.co/guide/en/logstash/current/output-plugins.html" target="_blank" rel="noopener noreferrer">https://www.elastic.co/guide/en/logstash/current/output-plugins.html</a>. Replace <<em id="css_01_0048__i1628913582507">EIP address of the <em id="css_01_0048__i192892589504">jump host</em></em>> with the IP address (with <span class="parmvalue" id="css_01_0048__parmvalue1516493516235"><b>EIP</b></span>) of the created jump host in the <span class="parmname" id="css_01_0048__parmname19290105814503"><b>IP Address</b></span> column in the ECS list on the ECS management console. <em id="css_01_0048__i1749321524419"><Number of the port assigned external network access permissions on the jump host></em> is the number of the port obtained in <a href="#css_01_0048__li1648853125014">1</a>, for example, <strong id="css_01_0048__b1049461554418">9200</strong>.</li></ul>
|
|
<p id="css_01_0048__p1164143513237">Consider the data files in the <span class="filepath" id="css_01_0048__filepath78321133133818"><b>/tmp/access_log/</b></span> path mentioned in <a href="#css_01_0048__li5164153542312">4</a> as an example. Assume that data import starts from data in the first row of the data file, the filtering condition is left unspecified (indicating no data processing operations are performed), the public IP address and port number of the jump host are <span class="parmvalue" id="css_01_0048__parmvalue1482182212512"><b>192.168.0.227</b></span> and <span class="parmvalue" id="css_01_0048__parmvalue10483622950"><b>9200</b></span>, respectively, and the name of the target index is <span class="parmvalue" id="css_01_0048__parmvalue1637077173915"><b>myindex</b></span>. Edit the configuration file as follows, and enter <span class="parmvalue" id="css_01_0048__parmvalue1495317521114"><b>:wq</b></span> to save the configuration file and exit.</p>
|
|
<pre class="screen" id="css_01_0048__screen816443562316">input {
|
|
file{
|
|
path => "/tmp/access_log/*"
|
|
start_position => "beginning"
|
|
}
|
|
}
|
|
filter {
|
|
}
|
|
output {
|
|
elasticsearch {
|
|
hosts => "192.168.0.227:9200"
|
|
index => "myindex"
|
|
|
|
}
|
|
}</pre>
|
|
<div class="note" id="css_01_0048__note29452405213"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="css_01_0048__p12945184092114">If a license error is reported, set <strong id="css_01_0048__b199511958194712">ilm_enabled</strong> to <strong id="css_01_0048__b17967105964711">false</strong>.</p>
|
|
</div></div>
|
|
<p id="css_01_0048__p154751181388">If the cluster has the security mode enabled, you need to download a certificate first.</p>
|
|
<ol type="a" id="css_01_0048__ol45911713483"><li id="css_01_0048__li45915135814">Download a certificate on the <strong id="css_01_0048__b116761120201015">Basic Information</strong> page of the cluster.<div class="fignone" id="css_01_0048__fig11031228135019"><span class="figcap"><b>Figure 2 </b>Downloading a certificate</span><br><span><img id="css_01_0048__image1045615019490" src="en-us_image_0000001503977516.png"></span></div>
|
|
</li><li id="css_01_0048__li107224618910">Store the certificate to the server where Logstash is deployed.</li><li id="css_01_0048__li84281357121313">Modify the <strong id="css_01_0048__b3469133941110">logstash-simple.conf</strong> configuration file.<div class="p" id="css_01_0048__p14897136193">Consider the data files in the <span class="filepath" id="css_01_0048__filepath376984121915"><b>/tmp/access_log/</b></span> path mentioned in <a href="#css_01_0048__li5164153542312">4</a> as an example. Assume that data import starts from data in the first row of the data file, the filtering condition is left unspecified (indicating no data processing operations are performed), and the public IP address and port number of the jump host are <span class="parmvalue" id="css_01_0048__parmvalue1774888517"><b>192.168.0.227</b></span> and <span class="parmvalue" id="css_01_0048__parmvalue1925348551"><b>9200</b></span>, respectively. The name of the index for importing data is <strong id="css_01_0048__b104220143134">myindex</strong>, and the certificate is stored in <strong id="css_01_0048__b177982020111311">/logstash/logstash6.8/config/CloudSearchService.cer</strong>. Edit the configuration file as follows, and enter <span class="parmvalue" id="css_01_0048__parmvalue197616546346"><b>:wq</b></span> to save the configuration file and exit.<pre class="screen" id="css_01_0048__screen588712113193">input{
|
|
file {
|
|
path => "/tmp/access_log/*"
|
|
start_position => "beginning"
|
|
}
|
|
}
|
|
filter {
|
|
}
|
|
output{
|
|
elasticsearch{
|
|
hosts => ["https://192.168.0.227:9200"]
|
|
index => "myindex"
|
|
user => "admin"
|
|
password => "******"
|
|
cacert => "/logstash/logstash6.8/config/CloudSearchService.cer"
|
|
}
|
|
}</pre>
|
|
<div class="note" id="css_01_0048__note1288791118191"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="css_01_0048__p1352736341"><strong id="css_01_0048__b432018559131">password</strong>: password for logging in to the cluster</p>
|
|
</div></div>
|
|
</div>
|
|
</li></ol>
|
|
</li><li id="css_01_0048__li14164163582313">Run the following command to import the data collected by Logstash to the cluster:<pre class="screen" id="css_01_0048__screen121641135152312">./bin/logstash -f logstash-simple.conf</pre>
|
|
<div class="note" id="css_01_0048__note19357111419509"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="css_01_0048__p3258910205713">This command must be executed in the directory where the <strong id="css_01_0048__b750418485328">logstash-simple.conf</strong> file is stored. For example, if the <strong id="css_01_0048__b733614818335">logstash-simple.conf</strong> file is stored in <strong id="css_01_0048__b15562151313313">/root/logstash-7.1.1/</strong>, go to the directory before running the command.</p>
|
|
</div></div>
|
|
</li><li id="css_01_0048__li11641535192318">Log in to the <span id="css_01_0048__text2182135054613">CSS</span> management console.</li><li id="css_01_0048__li1716433502318">In the left navigation pane, click <span class="uicontrol" id="css_01_0048__uicontrol393210564379"><b>Clusters</b></span> to switch to the <strong id="css_01_0048__b11932105673718">Clusters</strong> page.</li><li id="css_01_0048__li31807350237">From the cluster list, locate the row that contains the cluster to which you want to import data and click <span class="parmname" id="css_01_0048__parmname1569018551468"><b>Access Kibana</b></span> in the <span class="parmname" id="css_01_0048__parmname3690105511469"><b>Operation</b></span> column.</li><li id="css_01_0048__li9180335132310">In the left navigation pane of the displayed Kibana window, click <span class="parmname" id="css_01_0048__parmname13948820476"><b>Dev Tools</b></span>.<div class="fignone" id="css_01_0048__fig164001949115016"><span class="figcap"><b>Figure 3 </b>Logging in to Dev Tools</span><br><span><img id="css_01_0048__image15885941135018" src="en-us_image_0000001503657724.png"></span></div>
|
|
</li><li id="css_01_0048__li418033592313">On the <strong id="css_01_0048__b119311594497">Console</strong> page of Kibana, search for the imported data.<p id="css_01_0048__p118053518234">On the <strong id="css_01_0048__b1685619109106">Console</strong> page of Kibana, enter the following command to search for data. View the search results. If the searched data is consistent with the imported data, the data has been imported successfully.</p>
|
|
<pre class="screen" id="css_01_0048__screen1718063513237">GET myindex/_search</pre>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="css_01_0048__section1098217174335"><a name="css_01_0048__section1098217174335"></a><a name="section1098217174335"></a><h4 class="sectiontitle">Importing Data When Logstash Is Deployed on an ECS</h4><p id="css_01_0048__p110725613115"><a href="#css_01_0048__fig124034434127">Figure 4</a> illustrates how data is imported when Logstash is deployed on an ECS that resides in the same VPC as the cluster to which data is to be imported.</p>
|
|
<div class="fignone" id="css_01_0048__fig124034434127"><a name="css_01_0048__fig124034434127"></a><a name="fig124034434127"></a><span class="figcap"><b>Figure 4 </b>Importing data when Logstash is deployed on an ECS</span><br><span><img class="vsd" id="css_01_0048__image3127102312364" src="en-us_image_0000001503817616.png"></span></div>
|
|
<ol id="css_01_0048__ol3524114322310"><li id="css_01_0048__li1852474313235">Ensure that the ECS where Logstash is deployed and the cluster to which data is to be imported reside in the same VPC, port <strong id="css_01_0048__b694015623710">9200</strong> of the ECS security group has been assigned external network access permissions, and an EIP has been bound to the ECS.<div class="note" id="css_01_0048__note866415302112"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="css_01_0048__ul112263151393"><li id="css_01_0048__li1722615151999">If there are multiple servers in a VPC, you do not need to associate EIPs to other servers as long as one server is associated with an EIP. Switch to the node where Logstash is deployed from the node with which the EIP is associated.</li><li id="css_01_0048__li1363015162912">If a private line or VPN is available, you do not need to associate an EIP.</li></ul>
|
|
</div></div>
|
|
</li><li id="css_01_0048__li1652411439236"><a name="css_01_0048__li1652411439236"></a><a name="li1652411439236"></a>Use PuTTY to log in to the ECS.<div class="p" id="css_01_0048__p12524643162314"><a name="css_01_0048__li1652411439236"></a><a name="li1652411439236"></a>For example, data file <span class="filepath" id="css_01_0048__filepath1388819169466"><b>access_20181029_log</b></span> is stored in the <span class="filepath" id="css_01_0048__filepath138891516164613"><b>/tmp/access_log/</b></span> path of the ECS, and the data file includes the following data:<pre class="screen" id="css_01_0048__screen19524243192319">| All | Heap used for segments | | 18.6403 | MB |
|
|
| All | Heap used for doc values | | 0.119289 | MB |
|
|
| All | Heap used for terms | | 17.4095 | MB |
|
|
| All | Heap used for norms | | 0.0767822 | MB |
|
|
| All | Heap used for points | | 0.225246 | MB |
|
|
| All | Heap used for stored fields | | 0.809448 | MB |
|
|
| All | Segment count | | 101 | |
|
|
| All | Min Throughput | index-append | 66232.6 | docs/s |
|
|
| All | Median Throughput | index-append | 66735.3 | docs/s |
|
|
| All | Max Throughput | index-append | 67745.6 | docs/s |
|
|
| All | 50th percentile latency | index-append | 510.261 | ms |</pre>
|
|
</div>
|
|
</li><li id="css_01_0048__li9524143172314">Run the following command to create configuration file <strong id="css_01_0048__b1712944816374">logstash-simple.conf</strong> in the Logstash installation directory:<pre class="screen" id="css_01_0048__screen2524104318230">cd /<em id="css_01_0048__i42791750123716"><Logstash installation directory></em>/
|
|
vi logstash-simple.conf</pre>
|
|
<div class="p" id="css_01_0048__p15241243162314">Input the following content in <strong id="css_01_0048__b57081831161017">logstash-simple.conf</strong>:<pre class="screen" id="css_01_0048__screen652484313231">input {
|
|
<em id="css_01_0048__i1624279248">Location of data</em>
|
|
}
|
|
filter {
|
|
<em id="css_01_0048__i1602534201">Related data processing</em>
|
|
}
|
|
output {
|
|
elasticsearch{
|
|
hosts => "<em id="css_01_0048__i132814017518"><Private network address and port number of the node</em>>"}
|
|
(Optional) If communication encryption has been enabled on the cluster, you need to add the following configuration:
|
|
ssl => true
|
|
ssl_certificate_verification => false
|
|
}</pre>
|
|
<ul id="css_01_0048__ul165241743132318"><li id="css_01_0048__li952444318236">The <strong id="css_01_0048__b677640494">input</strong> parameter indicates the data source. Set this parameter based on the actual conditions. For details about the <strong id="css_01_0048__b584918558518">input</strong> parameter and parameter usage, visit the following website: <a href="https://www.elastic.co/guide/en/logstash/current/input-plugins.html" target="_blank" rel="noopener noreferrer">https://www.elastic.co/guide/en/logstash/current/input-plugins.html</a></li><li id="css_01_0048__li1524194372320">The <strong id="css_01_0048__b186751957731">filter</strong> parameter specifies the mode in which data is processed. For example, extract and process logs to convert unstructured information into structured information. For details about the <strong id="css_01_0048__b14934557951">filter</strong> parameter and parameter usage, visit the following website: <a href="https://www.elastic.co/guide/en/logstash/current/filter-plugins.html" target="_blank" rel="noopener noreferrer">https://www.elastic.co/guide/en/logstash/current/filter-plugins.html</a></li><li id="css_01_0048__li552418439238">The <strong id="css_01_0048__b480709037">output</strong> parameter indicates the destination address of the data. For details about the <strong id="css_01_0048__b1936214520613">output</strong> parameter and parameter usage, visit <a href="https://www.elastic.co/guide/en/logstash/current/output-plugins.html" target="_blank" rel="noopener noreferrer">https://www.elastic.co/guide/en/logstash/current/output-plugins.html</a>. <em id="css_01_0048__i10768174713615"><private network address and port number of a node></em> refers to the private network address and port number of a node in the cluster.<p id="css_01_0048__p18524164317236">If the cluster contains multiple nodes, you are advised to replace the value of <em id="css_01_0048__i1853041431119"><Private network address and port number of a node></em> with the private network addresses and port numbers of all nodes in the cluster to prevent node faults. Use commas (,) to separate the nodes' private network addresses and port numbers. The following is an example:</p>
|
|
<pre class="screen" id="css_01_0048__screen175241043182316">hosts => ["192.168.0.81:9200","192.168.0.24:9200"]</pre>
|
|
<p id="css_01_0048__p452413433238">If the cluster contains only one node, the format is as follows:</p>
|
|
<pre class="screen" id="css_01_0048__screen1352474320237">hosts => "192.168.0.81:9200"</pre>
|
|
</li></ul>
|
|
</div>
|
|
<p id="css_01_0048__p3524043192311">Consider the data files in the <span class="filepath" id="css_01_0048__filepath6760171019395"><b>/tmp/access_log/</b></span> path mentioned in <a href="#css_01_0048__li1652411439236">2</a> as an example. Assume that data import starts from data in the first row of the data file, the filtering condition is left unspecified (indicating no data processing operations are performed), the private network address and port number of the node in the cluster where data is to be imported are <span class="parmvalue" id="css_01_0048__parmvalue4202195195517"><b>192.168.0.81</b></span> and <span class="parmvalue" id="css_01_0048__parmvalue1262163213480"><b>9200</b></span>, respectively, and the name of the target index is <span class="parmvalue" id="css_01_0048__parmvalue8869139144819"><b>myindex</b></span>. Edit the configuration file as follows, and enter <span class="parmvalue" id="css_01_0048__parmvalue1933982954"><b>:wq</b></span> to save the configuration file and exit.</p>
|
|
<pre class="screen" id="css_01_0048__screen952414316233">input {
|
|
file{
|
|
path => "/tmp/access_log/*"
|
|
start_position => "beginning"
|
|
}
|
|
}
|
|
filter {
|
|
}
|
|
output {
|
|
elasticsearch {
|
|
hosts => "192.168.0.81:9200"
|
|
index => "myindex"
|
|
|
|
}
|
|
}</pre>
|
|
<p id="css_01_0048__p0370830125513">If the cluster has the security mode enabled, you need to download a certificate first.</p>
|
|
<div class="p" id="css_01_0048__p63682983810"><ol type="a" id="css_01_0048__ol1817087143812"><li id="css_01_0048__en-us_topic_0111222977_li45915135814">Download a certificate on the <strong id="css_01_0048__b172536140">Basic Information</strong> page of the cluster.<div class="fignone" id="css_01_0048__fig1877019549519"><span class="figcap"><b>Figure 5 </b>Downloading a certificate</span><br><span><img id="css_01_0048__image2793173420519" src="en-us_image_0000001554777349.png"></span></div>
|
|
</li><li id="css_01_0048__en-us_topic_0111222977_li107224618910">Store the certificate to the server where Logstash is deployed.</li><li id="css_01_0048__en-us_topic_0111222977_li84281357121313">Modify the <strong id="css_01_0048__b1330784117147">logstash-simple.conf</strong> configuration file.<div class="p" id="css_01_0048__en-us_topic_0111222977_p14897136193">Consider the data files in the <span class="filepath" id="css_01_0048__filepath16776174361415"><b>/tmp/access_log/</b></span> path mentioned in <a href="#css_01_0048__li1652411439236">2</a> as an example. Assume that data import starts from data in the first row of the data file, the filtering condition is left unspecified (indicating no data processing operations are performed), the public IP address and port number of the jump host are <span class="parmvalue" id="css_01_0048__parmvalue102421758201415"><b>192.168.0.227</b></span> and <span class="parmvalue" id="css_01_0048__parmvalue42431758131415"><b>9200</b></span>, respectively. The name of the index for importing data is <strong id="css_01_0048__b296498935">myindex</strong>, and the certificate is stored in <strong id="css_01_0048__b861142389">/logstash/logstash6.8/config/CloudSearchService.cer</strong>. Edit the configuration file as follows, and enter <span class="parmvalue" id="css_01_0048__parmvalue956834074"><b>:wq</b></span> to save the configuration file and exit.<pre class="screen" id="css_01_0048__en-us_topic_0111222977_screen588712113193">input{
|
|
file {
|
|
path => "/tmp/access_log/*"
|
|
start_position => "beginning"
|
|
}
|
|
}
|
|
filter {
|
|
}
|
|
output{
|
|
elasticsearch{
|
|
hosts => ["https://192.168.0.227:9200"]
|
|
index => "myindex"
|
|
user => "admin"
|
|
password => "******"
|
|
cacert => "/logstash/logstash6.8/config/CloudSearchService.cer"
|
|
}
|
|
}</pre>
|
|
<div class="note" id="css_01_0048__en-us_topic_0111222977_note1288791118191"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="css_01_0048__en-us_topic_0111222977_p1352736341"><strong id="css_01_0048__b689489574">password</strong>: password for logging in to the cluster</p>
|
|
</div></div>
|
|
</div>
|
|
</li></ol>
|
|
</div>
|
|
</li><li id="css_01_0048__li115241243142312">Run the following command to import the ECS data collected by Logstash to the cluster:<pre class="screen" id="css_01_0048__screen3524143132312">./bin/logstash -f logstash-simple.conf</pre>
|
|
</li><li id="css_01_0048__li75241643172310">Log in to the <span id="css_01_0048__text1041162757">CSS</span> management console.</li><li id="css_01_0048__li115241343152311">In the left navigation pane, click <span class="uicontrol" id="css_01_0048__uicontrol1253985733720"><b>Clusters</b></span> to switch to the <strong id="css_01_0048__b1553919572372">Clusters</strong> page.</li><li id="css_01_0048__li3524114302314">From the cluster list, locate the row that contains the cluster to which you want to import data and click <span class="parmname" id="css_01_0048__parmname1561592889"><b>Access Kibana</b></span> in the <span class="parmname" id="css_01_0048__parmname788721907"><b>Operation</b></span> column.</li><li id="css_01_0048__li18524164302316">In the left navigation pane of the displayed Kibana window, click <span class="parmname" id="css_01_0048__parmname264720407106"><b>Dev Tools</b></span>.<div class="fignone" id="css_01_0048__fig14934191615547"><span class="figcap"><b>Figure 6 </b>Choosing Dev Tools</span><br><span><img id="css_01_0048__en-us_topic_0000001268314481_image3902622164718" src="en-us_image_0000001503657720.png"></span></div>
|
|
</li><li id="css_01_0048__li1252410433236">On the <strong id="css_01_0048__b519438726">Console</strong> page of Kibana, search for the imported data.<p id="css_01_0048__p115241343172312">On the <strong id="css_01_0048__b217115189110">Console</strong> page of Kibana, enter the following command to search for data. View the search results. If the searched data is consistent with the imported data, the data has been imported successfully.</p>
|
|
<pre class="screen" id="css_01_0048__screen75241543182320">GET myindex/_search</pre>
|
|
</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="css_01_0045.html">Importing Data to Elasticsearch</a></div>
|
|
</div>
|
|
</div>
|
|
|