forked from docs/doc-exports
Yang, Tong
48706b7552
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
60 lines
15 KiB
HTML
60 lines
15 KiB
HTML
<a name="mrs_01_24510"></a><a name="mrs_01_24510"></a>
|
|
|
|
<h1 class="topictitle1">Importing IoTDB Data</h1>
|
|
<div id="body0000001583271061"><div class="section" id="mrs_01_24510__section057502435010"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_24510__p197821827145013">This section describes how to use <strong id="mrs_01_24510__b93832521501">import-csv.sh</strong> to import data in CSV format to IoTDB.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_24510__section1138685516519"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_24510__ul11736334115416"><li id="mrs_01_24510__li1473613445415">The client has been installed. For details, see . For example, the installation directory is <strong id="mrs_01_24510__b195961326171"><span id="mrs_01_24510__ph1859543261710">/opt/client</span></strong>. The client directory in the following operations is only an example. Change it based on the actual installation directory onsite.</li><li id="mrs_01_24510__li9736183418540">Service component users have been created by the MRS cluster administrator by referring to . In security mode, machine-machine users need to download the keytab file. For details, see . A human-machine user must change the password upon the first login.</li><li id="mrs_01_24510__li6528347103918">By default, SSL is enabled on the server. You have generated the <strong id="mrs_01_24510__b337193519518">truststore.jks</strong> certificate by following the instructions provided in <a href="mrs_01_24158.html">Using the IoTDB Client</a> and copied it to the <strong id="mrs_01_24510__b5517172114528"><em id="mrs_01_24510__i681343385017">Client installation directory</em></strong><strong id="mrs_01_24510__b351732116527">/IoTDB/iotdb/conf</strong> directory.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_24510__section1545519304532"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_24510__ol15963838205313"><li id="mrs_01_24510__li4604150164718"><a name="mrs_01_24510__li4604150164718"></a><a name="li4604150164718"></a>Prepare a CSV file named <strong id="mrs_01_24510__b169308591335126">example-filename.csv</strong> on the local PC with the following content:<pre class="screen" id="mrs_01_24510__screen196881544202212">Time,root.fit.d1.s1,root.fit.d1.s2,root.fit.d2.s1,root.fit.d2.s3,root.fit.p.s1
|
|
1,100,hello,200,300,400
|
|
2,500,world,600,700,800
|
|
3,900,"hello, \"world\"",1000,1100,1200</pre>
|
|
<div class="notice" id="mrs_01_24510__note141031056182513"><span class="noticetitle"><img src="public_sys-resources/notice_3.0-en-us.png"> </span><div class="noticebody"><p id="mrs_01_24510__p1211165272917">Before importing data, pay attention to the following:</p>
|
|
<ul id="mrs_01_24510__ul197504411373"><li id="mrs_01_24510__li2282415534">The data to be imported cannot contain spaces. Otherwise, importing that line of data fails and is skipped, but subsequent import operations are not affected.</li><li id="mrs_01_24510__li117505417712">Data that contains commas (,) must be enclosed in single or double quotation marks. For example, <strong id="mrs_01_24510__b8529151414277">hello,world</strong> is changed to <strong id="mrs_01_24510__b1130142017272">"hello,world"</strong>.</li><li id="mrs_01_24510__li075014411979">Quotation marks ("") in the data must be replaced with the escape character <strong id="mrs_01_24510__b419742754519">\"</strong>. For example, <strong id="mrs_01_24510__b18441934122917">"world"</strong> is changed to <strong id="mrs_01_24510__b98184415297">\"world\"</strong>.</li><li id="mrs_01_24510__li47501141776">Single quotation marks (') in the data must be replaced with the escape character <strong id="mrs_01_24510__b12340323453">\'</strong>. For example, <strong id="mrs_01_24510__b13491193684514">'world'</strong> will be changed to <strong id="mrs_01_24510__b16188151144515">\'world\'</strong>.</li><li id="mrs_01_24510__li4751134114711">If the data to be imported is time, the format is <strong id="mrs_01_24510__b4394287214">yyyy-MM-dd'T'HH:mm:ss</strong>, <strong id="mrs_01_24510__b9687193102115">yyy-MM-dd HH:mm:ss</strong> or <strong id="mrs_01_24510__b8144163562110">yyyy-MM-dd'T'HH:mm:ss.SSSZ</strong>, for example, <strong id="mrs_01_24510__b9432114342115">2022-02-28T11:07:00</strong>, <strong id="mrs_01_24510__b17651546202119">2022-02-28T11:07:00</strong>, or <strong id="mrs_01_24510__b3574165072116">2022-02-28T11:07:00.000Z</strong>.</li></ul>
|
|
</div></div>
|
|
</li><li id="mrs_01_24510__li83055911514">Use WinSCP to import the CSV file to the directory of the node where the client is installed, for example, <strong id="mrs_01_24510__b100127448435126"><span id="mrs_01_24510__ph63641435462">/opt/client</span>/IoTDB/iotdb/sbin</strong>.</li><li id="mrs_01_24510__li1726381018114">Log in to the node where the client is installed as the client installation user.</li><li id="mrs_01_24510__li1926341081117">Run the following command to switch to the client installation directory:<p id="mrs_01_24510__p182632101119"><a name="mrs_01_24510__li1926341081117"></a><a name="li1926341081117"></a><strong id="mrs_01_24510__b8263110121119">cd <span id="mrs_01_24510__ph1991791164613">/opt/client</span></strong></p>
|
|
</li><li id="mrs_01_24510__li1126312102112">Run the following command to configure environment variables:<p id="mrs_01_24510__p18263210141112"><a name="mrs_01_24510__li1126312102112"></a><a name="li1126312102112"></a><strong id="mrs_01_24510__b1126361081111">source bigdata_env</strong></p>
|
|
</li><li id="mrs_01_24510__li16611562035"><a name="mrs_01_24510__li16611562035"></a><a name="li16611562035"></a>(Optional) Perform this step to authenticate the current user if Kerberos authentication is enabled for the cluster. If Kerberos authentication is not enabled, skip this step.<p id="mrs_01_24510__p1913534557"><a name="mrs_01_24510__li16611562035"></a><a name="li16611562035"></a><strong id="mrs_01_24510__b192631310141113">kinit</strong> <em id="mrs_01_24510__i52638104119">Component service user</em></p>
|
|
</li><li id="mrs_01_24510__li22721597459">Run the following command to switch to the directory where the IoTDB client running script is stored:<p id="mrs_01_24510__p11263181041120"><a name="mrs_01_24510__li22721597459"></a><a name="li22721597459"></a><strong id="mrs_01_24510__b726351031117">cd <span id="mrs_01_24510__ph1749561964619">/opt/client</span>/IoTDB/iotdb/sbin</strong></p>
|
|
</li><li id="mrs_01_24510__li05599513462"><a name="mrs_01_24510__li05599513462"></a><a name="li05599513462"></a>Run the following command to log in to the client:<p id="mrs_01_24510__p25591518467"><a name="mrs_01_24510__li05599513462"></a><a name="li05599513462"></a><strong id="mrs_01_24510__b167723204035126">./start-cli.sh</strong> <strong id="mrs_01_24510__b39696254535126">-h</strong> <em id="mrs_01_24510__i136593282035126">Service IP address of the IoTDBServer instance node</em> <strong id="mrs_01_24510__b136751112635126">-p</strong> <em id="mrs_01_24510__i198603776935126">IoTDBServer RPC port</em></p>
|
|
<div class="note" id="mrs_01_24510__note255916513468"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_24510__ul34851622165212"><li id="mrs_01_24510__li136201025135219">You can log in to FusionInsight Manager and choose <strong id="mrs_01_24510__b126239193335126">Cluster</strong> > <strong id="mrs_01_24510__b15471407735126">Services</strong> > <strong id="mrs_01_24510__b69462018335126">IoTDB</strong> > <strong id="mrs_01_24510__b151045620935126">Instance</strong> to view the service IP address of the IoTDBServer instance node.</li><li id="mrs_01_24510__li04714595535">The default RPC port number is <strong id="mrs_01_24510__b77774870235126">22260</strong>. To obtain the port number, choose <strong id="mrs_01_24510__b205884288835126">Cluster</strong> > <strong id="mrs_01_24510__b8648514935126">Services</strong> > <strong id="mrs_01_24510__b192406103535126">IoTDB</strong>, choose <strong id="mrs_01_24510__b12814435126">Configurations</strong> > <strong id="mrs_01_24510__b110624580335126">All Configurations</strong>, and search for <strong id="mrs_01_24510__b203350509935126">rpc_port</strong>.</li></ul>
|
|
</div></div>
|
|
<p id="mrs_01_24510__p119591212583">After you run this command, specify the service username as required.</p>
|
|
<ul id="mrs_01_24510__ul9773644154020"><li id="mrs_01_24510__li1677324418404">To specify the service username, enter <strong id="mrs_01_24510__b11745112913212">yes</strong> and enter the service username and password as prompted.<p id="mrs_01_24510__p172381314553"><span><img id="mrs_01_24510__image49705115425" src="en-us_image_0000001532951928.png"></span></p>
|
|
</li></ul>
|
|
<ul id="mrs_01_24510__ul144912529459"><li id="mrs_01_24510__li34918526456">If you will not specify the service username, enter <strong id="mrs_01_24510__b1586192716569">no</strong>. In this case, you will perform subsequent operations as the user in <a href="#mrs_01_24510__li16611562035">6</a>.<p id="mrs_01_24510__p1449113415542"><span><img id="mrs_01_24510__image165962644411" src="en-us_image_0000001583272201.png"></span></p>
|
|
</li><li id="mrs_01_24510__li71731136114619">If you enter other information, you will log out.<p id="mrs_01_24510__p65611320419"><a name="mrs_01_24510__li71731136114619"></a><a name="li71731136114619"></a><span><img id="mrs_01_24510__image1824155773715" src="en-us_image_0000001582952133.png"></span></p>
|
|
</li></ul>
|
|
</li><li id="mrs_01_24510__li10886333161417">(Optional) Create metadata.<div class="p" id="mrs_01_24510__p5886333171412"><a name="mrs_01_24510__li10886333161417"></a><a name="li10886333161417"></a>IoTDB has the capability of type inference, so it is not necessary to create metadata before data import. However, it is recommended that you create metadata before using the CSV tool to import data, because this avoids unnecessary type conversion errors. The commands are as follows:<pre class="screen" id="mrs_01_24510__screen13101185252710">SET STORAGE GROUP TO root.fit.d1;
|
|
SET STORAGE GROUP TO root.fit.d2;
|
|
SET STORAGE GROUP TO root.fit.p;
|
|
CREATE TIMESERIES root.fit.d1.s1 WITH DATATYPE=INT32,ENCODING=RLE;
|
|
CREATE TIMESERIES root.fit.d1.s2 WITH DATATYPE=TEXT,ENCODING=PLAIN;
|
|
CREATE TIMESERIES root.fit.d2.s1 WITH DATATYPE=INT32,ENCODING=RLE;
|
|
CREATE TIMESERIES root.fit.d2.s3 WITH DATATYPE=INT32,ENCODING=RLE;
|
|
CREATE TIMESERIES root.fit.p.s1 WITH DATATYPE=INT32,ENCODING=RLE;</pre>
|
|
</div>
|
|
</li><li id="mrs_01_24510__li7282121182820">Run the following command to exit the client:<p id="mrs_01_24510__p58227518177"><a name="mrs_01_24510__li7282121182820"></a><a name="li7282121182820"></a><strong id="mrs_01_24510__b1084855816164">quit;</strong></p>
|
|
</li><li id="mrs_01_24510__li1451612116172">Run the following command to switch to the directory where the <strong id="mrs_01_24510__b116377614465">import-csv.sh</strong> script is stored:<p id="mrs_01_24510__p10531444151716"><strong id="mrs_01_24510__b1053114410177">cd <span id="mrs_01_24510__ph16933327194613">/opt/client</span>/IoTDB/iotdb/tools</strong></p>
|
|
</li><li id="mrs_01_24510__li87641654151912">Run the following command to run <strong id="mrs_01_24510__b105341325204619">import-csv.sh</strong> and import the <strong id="mrs_01_24510__b3519406466">example-filename.csv</strong> file:<p id="mrs_01_24510__p104011091517"><strong id="mrs_01_24510__b8803171314510">./import-csv.sh -h</strong> <em id="mrs_01_24510__i82599450495">Service IP address of the IoTDBServer instance</em> <strong id="mrs_01_24510__b20803151364512">-p</strong><em id="mrs_01_24510__i12645164620506">IoTDBServer RPC port</em> <strong id="mrs_01_24510__b19804171319457">-f </strong><em id="mrs_01_24510__i1580481374514">example-filename.csv</em></p>
|
|
<p id="mrs_01_24510__p15913101411336">Enter the service username and password in interactive mode as prompted. If information in the following figure is displayed, the CSV file is imported:</p>
|
|
<p id="mrs_01_24510__p17867248568"><span><img id="mrs_01_24510__image438417119564" src="en-us_image_0000001532792008.png"></span></p>
|
|
</li><li id="mrs_01_24510__li14472184010">Verify data consistency.<ol type="a" id="mrs_01_24510__ol1790417481607"><li id="mrs_01_24510__li8360331117">Run the following command to switch to the directory where the IoTDB client running script is stored:<p id="mrs_01_24510__p736014319118"><a name="mrs_01_24510__li8360331117"></a><a name="li8360331117"></a><strong id="mrs_01_24510__b1936063716">cd <span id="mrs_01_24510__ph1456473618466">/opt/client</span>/IoTDB/iotdb/sbin</strong></p>
|
|
</li><li id="mrs_01_24510__li1050694618016">Log in to the IoTDB client by referring to <a href="#mrs_01_24510__li05599513462">8</a>. Run SQL statements to query data and compare the data with that in the <a href="#mrs_01_24510__li4604150164718">1</a> file.</li><li id="mrs_01_24510__li1885220521313">Check whether the imported data is consistent with the data in the <a href="#mrs_01_24510__li4604150164718">1</a>. If they are, the import is successful.<p id="mrs_01_24510__p8734185914314">Run the following command to check the imported data:</p>
|
|
<p id="mrs_01_24510__p686819521219"><strong id="mrs_01_24510__b1430213819317">SELECT * FROM root.fit.**;</strong></p>
|
|
<div class="p" id="mrs_01_24510__p102366397584"><span><img id="mrs_01_24510__image179941636175710" src="en-us_image_0000001532951944.png"></span><div class="note" id="mrs_01_24510__note138692520119"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_24510__ul128693521210"><li id="mrs_01_24510__li386875213117">To prevent security risks, you are advised to import CSV files in interactive mode.</li><li id="mrs_01_24510__li108693521213">You can also import CSV files by running the <strong id="mrs_01_24510__b725514425">./import-csv.sh -h</strong> <em id="mrs_01_24510__i10159180401">Service IP address of the IoTDBServer instance</em> <strong id="mrs_01_24510__b125516427">-p</strong> <em id="mrs_01_24510__i19538202018">IoTDBServer RPC port</em> <strong id="mrs_01_24510__b11251111426">-u</strong> <em id="mrs_01_24510__i15645103120014">Service username</em> <strong id="mrs_01_24510__b20258184210">-pw</strong> <em id="mrs_01_24510__i1642323819011">Service user password</em><strong id="mrs_01_24510__b1425101114211">-f </strong><em id="mrs_01_24510__i14258134212">example-filename.csv</em> command.<p id="mrs_01_24510__p1786919521212">If information in the following figure is displayed, the CSV file is imported.</p>
|
|
<p id="mrs_01_24510__p917751613117"><span><img id="mrs_01_24510__image10982203920011" src="en-us_image_0000001583151917.png"></span></p>
|
|
</li><li id="mrs_01_24510__li98019173447">If nanosecond (ns) time precision is enabled for the IoTDB on the server, the <strong id="mrs_01_24510__b9912192319458">-tp ns</strong> parameter needs to be added when the client imports data with the nanosecond timestamp. To check whether nanosecond time precision is enabled for a cluster, log in to FusionInsight Manager, choose <strong id="mrs_01_24510__b113164815013">Cluster</strong> > <strong id="mrs_01_24510__b013631513015">Configurations</strong> > <strong id="mrs_01_24510__b823214145517">All Non-default Values</strong>, and search for <strong id="mrs_01_24510__b12748428035">timestamp_precision</strong>.</li></ul>
|
|
</div></div>
|
|
</div>
|
|
</li></ol>
|
|
</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24509.html">IoTDB Data Import and Export</a></div>
|
|
</div>
|
|
</div>
|
|
|