forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
24 lines
2.4 KiB
HTML
24 lines
2.4 KiB
HTML
<a name="mrs_01_1755"></a><a name="mrs_01_1755"></a>
|
|
|
|
<h1 class="topictitle1">How to Perform Operations on Local Files with Hive User-Defined Functions</h1>
|
|
<div id="body1595924880902"><div class="section" id="mrs_01_1755__s214c9940311449ceb9eb457da27ffbe3"><h4 class="sectiontitle">Question</h4><p id="mrs_01_1755__a58e54ac6ec9f41e5be83f347f77ed9d4">How to perform operations on local files (such as reading the content of a file) with Hive user-defined functions?</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1755__s9269a11c6cbd431787d9ce769058b21b"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_1755__aa40652ea99144ac7be7834777cae627d">By default, you can perform operations on local files with their relative paths in UDF. The following are sample codes:</p>
|
|
<pre class="screen" id="mrs_01_1755__se1dbc3daaf3749d682958823b58e79e0">public String evaluate(String text) {
|
|
// some logic
|
|
File file = new File("foo.txt");
|
|
// some logic
|
|
// do return here
|
|
}</pre>
|
|
<p id="mrs_01_1755__acfaa355e51dc46098d02bcc72c96ac7f">In Hive, upload the file <span class="filepath" id="mrs_01_1755__f1afe78f614954e9289546f193833ab1a"><b>foo.txt</b></span> used in UDF to HDFS, such as <span class="filepath" id="mrs_01_1755__fb4be0ebcc2bc48c7a531f598309a8df5"><b>hdfs://hacluster/tmp/foo.txt</b></span>. You can perform operations on the <span class="filepath" id="mrs_01_1755__faaa0b3bd25864fd78c44c5620d26b841"><b>foo.txt</b></span> file by creating UDF with the following sentences:</p>
|
|
<p id="mrs_01_1755__p51051842113916"><strong id="mrs_01_1755__b99053814018">create function testFunc as 'some.class' using jar 'hdfs://hacluster/somejar.jar', file 'hdfs://hacluster/tmp/foo.txt';</strong></p>
|
|
<p id="mrs_01_1755__en-us_topic_0116216710_p961631691913">In abnormal cases, if the value of <span class="parmname" id="mrs_01_1755__pbcabd41b69124169bf60564a888bc083"><b>hive.fetch.task.conversion</b></span> is <span class="parmvalue" id="mrs_01_1755__p0e7c03ef30bf4e3b873e6a52e0bbb2a1"><b>more</b></span>, you can perform operations on local files in UDF by using absolute path instead of relative path. In addition, you must ensure that the file exists on all HiveServer nodes and NodeManager nodes and <strong id="mrs_01_1755__a884ea8fdc01d4589932ddebb07103bcd">omm</strong> user have corresponding operation rights.</p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1752.html">Common Issues About Hive</a></div>
|
|
</div>
|
|
</div>
|
|
|