forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
22 lines
3.8 KiB
HTML
22 lines
3.8 KiB
HTML
<a name="mrs_01_2027"></a><a name="mrs_01_2027"></a>
|
|
|
|
<h1 class="topictitle1">Why Do I Fail to Delete the UDF Using Another Service?</h1>
|
|
<div id="body1595920220609"><div class="section" id="mrs_01_2027__s4f96710600e246a3bc66831c593d9e50"><h4 class="sectiontitle">Question</h4><p id="mrs_01_2027__a0c12bacfc506404cbac6ec56ede8b00c">Why do I fail to delete the UDF using another service, for example, delete the UDF created by Hive using Spark SQL.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_2027__s7ca44a54afb64ce792b7159cf4a2bcaf"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_2027__a82614c584a3f479f931cb95a12872685">The UDF can be created using any of the following services:</p>
|
|
<ol id="mrs_01_2027__o9872c3ba41504928a90c996ebf5ea003"><li id="mrs_01_2027__l6c3d36a1921e404bafb24a3ca9aad2f9">Hive client.</li><li id="mrs_01_2027__l4832956e2343448e9c8991ca6690b860">JDBCServer API. You can connect JDBCServer to Spark Beeline or JDBC client code, and run SQL statements to create the UDF.</li><li id="mrs_01_2027__l557c30de5f904f61ae92dd21541e2e8d">spark-sql.</li></ol>
|
|
<p id="mrs_01_2027__a4fc55251d27843ddb714ba9746974299">The scenarios in which the UDF failed to be deleted may be as follows:</p>
|
|
<ul id="mrs_01_2027__ua2b02b5eece142c4bc08af68837f6380"><li id="mrs_01_2027__l2246296afee4402ebbd834795fb3e9b4">If you use Spark Beeline to delete the UDF created by other services, you must restart the JDBCServer before the deletion. Otherwise, the deletion fails. If you use spark-sql to delete the UDF created by other services, you must restart the spark-sql before the deletion. Otherwise, the deletion fails.<p id="mrs_01_2027__a24a970d0fb4340b3b18cde8bdd696dfa"><a name="mrs_01_2027__l2246296afee4402ebbd834795fb3e9b4"></a><a name="l2246296afee4402ebbd834795fb3e9b4"></a>Cause: After the UDF is created, if the JDBCServer or the spark-sql has not been restarted, the newly created UDF will not be saved by the FunctionRegistry object in the thread where Spark locates. As a result, the UDF failed to be deleted.</p>
|
|
<p id="mrs_01_2027__ab94f11bca1104239a67fcdd0bda16120">Solution: Restart the JDBCServer and spark-sql of the Spark client and delete the UDF.</p>
|
|
</li><li id="mrs_01_2027__l5a54f0b9888f4a01a610cc977767ab9a">When creating UDF on the Hive client, the <b><span class="cmdname" id="mrs_01_2027__cmdname132001746122510">add jar</span></b> command (e.g. <b><span class="cmdname" id="mrs_01_2027__cmdname15602211241">add jar /opt/test/two_udfs.jar</span></b>) is used to add the <strong id="mrs_01_2027__ad3d7f593c88d4cf9adc8040723d9d2a5">.jar</strong> package instead of specifying the path of <strong id="mrs_01_2027__a35992ba8990046c3aa1b2d984cb23e08">.jar</strong> package in creating UDF statement. As a result, the <strong id="mrs_01_2027__a41883cad74ba4b869c36329f11b35bfb">ClassNotfound </strong>error occurs when you use other services to delete the UDF.<p id="mrs_01_2027__a0b8a7773325048c583ca5b847ac01bed">Cause: When you use a service to delete the UDF, the service will load the class that corresponds to the UDF to obtain the UDF. However, the <strong id="mrs_01_2027__a6e81e3b0f7ab4198817e8717f660297b">.jar</strong> package is added by the <i><b><span class="cmdname" style="font-family:Arial" id="mrs_01_2027__cee415823bd784e588b7d491d36230e05">add jar</span></b></i> command and jar package does not exist in the classpath of other services. As a result, the <strong id="mrs_01_2027__ad124276420f54f308e984d93cefa5083">ClassNotfound</strong> error occurs and the UDF failed to be deleted.</p>
|
|
<p id="mrs_01_2027__aab79ff07744b468cb8bcf4e519fcc5fb">Solution: The UDF created using the preceding approach must be deleted using the same approach. No other approaches are allowed.</p>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_2022.html">Spark SQL and DataFrame</a></div>
|
|
</div>
|
|
</div>
|
|
|