forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
27 lines
3.1 KiB
HTML
27 lines
3.1 KiB
HTML
<a name="mrs_01_2034"></a><a name="mrs_01_2034"></a>
|
|
|
|
<h1 class="topictitle1">Why Is a Task Suspended When the ANALYZE TABLE Statement Is Executed and Resources Are Insufficient?</h1>
|
|
<div id="body1595920221841"><div class="section" id="mrs_01_2034__s185b515f46134c85bc37ec3c5a176ce5"><h4 class="sectiontitle">Question</h4><p id="mrs_01_2034__a3794b038efb5457a9b04ea6ee3d337d9">When the <i><b><span class="cmdname" style="font-family:Arial" id="mrs_01_2034__cmdname177383410279">analyze table</span></b></i> statement is executed using spark-sql, the task is suspended and the information below is displayed. Why?</p>
|
|
<pre class="screen" id="mrs_01_2034__se3c7e676830548579ab77c8af0240671">spark-sql> analyze table hivetable2 compute statistics;
|
|
Query ID = root_20160716174218_90f55869-000a-40b4-a908-533f63866fed
|
|
Total jobs = 1
|
|
Launching Job 1 out of 1
|
|
Number of reduce tasks is set to 0 since there's no reduce operator
|
|
16/07/20 17:40:56 WARN JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
|
|
Starting Job = job_1468982600676_0002, Tracking URL = http://10-120-175-107:8088/proxy/application_1468982600676_0002/
|
|
Kill Command = /opt/hadoopclient/HDFS/hadoop/bin/hadoop job -kill job_1468982600676_0002</pre>
|
|
</div>
|
|
<div class="section" id="mrs_01_2034__s6350699b9f8c46279effa3011b5ba830"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_2034__a68d03769c1534ae5b652394763907703">When the statement is executed, the SQL statement starts the <i><b><span class="cmdname" style="font-family:Arial" id="mrs_01_2034__cmdname6958184782719">analyze table hivetable2 compute statistics</span></b></i> MapReduce tasks. On the ResourceManager Web UI of Yarn, the task is not executed due to insufficient resources. As a result, the task is suspended.</p>
|
|
<div class="fignone" id="mrs_01_2034__fig2992172914451"><span class="figcap"><b>Figure 1 </b>ResourceManager Web UI</span><br><span><img id="mrs_01_2034__image193461717204511" src="en-us_image_0000001388066504.png"></span></div>
|
|
</div>
|
|
<p id="mrs_01_2034__ae8fbdf37992843688e77447e982bc6d8">You are advised to add <strong id="mrs_01_2034__b187562175493055">noscan</strong> when running the <i><b><span class="cmdname" style="font-family:Arial" id="mrs_01_2034__cmdname1015133372912">analyze table</span></b></i> statement. The function of this statement is the same as that of the <i><b><span class="cmdname" style="font-family:Arial" id="mrs_01_2034__cmdname3951243142919">analyze table hivetable2 compute statistics</span></b></i> statement. The command is as follows:</p>
|
|
<pre class="screen" id="mrs_01_2034__s83ebeb307c3d42c6998d9c60d1ebe2d4">spark-sql> analyze table hivetable2 compute statistics noscan</pre>
|
|
<p id="mrs_01_2034__aa50cf387b7aa4447877968ef8fff7e10">This command does not start MapReduce tasks and does not occupy Yarn resources. Therefore, the tasks can be executed.</p>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_2022.html">Spark SQL and DataFrame</a></div>
|
|
</div>
|
|
</div>
|
|
|