forked from docs/doc-exports
Su, Xiaomeng
483e5cc6cc
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
142 lines
12 KiB
HTML
142 lines
12 KiB
HTML
<a name="dli_02_0309"></a><a name="dli_02_0309"></a>
|
|
|
|
<h1 class="topictitle1">Creating and Submitting a Spark Job</h1>
|
|
<div id="body1602473883610"><div class="section" id="dli_02_0309__section851316282599"><h4 class="sectiontitle">Scenario Description</h4><p id="dli_02_0309__p1822151903">This section describes how to create and submit Spark jobs using APIs.</p>
|
|
</div>
|
|
<div class="section" id="dli_02_0309__section155144213214"><h4 class="sectiontitle">Constraints</h4><ul id="dli_02_0309__ul759612191669"><li id="dli_02_0309__li45963199610">It takes 6 to 10 minutes to start a job using a new queue for the first time.</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_02_0309__section13856162971"><h4 class="sectiontitle">Involved APIs</h4><ul id="dli_02_0309__ul1862251633812"><li id="dli_02_0309__li462221673818"><a href="dli_02_0194.html">Creating a Queue</a>: Create a queue.</li><li id="dli_02_0309__li46223161388"><a href="dli_02_0130.html">Uploading a Package Group (Discarded)</a>: Upload the resource package required by the Spark job.</li><li id="dli_02_0309__li1137901761318"><a href="dli_02_0172.html">Querying Resource Packages in a Group (Discarded)</a>: Check whether the uploaded resource package is correct.</li><li id="dli_02_0309__li9641746152414"><a href="dli_02_0124.html">Creating a Batch Processing Job</a>: Create and submit a Spark batch processing job.</li><li id="dli_02_0309__li16848507143"><a href="dli_02_0127.html">Querying a Batch Job Status</a>: View the status of a batch processing job.</li><li id="dli_02_0309__li093312301692"><a href="dli_02_0128.html">Querying Batch Job Logs (Discarded)</a>: View batch processing job logs.</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_02_0309__section2742155213719"><h4 class="sectiontitle">Procedure</h4><ol id="dli_02_0309__ol454620011213"><li id="dli_02_0309__li135461061218">Create a common queue. For details, see <a href="dli_02_0307.html">Creating a Queue</a>.</li><li id="dli_02_0309__li117291344122510"><a name="dli_02_0309__li117291344122510"></a><a name="li117291344122510"></a>Upload a package group.<ul id="dli_02_0309__ul2729144122513"><li id="dli_02_0309__li57278446258">API<p id="dli_02_0309__p7727134452510"><a name="dli_02_0309__li57278446258"></a><a name="li57278446258"></a>URI format: POST /v2.0/{<em id="dli_02_0309__i59859124111">project_id</em>}/resources</p>
|
|
<ul id="dli_02_0309__ul14727644162512"><li id="dli_02_0309__li9727134442515">Obtain the value of {<em id="dli_02_0309__i038414794218">project_id</em>} from <a href="dli_02_0183.html">Obtaining a Project ID</a>.</li><li id="dli_02_0309__li1372710448250">For details about the request parameters, see <a href="dli_02_0130.html">Uploading a Package Group (Discarded)</a>.</li></ul>
|
|
</li><li id="dli_02_0309__li5728744172513">Request example<ul id="dli_02_0309__ul772864412254"><li id="dli_02_0309__li17728194415253">Description: Upload resources in the GATK group to the project whose ID is <strong id="dli_02_0309__b119371132154210">48cc2c48765f481480c7db940d6409d1</strong>.</li><li id="dli_02_0309__li137287448256">Example URL: POST https://{<em id="dli_02_0309__i1535683915422">endpoint</em>}/v2.0/48cc2c48765f481480c7db940d6409d1/resources</li><li id="dli_02_0309__li14728144414252">Body:<pre class="screen" id="dli_02_0309__screen672864492517">{
|
|
"paths": [
|
|
"https://test.obs.xxx.com/txr_test/jars/spark-sdv-app.jar"
|
|
],
|
|
"kind": "jar",
|
|
"group": "gatk",
|
|
"is_async":"true"
|
|
}</pre>
|
|
</li></ul>
|
|
</li><li id="dli_02_0309__li19729134452514">Example response<pre class="screen" id="dli_02_0309__en-us_topic_0142813183_en-us_topic_0103345073_screen281210137181">{
|
|
"group_name": "gatk",
|
|
"status": "READY",
|
|
"resources": [
|
|
"spark-sdv-app.jar",
|
|
"wordcount",
|
|
"wordcount.py"
|
|
],
|
|
"details": [
|
|
{
|
|
"create_time": 0,
|
|
"update_time": 0,
|
|
"resource_type": "jar",
|
|
"resource_name": "spark-sdv-app.jar",
|
|
"status": "READY",
|
|
"underlying_name": "987e208d-d46e-4475-a8c0-a62f0275750b_spark-sdv-app.jar"
|
|
},
|
|
{
|
|
"create_time": 0,
|
|
"update_time": 0,
|
|
"resource_type": "jar",
|
|
"resource_name": "wordcount",
|
|
"status": "READY",
|
|
"underlying_name": "987e208d-d46e-4475-a8c0-a62f0275750b_wordcount"
|
|
},
|
|
{
|
|
"create_time": 0,
|
|
"update_time": 0,
|
|
"resource_type": "jar",
|
|
"resource_name": "wordcount.py",
|
|
"status": "READY",
|
|
"underlying_name": "987e208d-d46e-4475-a8c0-a62f0275750b_wordcount.py"
|
|
}
|
|
],
|
|
"create_time": 1551334579654,
|
|
"update_time": 1551345369070
|
|
}</pre>
|
|
</li></ul>
|
|
</li><li id="dli_02_0309__li970315312304"><a name="dli_02_0309__li970315312304"></a><a name="li970315312304"></a>View resource packages in a group.<ul id="dli_02_0309__ul470310318306"><li id="dli_02_0309__li117031316309">API<p id="dli_02_0309__p0703173113019"><a name="dli_02_0309__li117031316309"></a><a name="li117031316309"></a>URI format: GET /v2.0/{<em id="dli_02_0309__i1518871612432">project_id</em>}/resources/{<em id="dli_02_0309__i935602212434">resource_name</em>}</p>
|
|
<ul id="dli_02_0309__ul187037363019"><li id="dli_02_0309__li370323113013">Obtain the value of {project_id} from <a href="dli_02_0183.html">Obtaining a Project ID</a>.</li><li id="dli_02_0309__li1670316318304">For details about the query parameters, see <a href="dli_02_0034.html">Creating a Table (Discarded)</a>.</li></ul>
|
|
</li><li id="dli_02_0309__li157031431301">Request example<ul id="dli_02_0309__ul1270319353013"><li id="dli_02_0309__li1070313113020">Description: Query the resource package named <strong id="dli_02_0309__b1843712370432">luxor-router-1.1.1.jar</strong> in the GATK group under the project whose ID is <strong id="dli_02_0309__b1548674612436">48cc2c48765f481480c7db940d6409d1</strong>.</li><li id="dli_02_0309__li870443193015">Example URL: GET https://{<em id="dli_02_0309__i43611256154315">endpoint</em>}/v2.0/48cc2c48765f481480c7db940d6409d1/resources/luxor-router-1.1.1.jar?group=gatk</li><li id="dli_02_0309__li270410303012">Body:<pre class="screen" id="dli_02_0309__screen77042314309">{}</pre>
|
|
</li></ul>
|
|
</li><li id="dli_02_0309__li17047343019">Example response<pre class="screen" id="dli_02_0309__screen137049318303">{
|
|
"create_time": 1522055409139,
|
|
"update_time": 1522228350501,
|
|
"resource_type": "jar",
|
|
"resource_name": "luxor-router-1.1.1.jar",
|
|
"status": "uploading",
|
|
"underlying_name": "7885d26e-c532-40f3-a755-c82c442f19b8_luxor-router-1.1.1.jar",
|
|
"owner": "****"
|
|
}</pre>
|
|
</li></ul>
|
|
</li><li id="dli_02_0309__li125211016104016">Create and submit a Spark batch processing job.<ul id="dli_02_0309__ul26571210114414"><li id="dli_02_0309__li146576102444">API<p id="dli_02_0309__p10657111019442"><a name="dli_02_0309__li146576102444"></a><a name="li146576102444"></a>URI format: POST /v2.0/{<em id="dli_02_0309__i250562054414">project_id</em>}/batches</p>
|
|
<ul id="dli_02_0309__ul56581107449"><li id="dli_02_0309__li17658191013442">Obtain the value of {<em id="dli_02_0309__i39711225144414">project_id</em>} from <a href="dli_02_0183.html">Obtaining a Project ID</a>.</li><li id="dli_02_0309__li3658201018442">For details about the request parameters, see <a href="dli_02_0124.html">Creating a Batch Processing Job</a>.</li></ul>
|
|
</li><li id="dli_02_0309__li10658410164410">Request example<ul id="dli_02_0309__ul14658161015442"><li id="dli_02_0309__li865851014449">Description: In the <strong id="dli_02_0309__b19641184374419">48cc2c48765f481480c7db940d6409d1</strong> project, create a batch processing job named <strong id="dli_02_0309__b1727784914443">TestDemo4</strong> in <strong id="dli_02_0309__b15624205154410">queue1</strong>.</li><li id="dli_02_0309__li6658171074411">Example URL: POST https://{<em id="dli_02_0309__i11456105917449">endpoint</em>}/v2.0/48cc2c48765f481480c7db940d6409d1/batches</li><li id="dli_02_0309__li165811084419">Body:<pre class="screen" id="dli_02_0309__screen3658131064420">{
|
|
"sc_type": "A",
|
|
"jars": [
|
|
|
|
"spark-examples_2.11-2.1.0.luxor.jar"
|
|
],
|
|
"driverMemory": "1G",
|
|
"driverCores": 1,
|
|
"executorMemory": "1G",
|
|
"executorCores": 1,
|
|
"numExecutors": 1,
|
|
"queue": "cce_general",
|
|
"file":
|
|
"spark-examples_2.11-2.1.0.luxor.jar",
|
|
"className":
|
|
"org.apache.spark.examples.SparkPi",
|
|
"minRecoveryDelayTime": 10000,
|
|
"maxRetryTimes": 20
|
|
}</pre>
|
|
</li></ul>
|
|
</li><li id="dli_02_0309__li2065912101446">Example response<pre class="screen" id="dli_02_0309__screen765961024418">{
|
|
"id": "07a3e4e6-9a28-4e92-8d3f-9c538621a166",
|
|
"appId": "",
|
|
"name": "",
|
|
"owner": "test1",
|
|
"proxyUser": "",
|
|
"state": "starting",
|
|
"kind": "",
|
|
"log": [],
|
|
"sc_type": "CUSTOMIZED",
|
|
"cluster_name": "aaa",
|
|
"queue": "aaa",
|
|
"create_time": 1607589874156,
|
|
"update_time": 1607589874156
|
|
}</pre>
|
|
</li></ul>
|
|
</li><li id="dli_02_0309__li18904826102815">Query a batch job status.<ul id="dli_02_0309__ul178791154132816"><li id="dli_02_0309__li1187919544287">API<p id="dli_02_0309__p198791154192815"><a name="dli_02_0309__li1187919544287"></a><a name="li1187919544287"></a>URI format: GET /v2.0/{<em id="dli_02_0309__i422292820451">project_id</em>}/batches/{<em id="dli_02_0309__i12646236184514">batch_id</em>}/state</p>
|
|
<ul id="dli_02_0309__ul158791254132815"><li id="dli_02_0309__li7879854172815">Obtain the value of {<em id="dli_02_0309__i181011740194511">project_id</em>} from <a href="dli_02_0183.html">Obtaining a Project ID</a>.</li><li id="dli_02_0309__li13879115411285">For details about the query parameters, see <a href="dli_02_0127.html">Querying a Batch Job Status</a>.</li></ul>
|
|
</li><li id="dli_02_0309__li7879185418289">Request example<ul id="dli_02_0309__ul1287995452812"><li id="dli_02_0309__li118791554122819">Description: Query the status of the batch processing job whose ID is <strong id="dli_02_0309__b113729541459">0a324461-d9d9-45da-a52a-3b3c7a3d809e</strong> in the project whose ID is <strong id="dli_02_0309__b2017419585451">48cc2c48765f481480c7db940d6409d1</strong>.</li><li id="dli_02_0309__li6879145416284">Example URL: GET https://{<em id="dli_02_0309__i20543197144617">endpoint</em>}/v2.0/48cc2c48765f481480c7db940d6409d1/batches/0a324461-d9d9-45da-a52a-3b3c7a3d809e/state</li><li id="dli_02_0309__li16880154182814">Body:<pre class="screen" id="dli_02_0309__screen488075411287">{}</pre>
|
|
</li></ul>
|
|
</li><li id="dli_02_0309__li198801754192819">Example response<pre class="screen" id="dli_02_0309__screen78801654182815">{
|
|
"id":"0a324461-d9d9-45da-a52a-3b3c7a3d809e",
|
|
"state":"Success"
|
|
}</pre>
|
|
</li></ul>
|
|
</li><li id="dli_02_0309__li598033665816">Query batch job logs.<ul id="dli_02_0309__ul2098015362589"><li id="dli_02_0309__li1098014362582">API<p id="dli_02_0309__p898016363588"><a name="dli_02_0309__li1098014362582"></a><a name="li1098014362582"></a>URI format: GET /v2.0/{<em id="dli_02_0309__i17641236194619">project_id</em>}/batches/{<em id="dli_02_0309__i19639173916462">batch_id</em>}/log</p>
|
|
<ul id="dli_02_0309__ul17980936185818"><li id="dli_02_0309__li89808369587">Obtain the value of {<em id="dli_02_0309__i20819245154615">project_id</em>} from <a href="dli_02_0183.html">Obtaining a Project ID</a>.</li><li id="dli_02_0309__li1198014369585">For details about the query parameters, see <a href="dli_02_0128.html">Querying Batch Job Logs (Discarded)</a>.</li></ul>
|
|
</li><li id="dli_02_0309__li998073675817">Request example<ul id="dli_02_0309__ul29801736125817"><li id="dli_02_0309__li10980183685815">Description: Query the background logs of the batch processing job <strong id="dli_02_0309__b1210299114712">0a324461-d9d9-45da-a52a-3b3c7a3d809e</strong> in the <strong id="dli_02_0309__b4300161518472">48cc2c48765f481480c7db940d6409d1</strong> project.</li><li id="dli_02_0309__li15980193614588">Example URL: GET https://{<em id="dli_02_0309__i11896921114713">endpoint</em>}/v2.0/48cc2c48765f481480c7db940d6409d1/batches/0a324461-d9d9-45da-a52a-3b3c7a3d809e/log</li><li id="dli_02_0309__li7981336145810">Body:<pre class="screen" id="dli_02_0309__screen598153655812">{}</pre>
|
|
</li></ul>
|
|
</li><li id="dli_02_0309__li1498133616589">Example response<pre class="screen" id="dli_02_0309__screen9981123618585">{
|
|
"id": "0a324461-d9d9-45da-a52a-3b3c7a3d809e",
|
|
"from": 0,
|
|
"total": 3,
|
|
"log": [
|
|
"Detailed information about job logs"
|
|
]
|
|
}</pre>
|
|
</li></ul>
|
|
</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_02_0306.html">Getting Started</a></div>
|
|
</div>
|
|
</div>
|
|
|