forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
38 lines
15 KiB
HTML
38 lines
15 KiB
HTML
<a name="mrs_01_1939"></a><a name="mrs_01_1939"></a>
|
|
|
|
<h1 class="topictitle1">Configuring Permissions for SparkSQL to Use Other Components</h1>
|
|
<div id="body1595920206236"><div class="section" id="mrs_01_1939__s2f5b2ba3a809410e8d6b89bb7034e8a6"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1939__a190935ede0064f1eb635dc8455540ba6">SparkSQL may need to be associated with other components. For example, Spark on HBase requires HBase permissions. The following describes how to associate SparkSQL with HBase.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1939__s747737774cbf4cbca381b53cf54e93e7"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_1939__u6615cfa93d6342e1a1e9149ec4e344c5"><li id="mrs_01_1939__lbbdc2dd0ee434779b0087db3a91b2e8a">The Spark client has been installed. For example, the installation directory is <strong id="mrs_01_1939__b568912555316">/opt</strong><strong id="mrs_01_1939__b3689175523117"></strong><strong id="mrs_01_1939__b968917557311">/client</strong>.</li><li id="mrs_01_1939__lf61cee0e0ab444b0965fd4d828ca6d07">You have obtained a user account with the <span id="mrs_01_1939__ph533920514111">system </span>administrator permissions, such as <span class="parmname" id="mrs_01_1939__p1a12c340e4ce4868bbd56ef6634e2045"><b>admin</b></span>.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1939__s61062a0cf1304ac685fac634d4b5466d"><h4 class="sectiontitle">Procedure</h4><ul id="mrs_01_1939__u1711213e9ce645dd8ea8815ec9af733e"><li id="mrs_01_1939__la23f144a3e4e4049a217cf02b47b08c6"><strong id="mrs_01_1939__adbfe1c93f63e470b8a0574b9907cf830">Spark on HBase authorization</strong><p id="mrs_01_1939__af8eee67327f14c1381ba294e990ec18e">After the permissions are assigned, you can use statements that are similar to SQL statements to access HBase tables from SparkSQL. The following uses the procedure for assigning a user the permissions to query HBase tables as an example.</p>
|
|
<div class="note" id="mrs_01_1939__n1b31ee3b74914d91bf778ff89357cb9b"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1939__af204aac45a6d4adeb4b912de2564821d">Set <span class="parmname" id="mrs_01_1939__pa06641bd3dcf4de8a4fbdc981bb52a15"><b>spark.yarn.security.credentials.hbase.enabled</b></span> to <span class="parmvalue" id="mrs_01_1939__pdf86a1183cea48ba8b1dd0d5a1712e99"><b>true</b></span>.</p>
|
|
</div></div>
|
|
<ol id="mrs_01_1939__oc4aafe3871c84587986e9172cb17fcac"><li id="mrs_01_1939__l38d34de0ce8941d98aa9a47964b8ef6c">On Manager, create a role, for example, <span class="parmname" id="mrs_01_1939__pc521f844b00e4cfbba95da433b8e6efb"><b>hive_hbase_create</b></span>, and grant the permission to create HBase tables to the role.<p id="mrs_01_1939__a5e26d905293d472eb489835e9c3122bb">In the <strong id="mrs_01_1939__b856023254113138">Configure Resource Permission</strong> table, choose <em id="mrs_01_1939__i1029615744113138">Name of the desired cluster</em> > <strong id="mrs_01_1939__b9606142015519">HBase </strong>> <strong id="mrs_01_1939__b5633224859">HBase Scope</strong> > <strong id="mrs_01_1939__b161861428851">global</strong>. Select <span class="parmname" id="mrs_01_1939__p4169e2d6d9104c5986f90e3d30e4fbd0"><b><span id="mrs_01_1939__text6782154216110">create</span></b></span> of the namespace <span class="parmname" id="mrs_01_1939__p9f191be364fd47e39bba9367b38f86b5"><b>default</b></span>, and click <span class="uicontrol" id="mrs_01_1939__u155141d880c64bbda0a21092a679ebed"><b>OK</b></span>.</p>
|
|
<div class="note" id="mrs_01_1939__n9b2be0e7b03d465f80812062cfd826db"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1939__a090c2fa074bb4465896fdd4e2b5aca77">In this example, the created table is saved in the default database of Hive and has the CREATE permission of the default database. If you save the table to a Hive database other than <strong id="mrs_01_1939__b24904175113138">default</strong>, perform the following operations:</p>
|
|
<p id="mrs_01_1939__ab358a876fedb4b12a30f01f8a8387afc">In the <strong id="mrs_01_1939__b1842179320113138">Configure Resource Permission</strong> table, choose <em id="mrs_01_1939__i227048684113138">Name of the desired cluster</em> > <strong id="mrs_01_1939__b1311560590113138">Hive</strong> > <strong id="mrs_01_1939__b538147614113138">Hive Read Write Privileges</strong>, select <strong id="mrs_01_1939__b641694024113138">CREATE</strong> for the desired database, and click <strong id="mrs_01_1939__b1746804351113138">OK</strong>.</p>
|
|
</div></div>
|
|
</li><li id="mrs_01_1939__laf7f09eb9dbb4b3e9ef44c4b609dd606">On Manager, create a role, for example, <span class="parmname" id="mrs_01_1939__pcc2d81b23e624e69a5e39fb528767fa9"><b>hive_hbase_submit</b></span>, and grant the permission to submit tasks to the Yarn queue.<p id="mrs_01_1939__a00ef07703ede4e37879c8d30e7817a6c">In the <strong id="mrs_01_1939__b1552450634113138">Configure Resource Permission</strong> table, choose <em id="mrs_01_1939__i313772369113138">Name of the desired cluster</em> > <strong id="mrs_01_1939__b591916546117">Yarn </strong>> <strong id="mrs_01_1939__b391017599118">Scheduling Queue</strong> > <strong id="mrs_01_1939__b44531985210">root</strong>. Select <span class="parmname" id="mrs_01_1939__p941de87003bc4e38b9397157743f57d2"><b><span id="mrs_01_1939__text9551124791610">Submit</span></b></span> of <span class="parmname" id="mrs_01_1939__pf253f6ce4f754e09ba887a67fa002981"><b>default</b></span>, and click <span class="uicontrol" id="mrs_01_1939__u71fa1f9bc66c4801af808e4d57bf5730"><b>OK</b></span>.</p>
|
|
</li><li id="mrs_01_1939__le40e88e4cd834abb99357b8a389d98db">On Manager, create a human-machine user, for example, <span class="parmname" id="mrs_01_1939__p1140d70c064b42409a5716ff5d7b1d57"><b>hbase_creates_user</b></span>, add the user to the <span class="parmname" id="mrs_01_1939__p99ea14ccfbf94ae6a9254535cb4c1ab5"><b>hive</b></span> group, and bind the <span class="parmname" id="mrs_01_1939__pab9bf514c6d248179593868e6192eb73"><b>hive_hbase_create</b></span> and <span class="parmname" id="mrs_01_1939__p5f3e67a6ebbb41c8812e5af7d19322e8"><b>hive_hbase_submit</b></span> roles to create SparkSQL and HBase tables.</li><li id="mrs_01_1939__lf9cb962670d548d98c7a6880e607cbab">Log in to the node where the client is installed as the client installation user.</li><li id="mrs_01_1939__l7428aeb2f3b84441861aff7de706e399">Run the following command to configure environment variables:<p id="mrs_01_1939__ab614ac1a6db747f6b57b43ac4bd1c817"><a name="mrs_01_1939__l7428aeb2f3b84441861aff7de706e399"></a><a name="l7428aeb2f3b84441861aff7de706e399"></a><strong id="mrs_01_1939__b185133421419">source /opt/client/bigdata_env</strong></p>
|
|
<p id="mrs_01_1939__p72418224267"><strong id="mrs_01_1939__b220554912436">source /opt/client/Spark2x/component_env</strong></p>
|
|
</li><li id="mrs_01_1939__l6de3616c11ab46629f935658a3ece580">Run the following command to authenticate the user:<p id="mrs_01_1939__abaa7a300f26b461f8bcd310cc93c5376"><a name="mrs_01_1939__l6de3616c11ab46629f935658a3ece580"></a><a name="l6de3616c11ab46629f935658a3ece580"></a><i><b><span class="cmdname" style="font-family:Arial" id="mrs_01_1939__c613cf506eb254af6b94e1f16d50ff42f">kinit hbase_creates_user</span></b></i></p>
|
|
</li><li id="mrs_01_1939__l64eb8cce531443dd8d750e488b490d13">Run the following commands to enter the shell environment on the Spark JDBCServer client:<p id="mrs_01_1939__a9cdb46cf905c48a4a9a99c24158d76f0"><a name="mrs_01_1939__l64eb8cce531443dd8d750e488b490d13"></a><a name="l64eb8cce531443dd8d750e488b490d13"></a><strong id="mrs_01_1939__b8163225195">/opt/client/Spark2x/spark/bin/beeline -u "jdbc:hive2://<zkNode1_IP>:<zkNode1_Port>,<zkNode2_IP>:<zkNode2_Port>,<zkNode3_IP>:<zkNode3_Port>/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=sparkthriftserver2x;user.principal=spark2x/hadoop.</strong><em id="mrs_01_1939__i131684213195"><system domain name></em><strong id="mrs_01_1939__b176974681915">@</strong><em id="mrs_01_1939__i67007671914"><system domain name></em><strong id="mrs_01_1939__b17925181618191">;saslQop=auth-conf;auth=KERBEROS;principal=spark2x/hadoop.</strong><em id="mrs_01_1939__i69271816171914"><system domain name></em><strong id="mrs_01_1939__b589211192195">@</strong><em id="mrs_01_1939__i188951219191910"><system domain name></em><strong id="mrs_01_1939__b10892319191919">;"</strong></p>
|
|
</li><li id="mrs_01_1939__l49794413abc148a583774adb7ed1197c">Run the following command to create a table in SparkSQL and HBase, for example, create the <strong id="mrs_01_1939__b1550677365113138">hbaseTable</strong> table:<p id="mrs_01_1939__aa7ee96d7a98d402fa86cc715e6f32599"><strong id="mrs_01_1939__b7797193518514">create table hbaseTable (id string, name string, age int) using org.apache.spark.sql.hbase.HBaseSource options (hbaseTableName "table1", keyCols "id", colsMapping = ", name=cf1.cq1, age=cf1.cq2");</strong></p>
|
|
<p id="mrs_01_1939__ab308900953f94e39a6d8d0e4685b83d5">The created SparkSQL table and the HBase table are stored in the Hive database <span class="parmname" id="mrs_01_1939__p859f13cbc4714fafb2cbd72289c4e1e3"><b>default</b></span> and the HBase namespace <span class="parmname" id="mrs_01_1939__pdfe8b87ee590445ca71a69f9eab85bbd"><b>default</b></span>, respectively.</p>
|
|
</li><li id="mrs_01_1939__l24bed30b407047fcab0e46cd91ec75ed">On Manager, create a role, for example, <span class="parmname" id="mrs_01_1939__p780e6a344e35424cb3c1fe340846e91c"><b>hive_hbase_select</b></span>, and grant the role the permission to query SparkSQL on HBase table <strong id="mrs_01_1939__b1879514361716">hbaseTable</strong> and HBase table <strong id="mrs_01_1939__b2798123614717">hbaseTable</strong>.<ul id="mrs_01_1939__u77d3e6afb3174307a3f07b3048192994"><li id="mrs_01_1939__leab9eb5ada234445943410f4bd6bf081">In the <strong id="mrs_01_1939__b1720622640113138">Configure Resource Permission</strong> table, choose <em id="mrs_01_1939__i702979975113138">Name of the desired cluster</em> > <strong id="mrs_01_1939__b802778653113138">HBase </strong>> <strong id="mrs_01_1939__b611911611113138">HBase Scope</strong> > <strong id="mrs_01_1939__b907309594113138">global </strong>> <strong id="mrs_01_1939__b731109880113138">default</strong>. Select <span class="parmname" id="mrs_01_1939__pdd486766cf8e4e9abd164c6f4fac8c4c"><b><span id="mrs_01_1939__text4929141391718">read</span></b></span> for the <strong id="mrs_01_1939__b594960798113138">hbaseTable</strong> table, and click <span class="uicontrol" id="mrs_01_1939__uicontrol921125704113138"><b>OK</b></span> to grant the table query permission to the HBase role.</li><li id="mrs_01_1939__lacf6986ebcf048628784f3193af59bc2">Edit the role. In the <strong id="mrs_01_1939__b1758388872113138">Configure Resource Permission</strong> table, choose <em id="mrs_01_1939__i504792084113138">Name of the desired cluster</em> > <strong id="mrs_01_1939__b4677192416118">HBase </strong>> <strong id="mrs_01_1939__b19191129151119">HBase Scope </strong>> <strong id="mrs_01_1939__b7972330181113">global </strong>> <strong id="mrs_01_1939__b839743231112">hbase</strong>. Select <span class="parmname" id="mrs_01_1939__pf6bd0f7359784fa69c5e8560301fdc1e"><b><span id="mrs_01_1939__text14582111915175">Execute</span></b></span> for <span class="parmname" id="mrs_01_1939__p28cbbb5fd3dd43de903123cf097a26fa"><b>hbase:meta</b></span>, and click <span class="uicontrol" id="mrs_01_1939__u65b10f650f304fc3a56cbabf6554e79f"><b>OK</b></span>.</li><li id="mrs_01_1939__lc020734a19fa49d38fad2aacef40e88e">Edit the role. In the <strong id="mrs_01_1939__b947741115113138">Configure Resource Permission</strong> table, choose <em id="mrs_01_1939__i374743790113138">Name of the desired cluster</em> > <strong id="mrs_01_1939__b2112634636113138">Hive </strong>> <strong id="mrs_01_1939__b2085444294113138">Hive<span id="mrs_01_1939__text4652633104618"> Read Write Privileges</span></strong> > <strong id="mrs_01_1939__b795778125113138">default</strong>. Select <span class="parmname" id="mrs_01_1939__peb94b137dc85440b91b8586a4b5d336b"><b>SELECT</b></span> for the <strong id="mrs_01_1939__b343982158113138">hbaseTable</strong> table, and click <span class="uicontrol" id="mrs_01_1939__ucaedd78144094641995b6413514cb241"><b>OK</b></span>.</li></ul>
|
|
</li><li id="mrs_01_1939__l04613440fb204a1fa017b8d65637fc98">On Manager, create a human-machine user, for example, <span class="parmname" id="mrs_01_1939__p954666859d2841f1a6f5616ae89f8fa1"><b>hbase_select_user</b></span>, add the user to the <span class="parmname" id="mrs_01_1939__p0340ee53389b44f389d8fc984dcb210a"><b>hive</b></span> group, and bind the <span class="parmname" id="mrs_01_1939__p9015726bce67413b90cf483bf9c1b4d1"><b>hive_hbase_select</b></span> role to the user for querying SparkSQL and HBase tables.</li><li id="mrs_01_1939__la752534c14164c25b2a4157b0e1ea6f4">Run the following command to configure environment variables:<p id="mrs_01_1939__a114b586b37a44a91a15ab70abb041d43"><a name="mrs_01_1939__la752534c14164c25b2a4157b0e1ea6f4"></a><a name="la752534c14164c25b2a4157b0e1ea6f4"></a><strong id="mrs_01_1939__b1030519448616">source /opt/client/bigdata_env</strong></p>
|
|
<p id="mrs_01_1939__p947115301267"><strong id="mrs_01_1939__b176611735194413">source /opt/client/Spark2x/component_env</strong></p>
|
|
</li><li id="mrs_01_1939__la3a50092f4ff48f2967af37573391188">Run the following command to authenticate users:<p id="mrs_01_1939__a374ebe1147ca488581018cdce65b7fe7"><a name="mrs_01_1939__la3a50092f4ff48f2967af37573391188"></a><a name="la3a50092f4ff48f2967af37573391188"></a><strong id="mrs_01_1939__b51614247712">kinit hbase_select_user</strong></p>
|
|
</li><li id="mrs_01_1939__l49e8872fa31b4a1484fd83bb0329c205">Run the following commands to enter the shell environment on the Spark JDBCServer client:<p id="mrs_01_1939__af81388de48aa475dabb0319c5034252a"><a name="mrs_01_1939__l49e8872fa31b4a1484fd83bb0329c205"></a><a name="l49e8872fa31b4a1484fd83bb0329c205"></a><strong id="mrs_01_1939__b132863751914">/opt/client/Spark2x/spark/bin/beeline -u "jdbc:hive2://<zkNode1_IP>:<zkNode1_Port>,<zkNode2_IP>:<zkNode2_Port>,<zkNode3_IP>:<zkNode3_Port>/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=sparkthriftserver2x;user.principal=spark2x/hadoop.</strong><em id="mrs_01_1939__i430113715193"><system domain name></em><strong id="mrs_01_1939__b4871104110195">@</strong><em id="mrs_01_1939__i8874184117198"><system domain name></em><strong id="mrs_01_1939__b33594549193">;saslQop=auth-conf;auth=KERBEROS;principal=spark2x/hadoop.</strong><em id="mrs_01_1939__i13362205417195"><system domain name></em><strong id="mrs_01_1939__b156057596197">@</strong><em id="mrs_01_1939__i4609195917197"><system domain name></em><strong id="mrs_01_1939__b18605759171917">;"</strong></p>
|
|
</li><li id="mrs_01_1939__lcfcc1f6ea5104624b97c9c352d3e4db9">Run the following command to use a SparkSQL statement to query HBase table data:<p id="mrs_01_1939__a7a59b1c97ed24fc4a86018f12b1c9718"><a name="mrs_01_1939__lcfcc1f6ea5104624b97c9c352d3e4db9"></a><a name="lcfcc1f6ea5104624b97c9c352d3e4db9"></a><strong id="mrs_01_1939__b631111551379">select * from hbaseTable;</strong></p>
|
|
</li></ol>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1935.html">SparkSQL Permission Management(Security Mode)</a></div>
|
|
</div>
|
|
</div>
|
|
|