forked from docs/doc-exports
Yang, Tong
3f5759eed2
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
41 lines
6.2 KiB
HTML
41 lines
6.2 KiB
HTML
<a name="mrs_01_1950"></a><a name="mrs_01_1950"></a>
|
|
|
|
<h1 class="topictitle1">Configuring Whether to Display Spark SQL Statements Containing Sensitive Words</h1>
|
|
<div id="body8662426"><div class="section" id="mrs_01_1950__en-us_topic_0000001173789640_s365e55cf47e74c1cb7dcabc669d6f919"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1950__en-us_topic_0000001173789640_adac0a8162e3d4876b19b6d8fb361e090">SQL statements executed by users may contain sensitive information (such as passwords). Disclosure of such information may incur security risks. You can configure the <strong id="mrs_01_1950__en-us_topic_0000001173789640_b1039477165414">spark.sql.redaction.string.regex</strong> parameter to shield SQL keywords that contain sensitive words in logs and on the web UI.</p>
|
|
<p id="mrs_01_1950__en-us_topic_0000001173789640_aee98f04e15b5440bac160edccac988d2">Spark SQL statements consist of two parts:</p>
|
|
<ol id="mrs_01_1950__en-us_topic_0000001173789640_o980d8ed1da6b41ca8bedb745cb426ced"><li id="mrs_01_1950__en-us_topic_0000001173789640_lbde09ab4b66441348811cbec8ae37b79">Spark SQL statements in logs:<ul id="mrs_01_1950__en-us_topic_0000001173789640_uceedeace9768450cb6bec921bba39bb8"><li id="mrs_01_1950__en-us_topic_0000001173789640_l76c3fd37c0dc4e3b85716363b461333d">Driver log: In the JDBCServer service, each time an SQL statement is executed through Beeline, the corresponding SQL statement is added to the Driver log, for example, <strong id="mrs_01_1950__en-us_topic_0000001173789640_b107315424493">Running query'show tables' with 0f8fee16-4291-4854-a7b4-b87a162f7cbb</strong>.</li><li id="mrs_01_1950__en-us_topic_0000001173789640_l0c9a5cbde0cc498d9ba9f56d076cbd7f">eventLog log: If the eventLog writing function is enabled for each Spark application (<strong id="mrs_01_1950__en-us_topic_0000001173789640_b16700546115112">spark.eventLog.enabled</strong> is set to <strong id="mrs_01_1950__en-us_topic_0000001173789640_b10700746185116">true</strong>), event logs will be written. SQL statements executed using JDBCServer and Spark SQL will also be added to the eventLog file.</li></ul>
|
|
</li><li id="mrs_01_1950__en-us_topic_0000001173789640_lce08455294cd4b5facb482fb4ed28ac6">Spark SQL statements on the web UI:<ul id="mrs_01_1950__en-us_topic_0000001173789640_u8c809ca661c44f9ba25928c3b19a6e86"><li id="mrs_01_1950__en-us_topic_0000001173789640_l3c946887a80a496ab81b90c456614f04">SparkUI: When executing SQL statements, you can view the executed SQL statements on the <strong id="mrs_01_1950__en-us_topic_0000001173789640_b16586586552">Jobs</strong> and <strong id="mrs_01_1950__en-us_topic_0000001173789640_b75901388551">Stages</strong> tab pages on the SparkUI.</li><li id="mrs_01_1950__en-us_topic_0000001173789640_ld1bca2b35c234d6389acc044d53c370c">HistoryServer: HistoryServer reads the eventLog file and displays app information on the page. Therefore, if the eventLog file contains SQL statement records, you can view the corresponding SQL statements on the HistoryServer page.</li></ul>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_1950__en-us_topic_0000001173789640_s634b2007306543aeb3eea984a5b841fe"><h4 class="sectiontitle">Configuration</h4>
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1950__en-us_topic_0000001173789640_t2e54c9d396a54b4faf756b7d4796f912" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1950__en-us_topic_0000001173789640_r3a27b9a38a1a480a8d1961fdd67481d5"><th align="left" class="cellrowborder" valign="top" width="33.21%" id="mcps1.3.2.2.2.4.1.1"><p id="mrs_01_1950__en-us_topic_0000001173789640_aafa76b2e2a7c4efcbe5930994bd5b9fb">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="49.62%" id="mcps1.3.2.2.2.4.1.2"><p id="mrs_01_1950__en-us_topic_0000001173789640_a3073c31d5fdd41e1b3373288cc31d4d3">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="17.169999999999998%" id="mcps1.3.2.2.2.4.1.3"><p id="mrs_01_1950__en-us_topic_0000001173789640_a18578a2acaa546f89e6dab002648709b">Default Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1950__en-us_topic_0000001173789640_row20478510142418"><td class="cellrowborder" valign="top" width="33.21%" headers="mcps1.3.2.2.2.4.1.1 "><p id="mrs_01_1950__en-us_topic_0000001173789640_p4479151014243">spark.sql.redaction.string.regex</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.62%" headers="mcps1.3.2.2.2.4.1.2 "><p id="mrs_01_1950__en-us_topic_0000001173789640_p64791010112419">Regular expression that determines which part in a string generated by Spark is sensitive words. If the regular expression matches sensitive words of the string, these sensitive words are replaced with <strong id="mrs_01_1950__en-us_topic_0000001173789640_b521125194314">*********(redacted)</strong>.</p>
|
|
<div class="note" id="mrs_01_1950__en-us_topic_0000001173789640_note624495517250"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1950__en-us_topic_0000001173789640_p625075572513">The value must be a regular expression.</p>
|
|
</div></div>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.169999999999998%" headers="mcps1.3.2.2.2.4.1.3 "><p id="mrs_01_1950__en-us_topic_0000001173789640_p647961016243">pwd|password</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="note" id="mrs_01_1950__en-us_topic_0000001173789640_nc14e69923c634d9d84cb326935263ac1"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1950__en-us_topic_0000001173789640_u6bba51a92b5946cc85b656fd9cbaca6d"><li id="mrs_01_1950__en-us_topic_0000001173789640_lf59bc1db26584c61a05c673dece5e42e">If Spark Beeline is used, you need to restart JDBCServer for the configuration to take effect. If spark-sql is used, you need to restart spark-sql for the configuration to take effect.</li><li id="mrs_01_1950__en-us_topic_0000001173789640_lc43457e6f0584242adf499efe983b337">The preceding parameters are valid only for the SQL statements that are executed after the configuration.</li></ul>
|
|
</div></div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1941.html">Scenario-Specific Configuration</a></div>
|
|
</div>
|
|
</div>
|
|
|