vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/dev/dli_09_0094.html
Hasko, Vladimir cfc48b3aed dli_dev_0104_version 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-committed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
2024-05-06 09:14:57 +00:00

604 lines
85 KiB
HTML
Raw Blame History

<a name="dli_09_0094"></a><a name="dli_09_0094"></a>
<h1 class="topictitle1">Scala Example Code</h1>
<div id="body8662426"><div class="section" id="dli_09_0094__section202271653155410"><h4 class="sectiontitle">Development Description</h4><p id="dli_09_0094__en-us_topic_0200509988_p492312464537">Redis supports only enhanced datasource connections. </p>
<ul id="dli_09_0094__ul217271155912"><li id="dli_09_0094__li317241175914">Prerequisites<p id="dli_09_0094__en-us_topic_0200509988_p1944354710257"><a name="dli_09_0094__li317241175914"></a><a name="li317241175914"></a>An enhanced datasource connection has been created on the DLI management console and bound to a queue in packages. </p>
<div class="note" id="dli_09_0094__note1358715714155"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0094__p692572617287">Hard-coded or plaintext passwords pose significant security risks. To ensure security, encrypt your passwords, store them in configuration files or environment variables, and decrypt them when needed.</p>
</div></div>
</li><li id="dli_09_0094__li1160074610010">Constructing dependency information and creating a Spark session<ol id="dli_09_0094__en-us_topic_0200509988_ol433013261137"><li id="dli_09_0094__en-us_topic_0200509988_li1825623917170">Import dependencies.<div class="p" id="dli_09_0094__en-us_topic_0200509988_p0101458105614"><a name="dli_09_0094__en-us_topic_0200509988_li1825623917170"></a><a name="en-us_topic_0200509988_li1825623917170"></a>Maven dependency involved<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen5760163172012"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span></pre></div></td><td class="code"><div><pre><span></span><span class="o">&lt;</span><span class="n">dependency</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">groupId</span><span class="o">&gt;</span><span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">spark</span><span class="o">&lt;/</span><span class="n">groupId</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">artifactId</span><span class="o">&gt;</span><span class="n">spark</span><span class="o">-</span><span class="n">sql_2</span><span class="mf">.11</span><span class="o">&lt;/</span><span class="n">artifactId</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">version</span><span class="o">&gt;</span><span class="mf">2.3.2</span><span class="o">&lt;/</span><span class="n">version</span><span class="o">&gt;</span>
<span class="o">&lt;/</span><span class="n">dependency</span><span class="o">&gt;</span>
<span class="o">&lt;</span><span class="n">dependency</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">groupId</span><span class="o">&gt;</span><span class="n">redis</span><span class="p">.</span><span class="n">clients</span><span class="o">&lt;/</span><span class="n">groupId</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">artifactId</span><span class="o">&gt;</span><span class="n">jedis</span><span class="o">&lt;/</span><span class="n">artifactId</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">version</span><span class="o">&gt;</span><span class="mf">3.1.0</span><span class="o">&lt;/</span><span class="n">version</span><span class="o">&gt;</span>
<span class="o">&lt;/</span><span class="n">dependency</span><span class="o">&gt;</span>
<span class="o">&lt;</span><span class="n">dependency</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">groupId</span><span class="o">&gt;</span><span class="n">com</span><span class="p">.</span><span class="n">redislabs</span><span class="o">&lt;/</span><span class="n">groupId</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">artifactId</span><span class="o">&gt;</span><span class="n">spark</span><span class="o">-</span><span class="n">redis</span><span class="o">&lt;/</span><span class="n">artifactId</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">version</span><span class="o">&gt;</span><span class="mf">2.4.0</span><span class="o">&lt;/</span><span class="n">version</span><span class="o">&gt;</span>
<span class="o">&lt;/</span><span class="n">dependency</span><span class="o">&gt;</span>
</pre></div></td></tr></table></div>
</div>
</div>
<div class="p" id="dli_09_0094__en-us_topic_0200509988_p9835524175614">Import dependency packages.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen539112516557"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">import</span><span class="w"> </span><span class="nn">org</span><span class="p">.</span><span class="nn">apache</span><span class="p">.</span><span class="nn">spark</span><span class="p">.</span><span class="nn">sql</span><span class="p">.{</span><span class="nc">Row</span><span class="p">,</span><span class="w"> </span><span class="nc">SaveMode</span><span class="p">,</span><span class="w"> </span><span class="nc">SparkSession</span><span class="p">}</span>
<span class="k">import</span><span class="w"> </span><span class="nn">org</span><span class="p">.</span><span class="nn">apache</span><span class="p">.</span><span class="nn">spark</span><span class="p">.</span><span class="nn">sql</span><span class="p">.</span><span class="nn">types</span><span class="p">.</span><span class="n">_</span>
<span class="k">import</span><span class="w"> </span><span class="nn">com</span><span class="p">.</span><span class="nn">redislabs</span><span class="p">.</span><span class="nn">provider</span><span class="p">.</span><span class="nn">redis</span><span class="p">.</span><span class="n">_</span>
<span class="k">import</span><span class="w"> </span><span class="nn">scala</span><span class="p">.</span><span class="nn">reflect</span><span class="p">.</span><span class="nn">runtime</span><span class="p">.</span><span class="nn">universe</span><span class="p">.</span><span class="n">_</span>
<span class="k">import</span><span class="w"> </span><span class="nn">org</span><span class="p">.</span><span class="nn">apache</span><span class="p">.</span><span class="nn">spark</span><span class="p">.{</span><span class="nc">SparkConf</span><span class="p">,</span><span class="w"> </span><span class="nc">SparkContext</span><span class="p">}</span>
</pre></div></td></tr></table></div>
</div>
</div>
</li></ol>
</li><li id="dli_09_0094__li422572717114">Connecting to data sources through DataFrame APIs<ol id="dli_09_0094__en-us_topic_0200509988_ol81572119429"><li id="dli_09_0094__en-us_topic_0200509988_li165683192421">Create a session.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen1434684012425"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="kd">val</span><span class="w"> </span><span class="n">sparkSession</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nc">SparkSession</span><span class="p">.</span><span class="n">builder</span><span class="p">().</span><span class="n">appName</span><span class="p">(</span><span class="s">&quot;datasource_redis&quot;</span><span class="p">).</span><span class="n">getOrCreate</span><span class="p">()</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0094__en-us_topic_0200509988_li159162206521">Construct a schema.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen255614411440"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span></pre></div></td><td class="code"><div><pre><span></span><span class="c1">//method one</span>
<span class="kd">var</span><span class="w"> </span><span class="n">schema</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nc">StructType</span><span class="p">(</span><span class="nc">Seq</span><span class="p">(</span><span class="nc">StructField</span><span class="p">(</span><span class="s">&quot;name&quot;</span><span class="p">,</span><span class="w"> </span><span class="nc">StringType</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</span><span class="w"> </span><span class="nc">StructField</span><span class="p">(</span><span class="s">&quot;age&quot;</span><span class="p">,</span><span class="w"> </span><span class="nc">IntegerType</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">)))</span>
<span class="kd">var</span><span class="w"> </span><span class="n">rdd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sparkContext</span><span class="p">.</span><span class="n">parallelize</span><span class="p">(</span><span class="nc">Seq</span><span class="p">(</span><span class="nc">Row</span><span class="p">(</span><span class="s">&quot;abc&quot;</span><span class="p">,</span><span class="mi">34</span><span class="p">),</span><span class="nc">Row</span><span class="p">(</span><span class="s">&quot;Bob&quot;</span><span class="p">,</span><span class="mi">19</span><span class="p">)))</span>
<span class="kd">var</span><span class="w"> </span><span class="n">dataFrame</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="n">createDataFrame</span><span class="p">(</span><span class="n">rdd</span><span class="p">,</span><span class="w"> </span><span class="n">schema</span><span class="p">)</span>
<span class="c1">// //method two</span>
<span class="c1">// var jdbcDF= sparkSession.createDataFrame(Seq((&quot;Jack&quot;,23)))</span>
<span class="c1">// val dataFrame = jdbcDF.withColumnRenamed(&quot;_1&quot;, &quot;name&quot;).withColumnRenamed(&quot;_2&quot;, &quot;age&quot;)</span>
<span class="c1">// //method three </span>
<span class="c1">// case class Person(name: String, age: Int)</span>
<span class="c1">// val dataFrame = sparkSession.createDataFrame(Seq(Person(&quot;John&quot;, 30), Person(&quot;Peter&quot;, 45)))</span>
</pre></div></td></tr></table></div>
</div>
<div class="note" id="dli_09_0094__en-us_topic_0200509988_note16116635184212"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0094__en-us_topic_0200509988_p211753574217"><strong id="dli_09_0094__b6612131313260">case class Person(name: String, age: Int)</strong> must be written outside the object. For details, see <a href="#dli_09_0094__li1640095910911">Connecting to data sources through DataFrame APIs</a>.</p>
</div></div>
</li><li id="dli_09_0094__en-us_topic_0200509988_li6847933185312">Import data to Redis.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen1550013195536"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span>
<span class="normal">8</span>
<span class="normal">9</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">dataFrame</span><span class="w"> </span><span class="p">.</span><span class="n">write</span>
<span class="w"> </span><span class="p">.</span><span class="n">format</span><span class="p">(</span><span class="s">&quot;redis&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;host&quot;</span><span class="p">,</span><span class="s">&quot;192.168.4.199&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;port&quot;</span><span class="p">,</span><span class="s">&quot;6379&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;table&quot;</span><span class="p">,</span><span class="s">&quot;person&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;password&quot;</span><span class="p">,</span><span class="s">&quot;******&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;key.column&quot;</span><span class="p">,</span><span class="s">&quot;name&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">mode</span><span class="p">(</span><span class="nc">SaveMode</span><span class="p">.</span><span class="nc">Overwrite</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">save</span><span class="p">()</span>
</pre></div></td></tr></table></div>
</div>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="dli_09_0094__en-us_topic_0200509988_table18619182584014" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Redis operation parameters</caption><thead align="left"><tr id="dli_09_0094__en-us_topic_0200509988_row14619102514015"><th align="left" class="cellrowborder" valign="top" width="16.56%" id="mcps1.3.1.3.3.1.3.2.2.3.1.1"><p id="dli_09_0094__en-us_topic_0200509988_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_a9dd69379a3f54904a9c510fb817659ad"><strong id="dli_09_0094__b4576813105612">Parameter</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="83.44%" id="mcps1.3.1.3.3.1.3.2.2.3.1.2"><p id="dli_09_0094__en-us_topic_0200509988_en-us_topic_0142907229_en-us_topic_0114776213_en-us_topic_0103157088_a7ede0d62539542baa7e367cf902bf2b2"><strong id="dli_09_0094__b1414481525619">Description</strong></p>
</th>
</tr>
</thead>
<tbody><tr id="dli_09_0094__en-us_topic_0200509988_row114553518494"><td class="cellrowborder" valign="top" width="16.56%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.1 "><p id="dli_09_0094__en-us_topic_0200509988_p15995195524912">host</p>
</td>
<td class="cellrowborder" valign="top" width="83.44%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.2 "><p id="dli_09_0094__en-us_topic_0200509988_p2995115515498">IP address of the Redis cluster to be connected.</p>
<p id="dli_09_0094__en-us_topic_0200509988_p13995155514912">To obtain the IP address, log in to the official website, search for <strong id="dli_09_0094__b1648415485111">redis</strong>, go to the console of Distributed Cache Service for Redis, and choose <strong id="dli_09_0094__b54841554145112">Cache Manager</strong>. Select an IP address (including the port information) based on the IP address required by the host name to copy the data.</p>
</td>
</tr>
<tr id="dli_09_0094__en-us_topic_0200509988_row18620172510403"><td class="cellrowborder" valign="top" width="16.56%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.1 "><p id="dli_09_0094__en-us_topic_0200509988_p1620425164020">port</p>
</td>
<td class="cellrowborder" valign="top" width="83.44%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.2 "><p id="dli_09_0094__en-us_topic_0200509988_p12620172564013">Access port.</p>
</td>
</tr>
<tr id="dli_09_0094__en-us_topic_0200509988_row13620122511409"><td class="cellrowborder" valign="top" width="16.56%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.1 "><p id="dli_09_0094__en-us_topic_0200509988_p962017258404">password</p>
</td>
<td class="cellrowborder" valign="top" width="83.44%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.2 "><p id="dli_09_0094__en-us_topic_0200509988_p8620725104013">Password for the connection. This parameter is optional if no password is required.</p>
</td>
</tr>
<tr id="dli_09_0094__en-us_topic_0200509988_row1362042512401"><td class="cellrowborder" valign="top" width="16.56%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.1 "><p id="dli_09_0094__en-us_topic_0200509988_p116201925164017">table</p>
</td>
<td class="cellrowborder" valign="top" width="83.44%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.2 "><p id="dli_09_0094__en-us_topic_0200509988_p3413914173920">Key or hash key in Redis.</p>
<ul id="dli_09_0094__en-us_topic_0200509988_ul1647472215396"><li id="dli_09_0094__en-us_topic_0200509988_li1647415222396">This parameter is mandatory when Redis data is inserted.</li><li id="dli_09_0094__en-us_topic_0200509988_li5474622103915">Either this parameter or the <strong id="dli_09_0094__b1927834675711">keys.pattern</strong> parameter when Redis data is queried.</li></ul>
</td>
</tr>
<tr id="dli_09_0094__en-us_topic_0200509988_row95051516496"><td class="cellrowborder" valign="top" width="16.56%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.1 "><p id="dli_09_0094__en-us_topic_0200509988_p1263223414340">keys.pattern</p>
</td>
<td class="cellrowborder" valign="top" width="83.44%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.2 "><p id="dli_09_0094__en-us_topic_0200509988_p19522133713392">Use a regular expression to match multiple keys or hash keys. This parameter is used only for query. Either this parameter or <strong id="dli_09_0094__b0286145285719">table</strong> is used to query Redis data.</p>
</td>
</tr>
<tr id="dli_09_0094__en-us_topic_0200509988_row9620425154020"><td class="cellrowborder" valign="top" width="16.56%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.1 "><p id="dli_09_0094__en-us_topic_0200509988_p1362072584011">key.column</p>
</td>
<td class="cellrowborder" valign="top" width="83.44%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.2 "><p id="dli_09_0094__en-us_topic_0200509988_p65414228213">Key value of a column. This parameter is optional. If a key is specified when data is written, the key must be specified during query. Otherwise, the key will be abnormally loaded during query.</p>
</td>
</tr>
<tr id="dli_09_0094__en-us_topic_0200509988_row210594195119"><td class="cellrowborder" valign="top" width="16.56%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.1 "><p id="dli_09_0094__en-us_topic_0200509988_p1051082954410">partitions.number</p>
</td>
<td class="cellrowborder" valign="top" width="83.44%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.2 "><p id="dli_09_0094__en-us_topic_0200509988_p4511122924416">Number of concurrent tasks during data reading.</p>
</td>
</tr>
<tr id="dli_09_0094__en-us_topic_0200509988_row31042435114"><td class="cellrowborder" valign="top" width="16.56%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.1 "><p id="dli_09_0094__en-us_topic_0200509988_p776343211447">scan.count</p>
</td>
<td class="cellrowborder" valign="top" width="83.44%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.2 "><p id="dli_09_0094__en-us_topic_0200509988_p6763232104413">Number of data records read in each batch. The default value is <strong id="dli_09_0094__b7376132135811">100</strong>. If the CPU usage of the Redis cluster still needs to be improved during data reading, increase the value of this parameter.</p>
</td>
</tr>
<tr id="dli_09_0094__en-us_topic_0200509988_row10104246515"><td class="cellrowborder" valign="top" width="16.56%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.1 "><p id="dli_09_0094__en-us_topic_0200509988_p12696183564418">iterator.grouping.size</p>
</td>
<td class="cellrowborder" valign="top" width="83.44%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.2 "><p id="dli_09_0094__en-us_topic_0200509988_p469663534418">Number of data records inserted in each batch. The default value is <strong id="dli_09_0094__b8558187115811">100</strong>. If the CPU usage of the Redis cluster still needs to be improved during the insertion, increase the value of this parameter.</p>
</td>
</tr>
<tr id="dli_09_0094__en-us_topic_0200509988_row1664332712311"><td class="cellrowborder" valign="top" width="16.56%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.1 "><p id="dli_09_0094__en-us_topic_0200509988_p474525520294">timeout</p>
</td>
<td class="cellrowborder" valign="top" width="83.44%" headers="mcps1.3.1.3.3.1.3.2.2.3.1.2 "><p id="dli_09_0094__en-us_topic_0200509988_p16745165592919">Timeout interval for connecting to the Redis, in milliseconds. The default value is <strong id="dli_09_0094__b43541914115818">2000</strong> (2 seconds).</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="note" id="dli_09_0094__en-us_topic_0200509988_note3841161165514"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_09_0094__en-us_topic_0200509988_ul6814141063"><li id="dli_09_0094__en-us_topic_0200509988_li626991818615">The options of <strong id="dli_09_0094__b19226174384211">mode</strong> are <strong id="dli_09_0094__b1259318116584">Overwrite</strong>, <strong id="dli_09_0094__b13593711125818">Append</strong>, <strong id="dli_09_0094__b459414112585">ErrorIfExis</strong>, and <strong id="dli_09_0094__b759416118582">Ignore</strong>.</li><li id="dli_09_0094__en-us_topic_0200509988_li12261202720614">To save nested DataFrames, use <strong id="dli_09_0094__b1230271765815">.option("model", "binary")</strong>.</li><li id="dli_09_0094__en-us_topic_0200509988_li3542271091">Specify the data expiration time by <strong id="dli_09_0094__b1989851925814">.option("ttl", 1000)</strong>. The unit is second.</li></ul>
</div></div>
</li><li id="dli_09_0094__en-us_topic_0200509988_li6157513420">Read data from Redis.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen5984155015578"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span>
<span class="normal">7</span>
<span class="normal">8</span>
<span class="normal">9</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">read</span>
<span class="w"> </span><span class="p">.</span><span class="n">format</span><span class="p">(</span><span class="s">&quot;redis&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;host&quot;</span><span class="p">,</span><span class="s">&quot;192.168.4.199&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;port&quot;</span><span class="p">,</span><span class="s">&quot;6379&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;table&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;person&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;password&quot;</span><span class="p">,</span><span class="s">&quot;######&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;key.column&quot;</span><span class="p">,</span><span class="s">&quot;name&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">load</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="n">show</span><span class="p">()</span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</li><li id="dli_09_0094__li2031119111529">Connecting to data sources using Spark RDDs<ol id="dli_09_0094__en-us_topic_0200509988_ol189671116155915"><li id="dli_09_0094__en-us_topic_0200509988_li14967101618594">Create a datasource connection.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen137824452596"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><span class="kd">val</span><span class="w"> </span><span class="n">sparkContext</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="nc">SparkContext</span><span class="p">(</span><span class="k">new</span><span class="w"> </span><span class="nc">SparkConf</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="n">setAppName</span><span class="p">(</span><span class="s">&quot;datasource_redis&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s">&quot;spark.redis.host&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;192.168.4.199&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s">&quot;spark.redis.port&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;6379&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s">&quot;spark.redis.auth&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;######&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s">&quot;spark.driver.allowMultipleContexts&quot;</span><span class="p">,</span><span class="s">&quot;true&quot;</span><span class="p">))</span>
</pre></div></td></tr></table></div>
</div>
<div class="note" id="dli_09_0094__en-us_topic_0200509988_note1484562321914"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="dli_09_0094__en-us_topic_0200509988_p24348905214">If <strong id="dli_09_0094__b1356195135914">spark.driver.allowMultipleContexts</strong> is set to <strong id="dli_09_0094__b17561175125910">true</strong>, only the current context is used when multiple contexts are started, to prevent context invoking conflicts.</p>
</div></div>
</li><li id="dli_09_0094__en-us_topic_0200509988_li894735914401">Insert data.<ol type="a" id="dli_09_0094__en-us_topic_0200509988_ol3489127133610"><li id="dli_09_0094__en-us_topic_0200509988_li548992773613">Save data in strings.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen106701346103615"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><span class="kd">val</span><span class="w"> </span><span class="n">stringRedisData</span><span class="p">:</span><span class="nc">RDD</span><span class="p">[(</span><span class="nc">String</span><span class="p">,</span><span class="nc">String</span><span class="p">)]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">parallelize</span><span class="p">(</span><span class="nc">Seq</span><span class="p">[(</span><span class="nc">String</span><span class="p">,</span><span class="nc">String</span><span class="p">)]((</span><span class="s">&quot;high&quot;</span><span class="p">,</span><span class="s">&quot;111&quot;</span><span class="p">),</span><span class="w"> </span><span class="p">(</span><span class="s">&quot;together&quot;</span><span class="p">,</span><span class="s">&quot;333&quot;</span><span class="p">)))</span>
<span class="n">sparkContext</span><span class="p">.</span><span class="n">toRedisKV</span><span class="p">(</span><span class="n">stringRedisData</span><span class="p">)</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0094__en-us_topic_0200509988_li129211826123711">Save data in hashes.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen73441846173714"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><span class="kd">val</span><span class="w"> </span><span class="n">hashRedisData</span><span class="p">:</span><span class="nc">RDD</span><span class="p">[(</span><span class="nc">String</span><span class="p">,</span><span class="nc">String</span><span class="p">)]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">parallelize</span><span class="p">(</span><span class="nc">Seq</span><span class="p">[(</span><span class="nc">String</span><span class="p">,</span><span class="nc">String</span><span class="p">)]((</span><span class="s">&quot;saprk&quot;</span><span class="p">,</span><span class="s">&quot;123&quot;</span><span class="p">),</span><span class="w"> </span><span class="p">(</span><span class="s">&quot;data&quot;</span><span class="p">,</span><span class="s">&quot;222&quot;</span><span class="p">)))</span>
<span class="n">sparkContext</span><span class="p">.</span><span class="n">toRedisHASH</span><span class="p">(</span><span class="n">hashRedisData</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;hashRDD&quot;</span><span class="p">)</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0094__en-us_topic_0200509988_li6605812383">Save data in lists.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen13497336143819"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span></pre></div></td><td class="code"><div><pre><span></span><span class="kd">val</span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nc">List</span><span class="p">((</span><span class="s">&quot;school&quot;</span><span class="p">,</span><span class="s">&quot;112&quot;</span><span class="p">),</span><span class="w"> </span><span class="p">(</span><span class="s">&quot;tom&quot;</span><span class="p">,</span><span class="s">&quot;333&quot;</span><span class="p">))</span>
<span class="kd">val</span><span class="w"> </span><span class="n">listRedisData</span><span class="p">:</span><span class="nc">RDD</span><span class="p">[</span><span class="nc">String</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">parallelize</span><span class="p">(</span><span class="nc">Seq</span><span class="p">[(</span><span class="nc">String</span><span class="p">)](</span><span class="n">data</span><span class="p">.</span><span class="n">toString</span><span class="p">()))</span>
<span class="n">sparkContext</span><span class="p">.</span><span class="n">toRedisLIST</span><span class="p">(</span><span class="n">listRedisData</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;listRDD&quot;</span><span class="p">)</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0094__en-us_topic_0200509988_li1522013144389">Save data in sets.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen7858192243911"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span></pre></div></td><td class="code"><div><pre><span></span><span class="kd">val</span><span class="w"> </span><span class="n">setData</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nc">Set</span><span class="p">((</span><span class="s">&quot;bob&quot;</span><span class="p">,</span><span class="s">&quot;133&quot;</span><span class="p">),(</span><span class="s">&quot;kity&quot;</span><span class="p">,</span><span class="s">&quot;322&quot;</span><span class="p">))</span>
<span class="kd">val</span><span class="w"> </span><span class="n">setRedisData</span><span class="p">:</span><span class="nc">RDD</span><span class="p">[(</span><span class="nc">String</span><span class="p">)]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">parallelize</span><span class="p">(</span><span class="nc">Seq</span><span class="p">[(</span><span class="nc">String</span><span class="p">)](</span><span class="n">setData</span><span class="p">.</span><span class="n">mkString</span><span class="p">))</span>
<span class="n">sparkContext</span><span class="p">.</span><span class="n">toRedisSET</span><span class="p">(</span><span class="n">setRedisData</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;setRDD&quot;</span><span class="p">)</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0094__en-us_topic_0200509988_li15311418133810">Save data in zsets.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen20209125773917"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span></pre></div></td><td class="code"><div><pre><span></span><span class="kd">val</span><span class="w"> </span><span class="n">zsetRedisData</span><span class="p">:</span><span class="nc">RDD</span><span class="p">[(</span><span class="nc">String</span><span class="p">,</span><span class="nc">String</span><span class="p">)]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">parallelize</span><span class="p">(</span><span class="nc">Seq</span><span class="p">[(</span><span class="nc">String</span><span class="p">,</span><span class="nc">String</span><span class="p">)]((</span><span class="s">&quot;whight&quot;</span><span class="p">,</span><span class="s">&quot;234&quot;</span><span class="p">),</span><span class="w"> </span><span class="p">(</span><span class="s">&quot;bobo&quot;</span><span class="p">,</span><span class="s">&quot;343&quot;</span><span class="p">)))</span>
<span class="n">sparkContext</span><span class="p">.</span><span class="n">toRedisZSET</span><span class="p">(</span><span class="n">zsetRedisData</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;zsetRDD&quot;</span><span class="p">)</span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</li><li id="dli_09_0094__en-us_topic_0200509988_li54733138416">Query data.<ol type="a" id="dli_09_0094__en-us_topic_0200509988_ol113711816426"><li id="dli_09_0094__en-us_topic_0200509988_li13371187429">Query data by traversing keys.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen15407850154215"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span>
<span class="normal">2</span>
<span class="normal">3</span>
<span class="normal">4</span>
<span class="normal">5</span>
<span class="normal">6</span></pre></div></td><td class="code"><div><pre><span></span><span class="kd">val</span><span class="w"> </span><span class="n">keysRDD</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisKeys</span><span class="p">(</span><span class="nc">Array</span><span class="p">(</span><span class="s">&quot;high&quot;</span><span class="p">,</span><span class="s">&quot;together&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;hashRDD&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;listRDD&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;setRDD&quot;</span><span class="p">,</span><span class="s">&quot;zsetRDD&quot;</span><span class="p">),</span><span class="w"> </span><span class="mi">6</span><span class="p">)</span>
<span class="n">keysRDD</span><span class="p">.</span><span class="n">getKV</span><span class="p">().</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">(</span><span class="n">println</span><span class="p">)</span>
<span class="n">keysRDD</span><span class="p">.</span><span class="n">getHash</span><span class="p">().</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">(</span><span class="n">println</span><span class="p">)</span>
<span class="n">keysRDD</span><span class="p">.</span><span class="n">getList</span><span class="p">().</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">(</span><span class="n">println</span><span class="p">)</span>
<span class="n">keysRDD</span><span class="p">.</span><span class="n">getSet</span><span class="p">().</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">(</span><span class="n">println</span><span class="p">)</span>
<span class="n">keysRDD</span><span class="p">.</span><span class="n">getZSet</span><span class="p">().</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">(</span><span class="n">println</span><span class="p">)</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0094__en-us_topic_0200509988_li0144348435">Query data by string.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen5690827114415"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisKV</span><span class="p">(</span><span class="nc">Array</span><span class="p">(</span><span class="w"> </span><span class="s">&quot;high&quot;</span><span class="p">,</span><span class="s">&quot;together&quot;</span><span class="p">)).</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">{</span><span class="n">println</span><span class="p">}</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0094__en-us_topic_0200509988_li6931183114413">Query data by hash.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen113020174454"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisHash</span><span class="p">(</span><span class="nc">Array</span><span class="p">(</span><span class="s">&quot;hashRDD&quot;</span><span class="p">)).</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">{</span><span class="n">println</span><span class="p">}</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0094__en-us_topic_0200509988_li15523173694418">Query data by list.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen141423484717"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisList</span><span class="p">(</span><span class="nc">Array</span><span class="p">(</span><span class="s">&quot;listRDD&quot;</span><span class="p">)).</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">{</span><span class="n">println</span><span class="p">}</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0094__en-us_topic_0200509988_li12244239174413">Query data by set.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen9121164674714"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisSet</span><span class="p">(</span><span class="nc">Array</span><span class="p">(</span><span class="s">&quot;setRDD&quot;</span><span class="p">)).</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">{</span><span class="n">println</span><span class="p">}</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0094__en-us_topic_0200509988_li155694804413">Query data by zset.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen1310914580475"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisZSet</span><span class="p">(</span><span class="nc">Array</span><span class="p">(</span><span class="s">&quot;zsetRDD&quot;</span><span class="p">)).</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">{</span><span class="n">println</span><span class="p">}</span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</li></ol>
</li><li id="dli_09_0094__li148199371022">Connecting to data sources through SQL APIs<ol id="dli_09_0094__en-us_topic_0200509988_ol11291435145014"><li id="dli_09_0094__en-us_topic_0200509988_li7129135105012">Create a table to connect to a Redis data source.<pre class="screen" id="dli_09_0094__screen1812862316408">sparkSession.sql(
"CREATE TEMPORARY VIEW person (name STRING, age INT) USING org.apache.spark.sql.redis OPTIONS (
'host' = '192.168.4.199',
'port' = '6379',
'password' = '######',
table 'person')".stripMargin)</pre>
</li><li id="dli_09_0094__en-us_topic_0200509988_li08868382528">Insert data.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen82431853135210"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">&quot;INSERT INTO TABLE person VALUES ('John', 30),('Peter', 45)&quot;</span><span class="p">.</span><span class="n">stripMargin</span><span class="p">)</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0094__en-us_topic_0200509988_li1777420614539">Query data.<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509988_screen7922121675310"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal">1</span></pre></div></td><td class="code"><div><pre><span></span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">&quot;SELECT * FROM person&quot;</span><span class="p">.</span><span class="n">stripMargin</span><span class="p">).</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">(</span><span class="n">println</span><span class="p">)</span>
</pre></div></td></tr></table></div>
</div>
</li></ol>
</li><li id="dli_09_0094__li2309506311">Submitting a Spark job<ol id="dli_09_0094__ol14491137636"><li id="dli_09_0094__li1692416144334">Generate a JAR package based on the code and upload the package to DLI.<p id="dli_09_0094__dli_09_0063_p1749619513385"><a name="dli_09_0094__li1692416144334"></a><a name="li1692416144334"></a></p>
<p id="dli_09_0094__dli_09_0063_p114961151385"></p>
</li><li id="dli_09_0094__li14768352172919">In the Spark job editor, select the corresponding dependency module and execute the Spark job.<p id="dli_09_0094__p1989515714295"><a name="dli_09_0094__li14768352172919"></a><a name="li14768352172919"></a></p>
<div class="p" id="dli_09_0094__p480645415297"><div class="note" id="dli_09_0094__en-us_topic_0200509988_note1435543551919"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="dli_09_0094__en-us_topic_0200509988_ul17825285811"><li id="dli_09_0094__en-us_topic_0197738142_li58215295819">If the Spark version is 2.3.2 (will be offline soon) or 2.4.5, specify the <strong id="dli_09_0094__b17739119155916">Module</strong> to <strong id="dli_09_0094__b173915193593">sys.datasource.redis</strong> when you submit a job.</li><li id="dli_09_0094__li6624653171317">If the Spark version is 3.1.1, you do not need to select a module. Configure <strong id="dli_09_0094__b79071120175917">Spark parameters (--conf)</strong>.<p id="dli_09_0094__p292175763011">spark.driver.extraClassPath=/usr/share/extension/dli/spark-jar/datasource/redis/*</p>
<p id="dli_09_0094__p392165713308">spark.executor.extraClassPath=/usr/share/extension/dli/spark-jar/datasource/redis/*</p>
</li></ul>
</div></div>
</div>
</li></ol>
</li></ul>
</div>
<div class="section" id="dli_09_0094__section1296017471949"><h4 class="sectiontitle">Complete Example Code</h4><ul id="dli_09_0094__ul251411571247"><li id="dli_09_0094__li1051465719415">Maven dependency<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509989_screen5760163172012"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span></pre></div></td><td class="code"><div><pre><span></span><span class="o">&lt;</span><span class="n">dependency</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">groupId</span><span class="o">&gt;</span><span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">spark</span><span class="o">&lt;/</span><span class="n">groupId</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">artifactId</span><span class="o">&gt;</span><span class="n">spark</span><span class="o">-</span><span class="n">sql_2</span><span class="mf">.11</span><span class="o">&lt;/</span><span class="n">artifactId</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">version</span><span class="o">&gt;</span><span class="mf">2.3.2</span><span class="o">&lt;/</span><span class="n">version</span><span class="o">&gt;</span>
<span class="o">&lt;/</span><span class="n">dependency</span><span class="o">&gt;</span>
<span class="o">&lt;</span><span class="n">dependency</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">groupId</span><span class="o">&gt;</span><span class="n">redis</span><span class="p">.</span><span class="n">clients</span><span class="o">&lt;/</span><span class="n">groupId</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">artifactId</span><span class="o">&gt;</span><span class="n">jedis</span><span class="o">&lt;/</span><span class="n">artifactId</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">version</span><span class="o">&gt;</span><span class="mf">3.1.0</span><span class="o">&lt;/</span><span class="n">version</span><span class="o">&gt;</span>
<span class="o">&lt;/</span><span class="n">dependency</span><span class="o">&gt;</span>
<span class="o">&lt;</span><span class="n">dependency</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">groupId</span><span class="o">&gt;</span><span class="n">com</span><span class="p">.</span><span class="n">redislabs</span><span class="o">&lt;/</span><span class="n">groupId</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">artifactId</span><span class="o">&gt;</span><span class="n">spark</span><span class="o">-</span><span class="n">redis</span><span class="o">&lt;/</span><span class="n">artifactId</span><span class="o">&gt;</span>
<span class="w"> </span><span class="o">&lt;</span><span class="n">version</span><span class="o">&gt;</span><span class="mf">2.4.0</span><span class="o">&lt;/</span><span class="n">version</span><span class="o">&gt;</span>
<span class="o">&lt;/</span><span class="n">dependency</span><span class="o">&gt;</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0094__li1936671714513">Connecting to data sources through SQL APIs<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509989_screen144461426184015"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">import</span><span class="w"> </span><span class="nn">org</span><span class="p">.</span><span class="nn">apache</span><span class="p">.</span><span class="nn">spark</span><span class="p">.</span><span class="nn">sql</span><span class="p">.{</span><span class="nc">SparkSession</span><span class="p">}</span><span class="err">;</span>
<span class="k">object</span><span class="w"> </span><span class="nc">Test_Redis_SQL</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">def</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">:</span><span class="w"> </span><span class="nc">Array</span><span class="p">[</span><span class="nc">String</span><span class="p">]):</span><span class="w"> </span><span class="nc">Unit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// Create a SparkSession session. </span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">sparkSession</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nc">SparkSession</span><span class="p">.</span><span class="n">builder</span><span class="p">().</span><span class="n">appName</span><span class="p">(</span><span class="s">&quot;datasource_redis&quot;</span><span class="p">).</span><span class="n">getOrCreate</span><span class="p">();</span><span class="w"> </span>
<span class="w"> </span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span>
<span class="w"> </span><span class="s">&quot;CREATE TEMPORARY VIEW person (name STRING, age INT) USING org.apache.spark.sql.redis OPTIONS (</span>
<span class="s"> 'host' = '192.168.4.199', 'port' = '6379', 'password' = '******',table 'person')&quot;</span><span class="p">.</span><span class="n">stripMargin</span><span class="p">)</span><span class="w"> </span>
<span class="w"> </span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">&quot;INSERT INTO TABLE person VALUES ('John', 30),('Peter', 45)&quot;</span><span class="p">.</span><span class="n">stripMargin</span><span class="p">)</span><span class="w"> </span>
<span class="w"> </span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sql</span><span class="p">(</span><span class="s">&quot;SELECT * FROM person&quot;</span><span class="p">.</span><span class="n">stripMargin</span><span class="p">).</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">(</span><span class="n">println</span><span class="p">)</span>
<span class="w"> </span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="n">close</span><span class="p">()</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0094__li1640095910911"><a name="dli_09_0094__li1640095910911"></a><a name="li1640095910911"></a>Connecting to data sources through DataFrame APIs<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509989_screen18873163294917"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span>
<span class="normal">27</span>
<span class="normal">28</span>
<span class="normal">29</span>
<span class="normal">30</span>
<span class="normal">31</span>
<span class="normal">32</span>
<span class="normal">33</span>
<span class="normal">34</span>
<span class="normal">35</span>
<span class="normal">36</span>
<span class="normal">37</span>
<span class="normal">38</span>
<span class="normal">39</span>
<span class="normal">40</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">import</span><span class="w"> </span><span class="nn">org</span><span class="p">.</span><span class="nn">apache</span><span class="p">.</span><span class="nn">spark</span><span class="p">.</span><span class="nn">sql</span><span class="p">.{</span><span class="nc">Row</span><span class="p">,</span><span class="w"> </span><span class="nc">SaveMode</span><span class="p">,</span><span class="w"> </span><span class="nc">SparkSession</span><span class="p">}</span>
<span class="k">import</span><span class="w"> </span><span class="nn">org</span><span class="p">.</span><span class="nn">apache</span><span class="p">.</span><span class="nn">spark</span><span class="p">.</span><span class="nn">sql</span><span class="p">.</span><span class="nn">types</span><span class="p">.</span><span class="n">_</span>
<span class="k">object</span><span class="w"> </span><span class="nc">Test_Redis_SparkSql</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">def</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">:</span><span class="w"> </span><span class="nc">Array</span><span class="p">[</span><span class="nc">String</span><span class="p">]):</span><span class="w"> </span><span class="nc">Unit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// Create a SparkSession session. </span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">sparkSession</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nc">SparkSession</span><span class="p">.</span><span class="n">builder</span><span class="p">().</span><span class="n">appName</span><span class="p">(</span><span class="s">&quot;datasource_redis&quot;</span><span class="p">).</span><span class="n">getOrCreate</span><span class="p">()</span>
<span class="w"> </span><span class="c1">// Set cross-source connection parameters. </span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">host</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&quot;192.168.4.199&quot;</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">port</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&quot;6379&quot;</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">table</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&quot;person&quot;</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">auth</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&quot;######&quot;</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">key_column</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&quot;name&quot;</span>
<span class="w"> </span>
<span class="w"> </span><span class="c1">// ******** setting DataFrame ******** </span>
<span class="w"> </span><span class="c1">// method one</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="n">schema</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nc">StructType</span><span class="p">(</span><span class="nc">Seq</span><span class="p">(</span><span class="nc">StructField</span><span class="p">(</span><span class="s">&quot;name&quot;</span><span class="p">,</span><span class="w"> </span><span class="nc">StringType</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">),</span><span class="nc">StructField</span><span class="p">(</span><span class="s">&quot;age&quot;</span><span class="p">,</span><span class="w"> </span><span class="nc">IntegerType</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">)))</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="n">rdd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="n">sparkContext</span><span class="p">.</span><span class="n">parallelize</span><span class="p">(</span><span class="nc">Seq</span><span class="p">(</span><span class="nc">Row</span><span class="p">(</span><span class="s">&quot;xxx&quot;</span><span class="p">,</span><span class="mi">34</span><span class="p">),</span><span class="nc">Row</span><span class="p">(</span><span class="s">&quot;Bob&quot;</span><span class="p">,</span><span class="mi">19</span><span class="p">)))</span>
<span class="w"> </span><span class="kd">var</span><span class="w"> </span><span class="n">dataFrame</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="n">createDataFrame</span><span class="p">(</span><span class="n">rdd</span><span class="p">,</span><span class="w"> </span><span class="n">schema</span><span class="p">)</span>
<span class="c1">// // method two</span>
<span class="c1">// var jdbcDF= sparkSession.createDataFrame(Seq((&quot;Jack&quot;,23)))</span>
<span class="c1">// val dataFrame = jdbcDF.withColumnRenamed(&quot;_1&quot;, &quot;name&quot;).withColumnRenamed(&quot;_2&quot;, &quot;age&quot;)</span>
<span class="c1">// // method three</span>
<span class="c1">// val dataFrame = sparkSession.createDataFrame(Seq(Person(&quot;John&quot;, 30), Person(&quot;Peter&quot;, 45)))</span>
<span class="w"> </span><span class="c1">// Write data to redis </span>
<span class="w"> </span><span class="n">dataFrame</span><span class="p">.</span><span class="n">write</span><span class="p">.</span><span class="n">format</span><span class="p">(</span><span class="s">&quot;redis&quot;</span><span class="p">).</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;host&quot;</span><span class="p">,</span><span class="n">host</span><span class="p">).</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;port&quot;</span><span class="p">,</span><span class="n">port</span><span class="p">).</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;table&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">table</span><span class="p">).</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;password&quot;</span><span class="p">,</span><span class="n">auth</span><span class="p">).</span><span class="n">mode</span><span class="p">(</span><span class="nc">SaveMode</span><span class="p">.</span><span class="nc">Overwrite</span><span class="p">).</span><span class="n">save</span><span class="p">()</span>
<span class="w"> </span><span class="c1">// Read data from redis </span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="n">read</span><span class="p">.</span><span class="n">format</span><span class="p">(</span><span class="s">&quot;redis&quot;</span><span class="p">).</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;host&quot;</span><span class="p">,</span><span class="n">host</span><span class="p">).</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;port&quot;</span><span class="p">,</span><span class="n">port</span><span class="p">).</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;table&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">table</span><span class="p">).</span><span class="n">option</span><span class="p">(</span><span class="s">&quot;password&quot;</span><span class="p">,</span><span class="n">auth</span><span class="p">).</span><span class="n">load</span><span class="p">().</span><span class="n">show</span><span class="p">()</span>
<span class="w"> </span><span class="c1">// Close session </span>
<span class="w"> </span><span class="n">sparkSession</span><span class="p">.</span><span class="n">close</span><span class="p">()</span><span class="w"> </span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
<span class="c1">// methoe two</span>
<span class="c1">// case class Person(name: String, age: Int)</span>
</pre></div></td></tr></table></div>
</div>
</li><li id="dli_09_0094__li82426312102">Connecting to data sources using Spark RDDs<div class="codecoloring" codetype="Scala" id="dli_09_0094__en-us_topic_0200509989_screen3740131813160"><div class="highlight"><table class="highlighttable"><tr><td class="linenos"><div class="linenodiv"><pre><span class="normal"> 1</span>
<span class="normal"> 2</span>
<span class="normal"> 3</span>
<span class="normal"> 4</span>
<span class="normal"> 5</span>
<span class="normal"> 6</span>
<span class="normal"> 7</span>
<span class="normal"> 8</span>
<span class="normal"> 9</span>
<span class="normal">10</span>
<span class="normal">11</span>
<span class="normal">12</span>
<span class="normal">13</span>
<span class="normal">14</span>
<span class="normal">15</span>
<span class="normal">16</span>
<span class="normal">17</span>
<span class="normal">18</span>
<span class="normal">19</span>
<span class="normal">20</span>
<span class="normal">21</span>
<span class="normal">22</span>
<span class="normal">23</span>
<span class="normal">24</span>
<span class="normal">25</span>
<span class="normal">26</span>
<span class="normal">27</span>
<span class="normal">28</span>
<span class="normal">29</span>
<span class="normal">30</span>
<span class="normal">31</span>
<span class="normal">32</span>
<span class="normal">33</span>
<span class="normal">34</span>
<span class="normal">35</span>
<span class="normal">36</span>
<span class="normal">37</span>
<span class="normal">38</span>
<span class="normal">39</span>
<span class="normal">40</span>
<span class="normal">41</span>
<span class="normal">42</span>
<span class="normal">43</span>
<span class="normal">44</span>
<span class="normal">45</span>
<span class="normal">46</span>
<span class="normal">47</span>
<span class="normal">48</span>
<span class="normal">49</span>
<span class="normal">50</span>
<span class="normal">51</span>
<span class="normal">52</span>
<span class="normal">53</span>
<span class="normal">54</span>
<span class="normal">55</span>
<span class="normal">56</span>
<span class="normal">57</span>
<span class="normal">58</span>
<span class="normal">59</span>
<span class="normal">60</span>
<span class="normal">61</span>
<span class="normal">62</span>
<span class="normal">63</span>
<span class="normal">64</span>
<span class="normal">65</span>
<span class="normal">66</span>
<span class="normal">67</span>
<span class="normal">68</span>
<span class="normal">69</span>
<span class="normal">70</span></pre></div></td><td class="code"><div><pre><span></span><span class="k">import</span><span class="w"> </span><span class="nn">com</span><span class="p">.</span><span class="nn">redislabs</span><span class="p">.</span><span class="nn">provider</span><span class="p">.</span><span class="nn">redis</span><span class="p">.</span><span class="n">_</span>
<span class="k">import</span><span class="w"> </span><span class="nn">org</span><span class="p">.</span><span class="nn">apache</span><span class="p">.</span><span class="nn">spark</span><span class="p">.</span><span class="nn">rdd</span><span class="p">.</span><span class="nc">RDD</span>
<span class="k">import</span><span class="w"> </span><span class="nn">org</span><span class="p">.</span><span class="nn">apache</span><span class="p">.</span><span class="nn">spark</span><span class="p">.{</span><span class="nc">SparkConf</span><span class="p">,</span><span class="w"> </span><span class="nc">SparkContext</span><span class="p">}</span>
<span class="k">object</span><span class="w"> </span><span class="nc">Test_Redis_RDD</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="k">def</span><span class="w"> </span><span class="nf">main</span><span class="p">(</span><span class="n">args</span><span class="p">:</span><span class="w"> </span><span class="nc">Array</span><span class="p">[</span><span class="nc">String</span><span class="p">]):</span><span class="w"> </span><span class="nc">Unit</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="c1">// Create a SparkSession session. </span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">sparkContext</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="nc">SparkContext</span><span class="p">(</span><span class="k">new</span><span class="w"> </span><span class="nc">SparkConf</span><span class="p">()</span>
<span class="w"> </span><span class="p">.</span><span class="n">setAppName</span><span class="p">(</span><span class="s">&quot;datasource_redis&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s">&quot;spark.redis.host&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;192.168.4.199&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s">&quot;spark.redis.port&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;6379&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s">&quot;spark.redis.auth&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;@@@@@@&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="s">&quot;spark.driver.allowMultipleContexts&quot;</span><span class="p">,</span><span class="s">&quot;true&quot;</span><span class="p">))</span>
<span class="w"> </span><span class="c1">//***************** Write data to redis ********************** </span>
<span class="w"> </span><span class="c1">// Save String type data </span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">stringRedisData</span><span class="p">:</span><span class="nc">RDD</span><span class="p">[(</span><span class="nc">String</span><span class="p">,</span><span class="nc">String</span><span class="p">)]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">parallelize</span><span class="p">(</span><span class="nc">Seq</span><span class="p">[(</span><span class="nc">String</span><span class="p">,</span><span class="nc">String</span><span class="p">)]((</span><span class="s">&quot;high&quot;</span><span class="p">,</span><span class="s">&quot;111&quot;</span><span class="p">),</span><span class="w"> </span><span class="p">(</span><span class="s">&quot;together&quot;</span><span class="p">,</span><span class="s">&quot;333&quot;</span><span class="p">)))</span><span class="w"> </span>
<span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">toRedisKV</span><span class="p">(</span><span class="n">stringRedisData</span><span class="p">)</span>
<span class="w"> </span>
<span class="w"> </span><span class="c1">// Save Hash type data </span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">hashRedisData</span><span class="p">:</span><span class="nc">RDD</span><span class="p">[(</span><span class="nc">String</span><span class="p">,</span><span class="nc">String</span><span class="p">)]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">parallelize</span><span class="p">(</span><span class="nc">Seq</span><span class="p">[(</span><span class="nc">String</span><span class="p">,</span><span class="nc">String</span><span class="p">)]((</span><span class="s">&quot;saprk&quot;</span><span class="p">,</span><span class="s">&quot;123&quot;</span><span class="p">),</span><span class="w"> </span><span class="p">(</span><span class="s">&quot;data&quot;</span><span class="p">,</span><span class="s">&quot;222&quot;</span><span class="p">)))</span><span class="w"> </span>
<span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">toRedisHASH</span><span class="p">(</span><span class="n">hashRedisData</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;hashRDD&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="c1">// Save List type data </span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nc">List</span><span class="p">((</span><span class="s">&quot;school&quot;</span><span class="p">,</span><span class="s">&quot;112&quot;</span><span class="p">),</span><span class="w"> </span><span class="p">(</span><span class="s">&quot;tom&quot;</span><span class="p">,</span><span class="s">&quot;333&quot;</span><span class="p">));</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">listRedisData</span><span class="p">:</span><span class="nc">RDD</span><span class="p">[</span><span class="nc">String</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">parallelize</span><span class="p">(</span><span class="nc">Seq</span><span class="p">[(</span><span class="nc">String</span><span class="p">)](</span><span class="n">data</span><span class="p">.</span><span class="n">toString</span><span class="p">()))</span>
<span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">toRedisLIST</span><span class="p">(</span><span class="n">listRedisData</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;listRDD&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="c1">// Save Set type data </span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">setData</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nc">Set</span><span class="p">((</span><span class="s">&quot;bob&quot;</span><span class="p">,</span><span class="s">&quot;133&quot;</span><span class="p">),(</span><span class="s">&quot;kity&quot;</span><span class="p">,</span><span class="s">&quot;322&quot;</span><span class="p">))</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">setRedisData</span><span class="p">:</span><span class="nc">RDD</span><span class="p">[(</span><span class="nc">String</span><span class="p">)]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">parallelize</span><span class="p">(</span><span class="nc">Seq</span><span class="p">[(</span><span class="nc">String</span><span class="p">)](</span><span class="n">setData</span><span class="p">.</span><span class="n">mkString</span><span class="p">))</span>
<span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">toRedisSET</span><span class="p">(</span><span class="n">setRedisData</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;setRDD&quot;</span><span class="p">)</span>
<span class="w"> </span>
<span class="w"> </span><span class="c1">// Save ZSet type data </span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">zsetRedisData</span><span class="p">:</span><span class="nc">RDD</span><span class="p">[(</span><span class="nc">String</span><span class="p">,</span><span class="nc">String</span><span class="p">)]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">parallelize</span><span class="p">(</span><span class="nc">Seq</span><span class="p">[(</span><span class="nc">String</span><span class="p">,</span><span class="nc">String</span><span class="p">)]((</span><span class="s">&quot;whight&quot;</span><span class="p">,</span><span class="s">&quot;234&quot;</span><span class="p">),</span><span class="w"> </span><span class="p">(</span><span class="s">&quot;bobo&quot;</span><span class="p">,</span><span class="s">&quot;343&quot;</span><span class="p">)))</span><span class="w"> </span>
<span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">toRedisZSET</span><span class="p">(</span><span class="n">zsetRedisData</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;zsetRDD&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="c1">// ***************************** Read data from redis ******************************************* </span>
<span class="w"> </span><span class="c1">// Traverse the specified key and get the value</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">keysRDD</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisKeys</span><span class="p">(</span><span class="nc">Array</span><span class="p">(</span><span class="s">&quot;high&quot;</span><span class="p">,</span><span class="s">&quot;together&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;hashRDD&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;listRDD&quot;</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;setRDD&quot;</span><span class="p">,</span><span class="s">&quot;zsetRDD&quot;</span><span class="p">),</span><span class="w"> </span><span class="mi">6</span><span class="p">)</span><span class="w"> </span>
<span class="w"> </span><span class="n">keysRDD</span><span class="p">.</span><span class="n">getKV</span><span class="p">().</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">(</span><span class="n">println</span><span class="p">)</span>
<span class="w"> </span><span class="n">keysRDD</span><span class="p">.</span><span class="n">getHash</span><span class="p">().</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">(</span><span class="n">println</span><span class="p">)</span>
<span class="w"> </span><span class="n">keysRDD</span><span class="p">.</span><span class="n">getList</span><span class="p">().</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">(</span><span class="n">println</span><span class="p">)</span>
<span class="w"> </span><span class="n">keysRDD</span><span class="p">.</span><span class="n">getSet</span><span class="p">().</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">(</span><span class="n">println</span><span class="p">)</span>
<span class="w"> </span><span class="n">keysRDD</span><span class="p">.</span><span class="n">getZSet</span><span class="p">().</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">(</span><span class="n">println</span><span class="p">)</span>
<span class="w"> </span>
<span class="w"> </span><span class="c1">// Read String type data//</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">stringRDD</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisKV</span><span class="p">(</span><span class="s">&quot;keyPattern *&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisKV</span><span class="p">(</span><span class="nc">Array</span><span class="p">(</span><span class="w"> </span><span class="s">&quot;high&quot;</span><span class="p">,</span><span class="s">&quot;together&quot;</span><span class="p">)).</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">{</span><span class="n">println</span><span class="p">}</span>
<span class="w"> </span>
<span class="w"> </span><span class="c1">// Read Hash type data//</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">hashRDD</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisHash</span><span class="p">(</span><span class="s">&quot;keyPattern *&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisHash</span><span class="p">(</span><span class="nc">Array</span><span class="p">(</span><span class="s">&quot;hashRDD&quot;</span><span class="p">)).</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">{</span><span class="n">println</span><span class="p">}</span>
<span class="w"> </span>
<span class="w"> </span><span class="c1">// Read List type data//</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">listRDD</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisList</span><span class="p">(</span><span class="s">&quot;keyPattern *&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisList</span><span class="p">(</span><span class="nc">Array</span><span class="p">(</span><span class="s">&quot;listRDD&quot;</span><span class="p">)).</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">{</span><span class="n">println</span><span class="p">}</span>
<span class="w"> </span>
<span class="w"> </span><span class="c1">// Read Set type data//</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">setRDD</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisSet</span><span class="p">(</span><span class="s">&quot;keyPattern *&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisSet</span><span class="p">(</span><span class="nc">Array</span><span class="p">(</span><span class="s">&quot;setRDD&quot;</span><span class="p">)).</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">{</span><span class="n">println</span><span class="p">}</span>
<span class="w"> </span><span class="c1">// Read ZSet type data//</span>
<span class="w"> </span><span class="kd">val</span><span class="w"> </span><span class="n">zsetRDD</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisZSet</span><span class="p">(</span><span class="s">&quot;keyPattern *&quot;</span><span class="p">)</span>
<span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">fromRedisZSet</span><span class="p">(</span><span class="nc">Array</span><span class="p">(</span><span class="s">&quot;zsetRDD&quot;</span><span class="p">)).</span><span class="n">collect</span><span class="p">().</span><span class="n">foreach</span><span class="p">{</span><span class="n">println</span><span class="p">}</span>
<span class="w"> </span>
<span class="w"> </span><span class="c1">// close session</span>
<span class="w"> </span><span class="n">sparkContext</span><span class="p">.</span><span class="n">stop</span><span class="p">()</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</pre></div></td></tr></table></div>
</div>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_09_0093.html">Connecting to Redis</a></div>
</div>
</div>
View Git Blame Copy Permalink