vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/umn/dli_03_0017.html
Su, Xiaomeng fdd43c552e dli_umn_20240808 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-08-09 11:00:57 +00:00

2.6 KiB
Raw Blame History

How Do I Set the AK/SK for a Queue to Operate an OBS Table?

Setting Up a Spark Jar Job to Obtain the AK/SK

  • To obtain the AK/SK, set the parameters as follows:
    • Create a SparkContext using code.
      val sc: SparkContext = new SparkContext()
sc.hadoopConfiguration.set("fs.obs.access.key", ak)
sc.hadoopConfiguration.set("fs.obs.secret.key", sk)
    • Create a SparkSession using code.
      val sparkSession: SparkSession = SparkSession
      .builder()
      .config("spark.hadoop.fs.obs.access.key", ak)
      .config("spark.hadoop.fs.obs.secret.key", sk)
      .enableHiveSupport()
      .getOrCreate()
  • To obtain the AK/SK and security token and use them together for authentication, set the parameters as follows:
    • Create a SparkContext using code.
      val sc: SparkContext = new SparkContext()
sc.hadoopConfiguration.set("fs.obs.access.key", ak)
sc.hadoopConfiguration.set("fs.obs.secret.key", sk)
sc.hadoopConfiguration.set("fs.obs.session.token", sts)
    • Create a SparkSession using code.
      val sparkSession: SparkSession = SparkSession
      .builder()
      .config("spark.hadoop.fs.obs.access.key", ak)
      .config("spark.hadoop.fs.obs.secret.key", sk)
      .config("spark.hadoop.fs.obs.session.token", sts)
      .enableHiveSupport()
      .getOrCreate()
Parent topic: Job Development