vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/cce/umn/cce_bestpractice_00281.html
Dong, Qiu Jian 86fb05065f CCE UMN for 24.2.0 version -20240428 
Reviewed-by: Eotvos, Oliver <oliver.eotvos@t-systems.com>
Co-authored-by: Dong, Qiu Jian <qiujiandong1@huawei.com>
Co-committed-by: Dong, Qiu Jian <qiujiandong1@huawei.com>
2024-06-10 08:19:07 +00:00

430 lines
42 KiB
HTML
Raw Blame History

<a name="cce_bestpractice_00281"></a><a name="cce_bestpractice_00281"></a>
<h1 class="topictitle1">Custom Storage Classes</h1>
<div id="body0000001148435277"><div class="section" id="cce_bestpractice_00281__section13347243125312"><h4 class="sectiontitle">Background</h4><p id="cce_bestpractice_00281__p19810642103411">When using storage resources in CCE, the most common method is to specify <strong id="cce_bestpractice_00281__b1439518042014">storageClassName</strong> to define the type of storage resources to be created when creating a PVC. The following configuration shows how to use a PVC to apply for an SAS (high I/O) EVS disk (block storage).</p>
<pre class="screen" id="cce_bestpractice_00281__screen146941631606">apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-evs-example
namespace: default
annotations:
everest.io/disk-volume-type: SAS
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: csi-disk</pre>
<p id="cce_bestpractice_00281__p10416164023811">To specify the EVS disk type, you can configure the <strong id="cce_bestpractice_00281__b196871918456">everest.io/disk-volume-type</strong> field. The value <strong id="cce_bestpractice_00281__b86884994512">SAS</strong> is used as an example here, indicating the high I/O EVS disk type. Or you can choose <strong id="cce_bestpractice_00281__b26888913457">SATA</strong> (common I/O) and <strong id="cce_bestpractice_00281__b116899934511">SSD</strong> (ultra-high I/O).</p>
<p id="cce_bestpractice_00281__p1992812244296">This configuration method may not work if you want to:</p>
<ul id="cce_bestpractice_00281__ul1666681113305"><li id="cce_bestpractice_00281__li26671211203012">Set <strong id="cce_bestpractice_00281__b7751649116">storageClassName</strong> only, which is simpler than specifying the EVS disk type by using <strong id="cce_bestpractice_00281__b147618498116">everest.io/disk-volume-type</strong>.</li><li id="cce_bestpractice_00281__li1430122517314">Avoid modifying YAML files or Helm charts. Some users switch from self-built or other Kubernetes services to CCE and have written YAML files of many applications. In these YAML files, different types of storage resources are specified by different StorageClassNames. When using CCE, they need to modify a large number of YAML files or Helm charts to use storage resources, which is labor-consuming and error-prone.</li><li id="cce_bestpractice_00281__li6714783711">Set the default <strong id="cce_bestpractice_00281__b3414714817">storageClassName</strong> for all applications to use the default storage class. In this way, you can create storage resources of the default type without needing to specify <strong id="cce_bestpractice_00281__b15163103104815">storageClassName</strong> in the YAML file.</li></ul>
</div>
<div class="section" id="cce_bestpractice_00281__section10387181592813"><h4 class="sectiontitle">Solution</h4><p id="cce_bestpractice_00281__p33361127181019">This section describes how to set a custom storage class in CCE and how to set the default storage class. You can specify different types of storage resources by setting <strong id="cce_bestpractice_00281__b15705152195010">storageClassName</strong>.</p>
<ul id="cce_bestpractice_00281__ul1394816541798"><li id="cce_bestpractice_00281__li1494865412920">For the first scenario, you can define custom storageClassNames for SAS and SSD EVS disks. For example, define a storage class named <strong id="cce_bestpractice_00281__b11954204618017">csi-disk-sas</strong> for creating SAS disks. The following figure shows the differences before and after you use a custom storage class.<p id="cce_bestpractice_00281__p1639843625220"><span><img id="cce_bestpractice_00281__image62065521613" src="en-us_image_0000001897905209.png"></span></p>
</li><li id="cce_bestpractice_00281__li112521973108">For the second scenario, you can define a storage class with the same name as that in the existing YAML file without needing to modify <strong id="cce_bestpractice_00281__b13346387104">storageClassName</strong> in the YAML file.</li><li id="cce_bestpractice_00281__li595443471215">For the third scenario, you can set the default storage class as described below to create storage resources without specifying <strong id="cce_bestpractice_00281__b863812014124">storageClassName</strong> in YAML files.<pre class="screen" id="cce_bestpractice_00281__screen22067304183">apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-evs-example
namespace: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi</pre>
</li></ul>
</div>
<div class="section" id="cce_bestpractice_00281__section11887112131418"><h4 class="sectiontitle">CCE Default Storage Classes</h4><p id="cce_bestpractice_00281__cce_10_0380_p6816345818">As of now, CCE provides storage classes such as csi-disk, csi-nas, and csi-obs by default. When defining a PVC, you can use a <strong id="cce_bestpractice_00281__cce_10_0380_b449415904410">storageClassName</strong> to automatically create a PV of the corresponding type and automatically create underlying storage resources.</p>
<p id="cce_bestpractice_00281__cce_10_0380_p5271142713310">Run the following kubectl command to obtain the storage classes that CCE supports. Use the CSI add-on provided by CCE to create a storage class.</p>
<pre class="screen" id="cce_bestpractice_00281__cce_10_0380_screen2027110274334"># kubectl get sc
NAME PROVISIONER AGE
csi-disk everest-csi-provisioner 17d # EVS disk
csi-disk-topology everest-csi-provisioner 17d # EVS disks created with delayed
csi-nas everest-csi-provisioner 17d # SFS 1.0
csi-obs everest-csi-provisioner 17d # OBS
csi-sfsturbo everest-csi-provisioner 17d # SFS Turbo</pre>
<div class="p" id="cce_bestpractice_00281__cce_10_0380_p18907106309">Each storage class contains the default parameters used for dynamically creating a PV. The following is an example of storage class for EVS disks:<pre class="screen" id="cce_bestpractice_00281__cce_10_0380_screen15484956183012">kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: csi-disk
<strong id="cce_bestpractice_00281__cce_10_0380_b1163493614338">provisioner: everest-csi-provisioner</strong>
<strong id="cce_bestpractice_00281__cce_10_0380_b4960133815330">parameters:</strong>
csi.storage.k8s.io/csi-driver-name: disk.csi.everest.io
csi.storage.k8s.io/fstype: ext4 # (Optional) Set the file system type to <strong id="cce_bestpractice_00281__cce_10_0380_b20965430133916">xfs</strong> or <strong id="cce_bestpractice_00281__cce_10_0380_b199655307391">ext4</strong>. If it is left blank, <strong id="cce_bestpractice_00281__cce_10_0380_b496510308398">ext4</strong> is used by default.
everest.io/disk-volume-type: SAS
everest.io/passthrough: 'true'
<strong id="cce_bestpractice_00281__cce_10_0380_b35742193319">reclaimPolicy: Delete</strong>
<strong id="cce_bestpractice_00281__cce_10_0380_b154087445332">allowVolumeExpansion: true</strong>
<strong id="cce_bestpractice_00281__cce_10_0380_b54085441337">volumeBindingMode: Immediate</strong></pre>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="cce_bestpractice_00281__cce_10_0380_table31743016321" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Key parameters</caption><thead align="left"><tr id="cce_bestpractice_00281__cce_10_0380_row517510016325"><th align="left" class="cellrowborder" valign="top" width="26.55%" id="mcps1.3.3.5.2.2.3.1.1"><p id="cce_bestpractice_00281__cce_10_0380_p12175160183211">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="73.45%" id="mcps1.3.3.5.2.2.3.1.2"><p id="cce_bestpractice_00281__cce_10_0380_p41759015322">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="cce_bestpractice_00281__cce_10_0380_row317517012329"><td class="cellrowborder" valign="top" width="26.55%" headers="mcps1.3.3.5.2.2.3.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p101751907321">provisioner</p>
</td>
<td class="cellrowborder" valign="top" width="73.45%" headers="mcps1.3.3.5.2.2.3.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p2017540183211">Specifies the storage resource provider, which is the Everest add-on for CCE. Set this parameter to <strong id="cce_bestpractice_00281__cce_10_0380_b19256101516334">everest-csi-provisioner</strong>.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row101757012320"><td class="cellrowborder" valign="top" width="26.55%" headers="mcps1.3.3.5.2.2.3.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p8175180143213">parameters</p>
</td>
<td class="cellrowborder" valign="top" width="73.45%" headers="mcps1.3.3.5.2.2.3.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p13175150183217">Specifies the storage parameters, which vary with storage types. For details, see <a href="#cce_bestpractice_00281__cce_10_0380_table15415188175413">Table 2</a>.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row3175200173211"><td class="cellrowborder" valign="top" width="26.55%" headers="mcps1.3.3.5.2.2.3.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p5175160103210">reclaimPolicy</p>
</td>
<td class="cellrowborder" valign="top" width="73.45%" headers="mcps1.3.3.5.2.2.3.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p18949625163517">Specifies the value of <strong id="cce_bestpractice_00281__cce_10_0380_b1731819915357">persistentVolumeReclaimPolicy</strong> for creating a PV. The value can be <strong id="cce_bestpractice_00281__cce_10_0380_b8552143355">Delete</strong> or <strong id="cce_bestpractice_00281__cce_10_0380_b33571416133515">Retain</strong>. If <strong id="cce_bestpractice_00281__cce_10_0380_b1548124873516">reclaimPolicy</strong> is not specified when a StorageClass object is created, the value defaults to <strong id="cce_bestpractice_00281__cce_10_0380_b17346143983611">Delete</strong>.</p>
<ul id="cce_bestpractice_00281__cce_10_0380_ul169491925103515"><li id="cce_bestpractice_00281__cce_10_0380_li594992515358"><strong id="cce_bestpractice_00281__cce_10_0380_b347294583618">Delete</strong>: indicates that a dynamically created PV will be automatically destroyed.</li><li id="cce_bestpractice_00281__cce_10_0380_li14949325163516"><strong id="cce_bestpractice_00281__cce_10_0380_b18552142618375">Retain</strong>: indicates that a dynamically created PV will not be automatically destroyed.</li></ul>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row895192617325"><td class="cellrowborder" valign="top" width="26.55%" headers="mcps1.3.3.5.2.2.3.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p7951726113217">allowVolumeExpansion</p>
</td>
<td class="cellrowborder" valign="top" width="73.45%" headers="mcps1.3.3.5.2.2.3.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p395426173217">Specifies whether the PV of this storage class supports dynamic capacity expansion. The default value is <strong id="cce_bestpractice_00281__cce_10_0380_b16115144215388">false</strong>. Dynamic capacity expansion is implemented by the underlying storage add-on. This is only a switch.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row79248308324"><td class="cellrowborder" valign="top" width="26.55%" headers="mcps1.3.3.5.2.2.3.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p792483063217">volumeBindingMode</p>
</td>
<td class="cellrowborder" valign="top" width="73.45%" headers="mcps1.3.3.5.2.2.3.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p18924030103210">Specifies the volume binding mode, that is, the time when a PV is dynamically created. The value can be <strong id="cce_bestpractice_00281__cce_10_0380_b19898101400">Immediate</strong> or <strong id="cce_bestpractice_00281__cce_10_0380_b3336116104018">WaitForFirstConsumer</strong>.</p>
<ul id="cce_bestpractice_00281__cce_10_0380_ul541043753817"><li id="cce_bestpractice_00281__cce_10_0380_li184104378381"><strong id="cce_bestpractice_00281__cce_10_0380_b17478101684017">Immediate</strong>: PV binding and dynamic creation are completed when a PVC is created.</li><li id="cce_bestpractice_00281__cce_10_0380_li35152173915"><strong id="cce_bestpractice_00281__cce_10_0380_b975442814405">WaitForFirstConsumer</strong>: PV binding and creation are delayed. The PV creation and binding processes are executed only when the PVC is used in the workload.</li></ul>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row47091406405"><td class="cellrowborder" valign="top" width="26.55%" headers="mcps1.3.3.5.2.2.3.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p1570914409400">mountOptions</p>
</td>
<td class="cellrowborder" valign="top" width="73.45%" headers="mcps1.3.3.5.2.2.3.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p167092040194019">This field must be supported by the underlying storage. If this field is not supported but is specified, the PV creation will fail.</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="tablenoborder"><a name="cce_bestpractice_00281__cce_10_0380_table15415188175413"></a><a name="cce_10_0380_table15415188175413"></a><table cellpadding="4" cellspacing="0" summary="" id="cce_bestpractice_00281__cce_10_0380_table15415188175413" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Parameters</caption><thead align="left"><tr id="cce_bestpractice_00281__cce_10_0380_row5416785549"><th align="left" class="cellrowborder" valign="top" width="11.379999999999999%" id="mcps1.3.3.5.3.2.5.1.1"><p id="cce_bestpractice_00281__cce_10_0380_p11404149135819">Volume Type</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="25.56%" id="mcps1.3.3.5.3.2.5.1.2"><p id="cce_bestpractice_00281__cce_10_0380_p3416888545">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="11.63%" id="mcps1.3.3.5.3.2.5.1.3"><p id="cce_bestpractice_00281__cce_10_0380_p1265895014553">Mandatory</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="51.43%" id="mcps1.3.3.5.3.2.5.1.4"><p id="cce_bestpractice_00281__cce_10_0380_p1141678165411">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="cce_bestpractice_00281__cce_10_0380_row941612810542"><td class="cellrowborder" rowspan="4" valign="top" width="11.379999999999999%" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p102718119016">EVS</p>
</td>
<td class="cellrowborder" valign="top" width="25.56%" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p17416168135419">csi.storage.k8s.io/csi-driver-name</p>
</td>
<td class="cellrowborder" valign="top" width="11.63%" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p11658850145513">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="51.43%" headers="mcps1.3.3.5.3.2.5.1.4 "><p id="cce_bestpractice_00281__cce_10_0380_p177039251438">Driver type. If an EVS disk is used, the parameter value is fixed at <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue146054172413"><b>disk.csi.everest.io</b></span>.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row4196200353"><td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p34166812546">csi.storage.k8s.io/fstype</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p18658650105516">Yes</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p18225379010">If an EVS disk is used, the parameter value can be <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue1871318221243"><b>ext4</b></span> or <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue11460152514413"><b>xfs</b></span>.</p>
<div class="p" id="cce_bestpractice_00281__cce_10_0380_p1521619251205">The restrictions on using <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue576411298414"><b>xfs</b></span> are as follows:<ul id="cce_bestpractice_00281__cce_10_0380_en-us_topic_0000001492579890_ul3227931165915"><li id="cce_bestpractice_00281__cce_10_0380_en-us_topic_0000001492579890_li3268173318591">The nodes must run CentOS 7 or Ubuntu 22.04, and the Everest version in the cluster must be 2.3.2 or later.</li><li id="cce_bestpractice_00281__cce_10_0380_en-us_topic_0000001492579890_li022715319593">Only common containers are supported.</li></ul>
</div>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row13416481549"><td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p94161887548">everest.io/disk-volume-type</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p19658750105513">Yes</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.3 "><div class="p" id="cce_bestpractice_00281__cce_10_0380_p191376271512">EVS disk type. All letters are in uppercase.<ul id="cce_bestpractice_00281__cce_10_0380_cce_10_0614_ul1099913293217"><li id="cce_bestpractice_00281__cce_10_0380_cce_10_0614_li769621143010"><strong id="cce_bestpractice_00281__cce_10_0380_cce_10_0614_b571854437105541">SATA</strong>: common I/O</li><li id="cce_bestpractice_00281__cce_10_0380_cce_10_0614_li11999525321"><strong id="cce_bestpractice_00281__cce_10_0380_cce_10_0614_b1568717650105541">SAS</strong>: high I/O</li><li id="cce_bestpractice_00281__cce_10_0380_cce_10_0614_li199991629326"><strong id="cce_bestpractice_00281__cce_10_0380_cce_10_0614_b595954424105541">SSD</strong>: ultra-high I/O</li></ul>
</div>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row256611814565"><td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p256610810566">everest.io/passthrough</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p1356610825615">Yes</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p065423321315">The parameter value is fixed at <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue8875203711134"><b>true</b></span>, which indicates that the EVS device type is <strong id="cce_bestpractice_00281__cce_10_0380_b34051368211">SCSI</strong>. Other parameter values are not allowed.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row1967312234013"><td class="cellrowborder" rowspan="6" valign="top" width="11.379999999999999%" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p10936142812010">SFS</p>
</td>
<td class="cellrowborder" valign="top" width="25.56%" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p1582818247011">csi.storage.k8s.io/csi-driver-name</p>
</td>
<td class="cellrowborder" valign="top" width="11.63%" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p7828192419013">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="51.43%" headers="mcps1.3.3.5.3.2.5.1.4 "><p id="cce_bestpractice_00281__cce_10_0380_p188289246012">Driver type. If SFS is used, the parameter value is fixed at <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue38476351140"><b>nas.csi.everest.io</b></span>.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row2022892113012"><td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p18828112415011">csi.storage.k8s.io/fstype</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p98281424607">Yes</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p27738381637">If SFS is used, the value can be <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue1041520406416"><b>nfs</b></span>.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row20421151685616"><td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p169152465814">everest.io/share-access-level</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p1168142411585">Yes</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p16421111645610">The parameter value is fixed at <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue138731418101611"><b>rw</b></span>, indicating that the SFS data is readable and writable.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row205664321916"><td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p1566232515">everest.io/share-access-to</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p195677321212">Yes</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p15671232216">VPC ID of the cluster.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row5399138714"><td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p339913381013">everest.io/share-is-public</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p19399143819114">No</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p4399175941914">The parameter value is fixed at <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue8399155917194"><b>false</b></span>, indicating that the file is shared to private.</p>
<p id="cce_bestpractice_00281__cce_10_0380_p11399338618">You do not need to configure this parameter when SFS 3.0 is used.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row4121194610111"><td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p1612111462117">everest.io/sfs-version</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p161213465119">No</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p131217469118">This parameter is mandatory only when SFS 3.0 is used. The value is fixed at <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue12581647191619"><b>sfs3.0</b></span>.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row192323510015"><td class="cellrowborder" rowspan="6" valign="top" width="11.379999999999999%" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p14923535708">SFS Turbo</p>
</td>
<td class="cellrowborder" valign="top" width="25.56%" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p154388131829">csi.storage.k8s.io/csi-driver-name</p>
</td>
<td class="cellrowborder" valign="top" width="11.63%" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p1343819131220">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="51.43%" headers="mcps1.3.3.5.3.2.5.1.4 "><p id="cce_bestpractice_00281__cce_10_0380_p1743810134210">Driver type. If SFS Turbo is used, the parameter value is fixed at <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue1686164342020"><b>sfsturbo.csi.everest.io</b></span>.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row626614125"><td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p174380137212">csi.storage.k8s.io/fstype</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p16438121317219">Yes</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p1143812131021">If SFS Turbo is used, the value can be <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue1318926172113"><b>nfs</b></span>.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row138271317212"><td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p784316361408">everest.io/share-access-to</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p9843336305">Yes</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p18843936203">VPC ID of the cluster.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row4176696319"><td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p15176192037">everest.io/share-expand-type</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p5176109131">No</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p122035832513">Extension type. The default value is <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue52351314163212"><b>bandwidth</b></span>, indicating an enhanced file system. This parameter does not take effect.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row388310431332"><td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p1588317438319">everest.io/share-source</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p1088316437318">Yes</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p138834431231">The parameter value is fixed at <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue1533433215214"><b>sfs-turbo</b></span>.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row1782964118317"><td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p11829154119318">everest.io/share-volume-type</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p2082920411133">No</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p682918414317">SFS Turbo storage class. The default value is <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue329611315238"><b>STANDARD</b></span>, indicating standard and standard enhanced editions. This parameter does not take effect.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row4477271326"><td class="cellrowborder" rowspan="3" valign="top" width="11.379999999999999%" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p1405749135813">OBS</p>
</td>
<td class="cellrowborder" valign="top" width="25.56%" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p15210115532517">csi.storage.k8s.io/csi-driver-name</p>
</td>
<td class="cellrowborder" valign="top" width="11.63%" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p0210125582518">Yes</p>
</td>
<td class="cellrowborder" valign="top" width="51.43%" headers="mcps1.3.3.5.3.2.5.1.4 "><p id="cce_bestpractice_00281__cce_10_0380_p195591759192510">Driver type. If OBS is used, the parameter value is fixed at <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue8210155102512"><b>obs.csi.everest.io</b></span>.</p>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row137212044112517"><td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p721005517254">csi.storage.k8s.io/fstype</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p6210155552514">Yes</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p121005502511">Instance type, which can be <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue24158172292"><b>obsfs</b></span> or <span class="parmvalue" id="cce_bestpractice_00281__cce_10_0380_parmvalue1021020555258"><b>s3fs</b></span>.</p>
<ul id="cce_bestpractice_00281__cce_10_0380_ul6315195541919"><li id="cce_bestpractice_00281__cce_10_0380_li9622106182011"><strong id="cce_bestpractice_00281__cce_10_0380_b1661617391115">obsfs</strong>: Parallel file system, which is mounted using obsfs (recommended).</li><li id="cce_bestpractice_00281__cce_10_0380_li1731525517197"><strong id="cce_bestpractice_00281__cce_10_0380_b38111554121115">s3fs</strong>: Object bucket, which is mounted using s3fs.</li></ul>
</td>
</tr>
<tr id="cce_bestpractice_00281__cce_10_0380_row11728154218251"><td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.1 "><p id="cce_bestpractice_00281__cce_10_0380_p16152185120254">everest.io/obs-volume-type</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.2 "><p id="cce_bestpractice_00281__cce_10_0380_p1515285182515">Yes</p>
</td>
<td class="cellrowborder" valign="top" headers="mcps1.3.3.5.3.2.5.1.3 "><p id="cce_bestpractice_00281__cce_10_0380_p17175165413349">OBS storage class.</p>
<ul id="cce_bestpractice_00281__cce_10_0380_ul10665173614443"><li id="cce_bestpractice_00281__cce_10_0380_li4665103664415">If <strong id="cce_bestpractice_00281__cce_10_0380_b984818231125">fsType</strong> is set to <strong id="cce_bestpractice_00281__cce_10_0380_b884872301216">s3fs</strong>, <strong id="cce_bestpractice_00281__cce_10_0380_b1184872321213">STANDARD</strong> (standard bucket) and <strong id="cce_bestpractice_00281__cce_10_0380_b5848142341213">WARM</strong> (infrequent access bucket) are supported.</li><li id="cce_bestpractice_00281__cce_10_0380_li1665113614444">This parameter is invalid when <strong id="cce_bestpractice_00281__cce_10_0380_b10146113910126">fsType</strong> is set to <strong id="cce_bestpractice_00281__cce_10_0380_b814673971219">obsfs</strong>.</li></ul>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<div class="section" id="cce_bestpractice_00281__section179154267162"><h4 class="sectiontitle">Custom Storage Classes</h4><p id="cce_bestpractice_00281__p9601335151617">You can customize a high I/O storage class in a YAML file. For example, the name <strong id="cce_bestpractice_00281__b186185811594">csi-disk-sas</strong> indicates that the disk type is SAS (high I/O).</p>
<pre class="screen" id="cce_bestpractice_00281__screen659210819391">apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-disk-sas # Name of the high I/O storage class, which can be customized.
parameters:
csi.storage.k8s.io/csi-driver-name: disk.csi.everest.io
csi.storage.k8s.io/fstype: ext4
everest.io/disk-volume-type: SAS # High I/O EVS disk type, which cannot be customized.
everest.io/passthrough: "true"
provisioner: everest-csi-provisioner
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true # <strong id="cce_bestpractice_00281__b148711949142017">true</strong> indicates that capacity expansion is allowed.</pre>
<p id="cce_bestpractice_00281__p1281618138471">For an ultra-high I/O storage class, you can set the class name to <strong id="cce_bestpractice_00281__b722754812">csi-disk-ssd</strong> to create SSD EVS disk (ultra-high I/O).</p>
<pre class="screen" id="cce_bestpractice_00281__screen199713432478">apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-disk-ssd # Name of the ultra-high I/O storage class, which can be customized.
parameters:
csi.storage.k8s.io/csi-driver-name: disk.csi.everest.io
csi.storage.k8s.io/fstype: ext4
everest.io/disk-volume-type: SSD # Ultra-high I/O EVS disk type, which cannot be customized.
everest.io/passthrough: "true"
provisioner: everest-csi-provisioner
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true</pre>
<p id="cce_bestpractice_00281__p146067122106"><strong id="cce_bestpractice_00281__b69321846286">reclaimPolicy</strong>: indicates the recycling policies of the underlying cloud storage. The value can be <strong id="cce_bestpractice_00281__b9392322125012">Delete</strong> or <strong id="cce_bestpractice_00281__b91651126105014">Retain</strong>.</p>
<ul id="cce_bestpractice_00281__ul42481619111116"><li id="cce_bestpractice_00281__li1624871911111"><strong id="cce_bestpractice_00281__b1313332805014">Delete</strong>: When a PVC is deleted, both the PV and the EVS disk are deleted.</li><li id="cce_bestpractice_00281__li9248141919118"><strong id="cce_bestpractice_00281__b26601433105212">Retain</strong>: When a PVC is deleted, the PV and underlying storage resources are not deleted. Instead, you must manually delete these resources. After that, the PV resource is in the <strong id="cce_bestpractice_00281__b1956334045518">Released</strong> state and cannot be bound to the PVC again.</li></ul>
<div class="note" id="cce_bestpractice_00281__note8801953114018"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="cce_bestpractice_00281__p178035314020">The reclamation policy configured here has no impact on the SFS Turbo storage.</p>
</div></div>
<p id="cce_bestpractice_00281__p778102518503">If high data security is required, you are advised to select <strong id="cce_bestpractice_00281__b1478374995218">Retain</strong> to prevent data from being deleted by mistake.</p>
<p id="cce_bestpractice_00281__p824713561207">After the definition is complete, run the <strong id="cce_bestpractice_00281__b14281158232">kubectl create</strong> commands to create storage resources.</p>
<pre class="screen" id="cce_bestpractice_00281__screen164494218115"># kubectl create -f sas.yaml
storageclass.storage.k8s.io/csi-disk-sas created
# kubectl create -f ssd.yaml
storageclass.storage.k8s.io/csi-disk-ssd created</pre>
<p id="cce_bestpractice_00281__p1412183715488">Query the storage class again. Two more types of storage classes are displayed in the command output, as shown below.</p>
<pre class="screen" id="cce_bestpractice_00281__screen13784148495"># kubectl get sc
NAME PROVISIONER AGE
csi-disk everest-csi-provisioner 17d
csi-disk-sas everest-csi-provisioner 2m28s
csi-disk-ssd everest-csi-provisioner 16s
csi-disk-topology everest-csi-provisioner 17d
csi-nas everest-csi-provisioner 17d
csi-obs everest-csi-provisioner 17d
csi-sfsturbo everest-csi-provisioner 17d</pre>
<p id="cce_bestpractice_00281__p05291082227">Other types of storage resources can be defined in the similar way. You can use kubectl to obtain the YAML file and modify it as required.</p>
<ul id="cce_bestpractice_00281__ul38311130152115"><li id="cce_bestpractice_00281__li18831113072113">File storage<pre class="screen" id="cce_bestpractice_00281__screen181646213295"># kubectl get sc csi-nas -oyaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: csi-nas
provisioner: everest-csi-provisioner
parameters:
csi.storage.k8s.io/csi-driver-name: nas.csi.everest.io
csi.storage.k8s.io/fstype: nfs
everest.io/share-access-level: rw
everest.io/share-access-to: 5e3864c6-e78d-4d00-b6fd-de09d432c632 # ID of the VPC to which the cluster belongs
everest.io/share-is-public: 'false'
everest.io/zone: xxxxx # AZ
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate</pre>
</li><li id="cce_bestpractice_00281__li4390135712118">Object storage<pre class="screen" id="cce_bestpractice_00281__screen1745353552911"># kubectl get sc csi-obs -oyaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: csi-obs
provisioner: everest-csi-provisioner
parameters:
csi.storage.k8s.io/csi-driver-name: obs.csi.everest.io
csi.storage.k8s.io/fstype: s3fs # Object storage type. <strong id="cce_bestpractice_00281__b581916217422">s3fs</strong> indicates an object bucket, and <strong id="cce_bestpractice_00281__b19130111019424">obsfs</strong> indicates a parallel file system.
everest.io/obs-volume-type: STANDARD # Storage class of the OBS bucket
reclaimPolicy: Delete
volumeBindingMode: Immediate</pre>
</li></ul>
</div>
<div class="section" id="cce_bestpractice_00281__section13586291218"><h4 class="sectiontitle">Specifying a Default Storage Class</h4><p id="cce_bestpractice_00281__p163571329132111">You can specify a storage class as the default class. In this way, if you do not specify <strong id="cce_bestpractice_00281__b133883591469">storageClassName</strong> when creating a PVC, the PVC is created using the default storage class.</p>
<p id="cce_bestpractice_00281__p1435710291217">For example, to specify <strong id="cce_bestpractice_00281__b1964515812715">csi-disk-ssd</strong> as the default storage class, edit your YAML file as follows:</p>
<pre class="screen" id="cce_bestpractice_00281__screen5357429172112">apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-disk-ssd
annotations:
storageclass.kubernetes.io/is-default-class: "true" # Specifies the default storage class in a cluster. A cluster can have only one default storage class.
parameters:
csi.storage.k8s.io/csi-driver-name: disk.csi.everest.io
csi.storage.k8s.io/fstype: ext4
everest.io/disk-volume-type: SSD
everest.io/passthrough: "true"
provisioner: everest-csi-provisioner
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true</pre>
<p id="cce_bestpractice_00281__p1135711298213">Delete the created csi-disk-ssd disk, run the <strong id="cce_bestpractice_00281__b152561717817">kubectl create</strong> command to create a csi-disk-ssd disk again, and then query the storage class. The following information is displayed.</p>
<pre class="screen" id="cce_bestpractice_00281__screen135872982120"># kubectl delete sc csi-disk-ssd
storageclass.storage.k8s.io "csi-disk-ssd" deleted
# kubectl create -f ssd.yaml
storageclass.storage.k8s.io/csi-disk-ssd created
# kubectl get sc
NAME PROVISIONER AGE
csi-disk everest-csi-provisioner 17d
csi-disk-sas everest-csi-provisioner 114m
csi-disk-ssd (default) everest-csi-provisioner 9s
csi-disk-topology everest-csi-provisioner 17d
csi-nas everest-csi-provisioner 17d
csi-obs everest-csi-provisioner 17d
csi-sfsturbo everest-csi-provisioner 17d</pre>
</div>
<div class="section" id="cce_bestpractice_00281__section113669814307"><h4 class="sectiontitle">Verification</h4><ul id="cce_bestpractice_00281__ul326513332433"><li id="cce_bestpractice_00281__li1426593312439">Use <strong id="cce_bestpractice_00281__b72318715919">csi-disk-sas</strong> to create a PVC.<pre class="screen" id="cce_bestpractice_00281__screen164121655153014">apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: sas-disk
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: csi-disk-sas</pre>
<p id="cce_bestpractice_00281__p1331523183215">Create a storage class and view its details. As shown below, the object can be created and the value of <strong id="cce_bestpractice_00281__b2841103141010">STORAGECLASS</strong> is <strong id="cce_bestpractice_00281__b1290115143103">csi-disk-sas</strong>.</p>
<pre class="screen" id="cce_bestpractice_00281__screen3819181515400"># kubectl create -f sas-disk.yaml
persistentvolumeclaim/sas-disk created
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
sas-disk Bound pvc-6e2f37f9-7346-4419-82f7-b42e79f7964c 10Gi RWO csi-disk-sas 24s
# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-6e2f37f9-7346-4419-82f7-b42e79f7964c 10Gi RWO Delete Bound default/sas-disk csi-disk-sas 30s</pre>
<p id="cce_bestpractice_00281__p296654417436">View the PVC details on the CCE console. On the PV details page, you can see that the disk type is high I/O.</p>
<p id="cce_bestpractice_00281__p1365215405213"></p>
</li><li id="cce_bestpractice_00281__li16722125154315">If <strong id="cce_bestpractice_00281__b53910164176">storageClassName</strong> is not specified, the default configuration is used, as shown below.<pre class="screen" id="cce_bestpractice_00281__screen623034510524">apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ssd-disk
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi</pre>
<p id="cce_bestpractice_00281__p14101131875619">Create and view the storage resource. You can see that the storage class of PVC ssd-disk is csi-disk-ssd, indicating that csi-disk-ssd is used by default.</p>
<pre class="screen" id="cce_bestpractice_00281__screen133543256569"># kubectl create -f ssd-disk.yaml
persistentvolumeclaim/ssd-disk created
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
sas-disk Bound pvc-6e2f37f9-7346-4419-82f7-b42e79f7964c 10Gi RWO csi-disk-sas 16m
ssd-disk Bound pvc-4d2b059c-0d6c-44af-9994-f74d01c78731 10Gi RWO csi-disk-ssd 10s
# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-4d2b059c-0d6c-44af-9994-f74d01c78731 10Gi RWO Delete Bound default/ssd-disk csi-disk-ssd 15s
pvc-6e2f37f9-7346-4419-82f7-b42e79f7964c 10Gi RWO Delete Bound default/sas-disk csi-disk-sas 17m</pre>
<p id="cce_bestpractice_00281__p14639104411583">View the PVC details on the CCE console. On the PV details page, you can see that the disk type is ultra-high I/O.</p>
<p id="cce_bestpractice_00281__p76651917155918"></p>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="cce_bestpractice_0053.html">Storage</a></div>
</div>
</div>
View Git Blame Copy Permalink