PersistentVolumes (PVs)

Notes and Constraints

• On the new CCE console (the cluster needs to be upgraded to v1.19.10 or later and the everest add-on needs to be upgraded to v1.2.10 or later), PVs are open to you for management. On the old CCE console, PVs can only be imported or dynamically created. You cannot manage the PV lifecycle on the console.
• Multiple PVs can use the same SFS or SFS Turbo file system with the following restrictions:
  • An error may occur if multiple PVCs/PVs that use the same underlying SFS or SFS Turbo file system are mounted to the same pod.
  • The persistentVolumeReclaimPolicy parameter in the PVs must be set to Retain. Otherwise, when a PV is deleted, the associated underlying volume may be deleted. In this case, other PVs associated with the underlying volume may be abnormal.
  • When the underlying volume is repeatedly used, it is recommended that ReadWriteMany be implemented at the application layer to prevent data overwriting and loss.

Volume Access Modes

PVs can be mounted to the host system only in the mode supported by underlying storage resources. For example, a file storage system can be read and written by multiple nodes, but an EVS disk can be read and written by only one node.

• ReadWriteOnce: A volume can be mounted as read-write by a single node. This access mode is supported by EVS.
• ReadWriteMany: A volume can be mounted as read-write by multiple nodes. This access mode is supported by SFS, SFS Turbo, and OBS.

PV Reclaim Policy

A PV reclaim policy is used to delete or reclaim underlying volumes when a PVC is deleted. The value can be Delete or Retain.

• Delete: When a PVC is deleted, the PV and underlying storage resources are deleted.
• Retain: When a PVC is deleted, the PV and underlying storage resources are not deleted. Instead, you must manually delete these resources. After a PVC is deleted, the PV resource is in the Released state and cannot be bound to the PVC again.

Everest also allows you to delete a PVC without deleting underlying storage resources. This function can be achieved only by using a YAML file. Set the PV reclaim policy to Delete and add annotations"everest.io/reclaim-policy: retain-volume-only". In this way, when the PVC is deleted, the PV resource is deleted, but the underlying storage resources are retained.

Creating an EVS Volume

The requirements for creating an EVS volume are as follows:

• System disks, DSS disks, and shared disks cannot be used.
• The EVS disk is one of the supported types (common I/O, high I/O, and ultra-high I/O), and the EVS disk device type is SCSI.
• The EVS disk is not frozen or used, and the status is available.
• If the EVS disk is encrypted, the key must be available.

Using the CCE Console

1. Log in to the CCE console.
2. Access the cluster details page, choose Storage from the navigation pane, and click the Volumes tab.
3. Click Create Volume in the upper right corner. In the dialog box displayed, set the volume parameters.
   • Volume Type: Select EVS.
   • EVS:
   • PV Name: Enter a PV name.
   • Access Mode: ReadWriteOnce
   • Reclaim Policy: Select Delete or Retain as required. For details, see PV Reclaim Policy.

4. Click Create.

Using YAML

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: everest-csi-provisioner
    everest.io/reclaim-policy: retain-volume-only         # (Optional) The PV is deleted while the underlying volume is retained.
  name: cce-evs-test
  labels:
    failure-domain.beta.kubernetes.io/region: eu-de
    failure-domain.beta.kubernetes.io/zone: eu-de-01
spec:
  accessModes:
    - ReadWriteOnce     # Access mode. The value is fixed to ReadWriteOnce for EVS.
  capacity:
    storage: 10Gi       #  EVS disk capacity, in the unit of Gi. The value ranges from 1 to 32768.
  csi:
    driver: disk.csi.everest.io     # Dependent storage driver for the mounting.
    fsType: ext4
    volumeHandle: 459581af-e78c-4356-9e78-eaf9cd8525eb   # Volume ID of the EVS disk.
    volumeAttributes:
      everest.io/disk-mode: SCSI           # Device type of the EVS disk. Only SCSI is supported.
      everest.io/disk-volume-type: SAS     # EVS disk type.
      storage.kubernetes.io/csiProvisionerIdentity: everest-csi-provisioner
      everest.io/crypt-key-id: 0992dbda-6340-470e-a74e-4f0db288ed82    # (Optional) Encryption key ID. Mandatory for an encrypted disk.
  persistentVolumeReclaimPolicy: Delete    # Reclain policy.
  storageClassName: csi-disk               # Storage class name. The value must be csi-disk.

Creating an SFS Volume

• The SFS file system and the cluster must be in the same VPC.

Using YAML

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: everest-csi-provisioner
    everest.io/reclaim-policy: retain-volume-only      # (Optional) The PV is deleted while the underlying volume is retained.
  name: cce-sfs-test
spec:
  accessModes:
  - ReadWriteMany      # Access mode. The value must be ReadWriteMany for SFS.
  capacity:
    storage: 1Gi       # File storage capacity.
  csi:
    driver: disk.csi.everest.io   # Mount the dependent storage driver.
    fsType: nfs
    volumeHandle: 30b3d92a-0bc7-4610-b484-534660db81be   # SFS file system ID.
    volumeAttributes:
      everest.io/share-export-location:   # Shared path of the file storage
      storage.kubernetes.io/csiProvisionerIdentity: everest-csi-provisioner
  persistentVolumeReclaimPolicy: Retain    # Reclaim policy.
  storageClassName: csi-nas                # Storage class name. The value must be csi-nas for SFS.
  mountOptions: []                         # Mount options

mountOptions:
- vers=3
- timeo=600
- nolock
- hard

Creating an OBS Volume

Secure containers do not support OBS volumes.

A single user can create a maximum of 100 OBS buckets on the console. If you have a large number of CCE workloads and you want to mount an OBS bucket to every workload, you may easily run out of buckets. In this scenario, you are advised to use OBS through the OBS API or SDK and do not mount OBS buckets to the workload on the console.

Using the CCE Console

1. Log in to the CCE console.
2. Access the cluster details page, choose Storage from the navigation pane, and click the Volumes tab.
3. Click Create Volume in the upper right corner. In the dialog box displayed, set the volume parameters.
   • Volume Type: Select OBS.
   • Select OBS resources.
   • PV Name: Enter a PV name.
   • Access Mode: ReadWriteMany
   • Reclaim Policy: Select Delete or Retain as required. For details, see PV Reclaim Policy.
   • Key: You can customize the access key (AK/SK) for mounting an OBS volume. You can use the AK/SK to create a secret and mount the secret to the PV. For details, see Using a Custom AK/SK to Mount an OBS Volume.
   • Mount Options: mount options. For details about the options, see Setting Mount Options.

4. Click Create.

Using YAML

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: everest-csi-provisioner
    everest.io/reclaim-policy: retain-volume-only         # (Optional) The PV is deleted while the underlying volume is retained.
  name: cce-obs-test
spec:
  accessModes:
  - ReadWriteMany                      # Access mode. The value must be ReadWriteMany for OBS.
  capacity:
    storage: 1Gi      # Storage capacity. This parameter is set only to meet the PV format requirements. It can be set to any value. The actual OBS space size is not limited by this value.
  csi:
    driver: obs.csi.everest.io        # Dependent storage driver for the mounting.
    fsType: obsfs                      # OBS file type.
    volumeHandle: cce-obs-bucket       # OBS bucket name.
    volumeAttributes:
      everest.io/obs-volume-type: STANDARD
      everest.io/region: eu-de
      
      storage.kubernetes.io/csiProvisionerIdentity: everest-csi-provisioner
    nodePublishSecretRef:
      name: test-user
      namespace: default
  persistentVolumeReclaimPolicy: Retain       # Reclaim policy.
  storageClassName: csi-obs                   # Storage class name. The value must be csi-obs for OBS.
  mountOptions: []                            # Mount options.

Creating an SFS Turbo Volume

SFS Turbo and the cluster must be in the same VPC.

Using the CCE Console

1. Log in to the CCE console.
2. Access the cluster details page, choose Storage from the navigation pane, and click the Volumes tab.
3. Click Create Volume in the upper right corner. In the dialog box displayed, set the volume parameters.
   • Volume Type: Select SFS Turbo.
   • SFS Turbo: Select SFS Turbo resources.
   • PV Name: Enter a PV name.
   • Access Mode: ReadWriteMany
   • Reclaim Policy: Select Retain. For details, see PV Reclaim Policy.
   • Mount Options: mount options. For details about the options, see Setting Mount Options.

4. Click Create.

Using YAML

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: everest-csi-provisioner
  name: cce-sfsturbo-test
spec:
  accessModes:
    - ReadWriteMany       # Access mode. The value must be ReadWriteMany for SFS Turbo.
  capacity:
    storage: 100.00Gi     # SFS Turbo volume capacity.
  csi:
    driver: sfsturbo.csi.everest.io    # Dependent storage driver for the mounting.
    fsType: nfs
    volumeHandle: 6674bd0a-d760-49de-bb9e-805c7883f047      # SFS Turbo volume ID.
    volumeAttributes:
      everest.io/share-export-location: 192.168.0.85:/      # Shared path of the SFS Turbo volume.
      storage.kubernetes.io/csiProvisionerIdentity: everest-csi-provisioner
  persistentVolumeReclaimPolicy: Retain     # Reclaim policy.
  storageClassName: csi-sfsturbo            # Storage class name. The value must be csi-sfsturbo for SFS Turbo.
  mountOptions: []                          # Mount options.

mountOptions:
- vers=3
- timeo=600
- nolock
- hard