Manually expanding VMware VMDK Persistent Volume in OpenShift

You may have application running inside OpenShift Container Platform, and that uses a Persistent Volume provisioned from vSphere VMDK. One day, when you find the disk becoming full and nothing could be cleaned up, the problem arise. How to directly expand the volume without losing any data?

Suppose you have a Deployment called MyApp , with PVC MyApp-pvc and PV MyApp-pv , here is how to expand the storage.

Step 1) Stop the application

Stop the application by scaling down the deployment to 0 replicas:

oc scale deploy MyApp --replicas=0

Step 2) Identify the VMDK file of the Persistent Volume

Use below command to get the name of the VMDK disk including its path.

oc get pv MyApp-pv -o jsonpath='{.spec.vsphereVolume.volumePath}' && echo

Step 3) Expand the vSphere VMDK disk

You need to find a Linux VM as helper machine to expand the VMDK disk.

From vCenter web console, attach the VMDK disk to the Linux helper machine.

Verify the VMDK disk has been attached. In below example, it is sdb .

[root@helper ~]# lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda             8:0    0   200G  0 disk
├─sda1          8:1    0   600M  0 part /boot/efi
├─sda2          8:2    0     1G  0 part /boot
└─sda3          8:3    0 198.4G  0 part
  ├─rhel-root 253:0    0    50G  0 lvm  /
  ├─rhel-swap 253:1    0   7.9G  0 lvm  [SWAP]
  └─rhel-home 253:2    0 140.5G  0 lvm  /home
sdb             8:16   0    50G  0 disk

Go back to vCenter console again, change the size of the VMDK disk to the new desired size. For example, changing it from 50G to 200G.

In the helper machine, rescan the disk and verify that it has been expanded.

[root@helper ~]# echo 1>/sys/class/block/sdb/device/rescan
[root@helper ~]# lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda             8:0    0   200G  0 disk
├─sda1          8:1    0   600M  0 part /boot/efi
├─sda2          8:2    0     1G  0 part /boot
└─sda3          8:3    0 198.4G  0 part
  ├─rhel-root 253:0    0    50G  0 lvm  /
  ├─rhel-swap 253:1    0   7.9G  0 lvm  [SWAP]
  └─rhel-home 253:2    0 140.5G  0 lvm  /home
sdb             8:16   0   200G  0 disk

Step 4) Expand the filesystem

Mount the disk on the helper machine, check the current filesystem size and verify the contents accordingly.

[root@helper ~]# mkdir /mnt/datadisk
[root@helper ~]# mount -t ext4 /dev/sdb /mnt/datadisk
[root@helper ~]# cd /mnt/datadisk
[root@helper ~]# df -Th
Filesystem  Type  Size  Used  Avail Use% Mounted on
...
/dev/sdb    ext4  49G    49G      0 100% /mnt/datadisk
[root@helper ~]# ls -l
...

Expand the filesystem.

[root@helper ~]# resize2fs /dev/sdb

Verify that the filesystem has been expanded.

[root@helper ~]# df -Th
Filesystem  Type   Size  Used  Avail Use% Mounted on
...
/dev/sdb    ext4   199G   49G   150G  25% /mnt/datadisk

Umount the disks from the helper machine.

[root@helper ~]# umount /mnt/datadisk

Step 5) Detach the vSphere VMDK disk from the helper machine

You can now detach the VMDK disk from your helper machine via vCenter web console.

Step 6) Update the PV configuration

Check if the PV’s Reclaim Policy is Retain. If not, modify it accordingly.

$ oc get pv MyApp-pv -o jsonpath='{.spec.persistentVolumeReclaimPolicy}' && echo
Delete
$ oc patch pv MyApp-pv --patch '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' --type=merge
persistentvolume/MyApp-pv patched
$ oc get pv MyApp-pv -o jsonpath='{.spec.persistentVolumeReclaimPolicy}' && echo
Retain

Modify the capacity of the PV to match with the new disk size.

$ oc patch pv MyApp-pv --patch '{"spec":{"capacity":{"storage":"200Gi"}}}' --type=merge
persistentvolume/MyApp-pv patched
$ oc get pv MyApp-pv -o jsonpath='{.spec.capacity.storage}' && echo
200Gi

Step 7) Recreate the PVC

Dump the current PVC config.

oc get pvc MyApp-pvc -o yaml > MyApp-pvc.yaml

Edit the YAML file of the PVC with the new storage size.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume
  finalizers:
  - kubernetes.io/pvc-protection
  name: MyApp-pvc
  namespace: MySpace
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi
  storageClassName: vsphere
  volumeMode: Filesystem
  volumeName: MyApp-pv

Delete the current PVC and create it again using the YAML file.

oc delete pvc MyApp-pvc
oc create -f MyApp-pvc.yaml

Check the status of the PVC, you will find that it could not be bound.

$ oc get pvc -n openshift-monitoring
NAME        STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
MyApp-pvc   Lost     MyApp-pv   0                         vsphere        56s

To fix this, patch the PVs by setting the claimRef to null .

oc patch pv MyApp-pv -p '{"spec":{"claimRef": null}}'

Verify the status of the PVC again, it should now be bound.

$ oc get pvc -n openshift-monitoring
NAME        STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
MyApp-pvc   Bound    MyApp-pv   200G                      vsphere        3m

Step 8) Resume the application

Resume the application by scaling up the deployment back to 1 replicas:

oc scale deploy MyApp --replicas=1

The application is now resumed with all existing data persisted, and has a larger disk to contain more new data.