If we ever reach the storage limit of our persistent volume, we'll need to make sure we're able to resize it on the fly.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | brouberol | T362788 Migrate Airflow to the dse-k8s cluster | |||
| Resolved | BTullis | T364386 Validate postgres operator and Ceph integration | |||
| Resolved | brouberol | T372276 Validate that we can resize the ceph persistent volume |
Event Timeline
Comment Actions
root@deploy1003:/home/brouberol/cloudnative-pg# kubectl get pvc -n pgcluster-test NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pgcluster-example-1 Bound pvc-e5643a97-5a7d-455e-ba35-4f9a4ac98b95 8Gi RWO ceph-rbd-ssd 38m pgcluster-example-1-wal Bound pvc-a3abc301-976c-4539-9130-d6da8ab238dd 1Gi RWO ceph-rbd-ssd 38m pgcluster-example-2 Bound pvc-c2f7ab6c-51b7-4276-bd81-9b3e063b96a2 8Gi RWO ceph-rbd-ssd 37m pgcluster-example-2-wal Bound pvc-329d8497-c10e-4fb0-bbe5-60df79da7e82 1Gi RWO ceph-rbd-ssd 37m pgcluster-example-3 Bound pvc-460fdc26-8021-4ded-9976-b44aa0051f7d 8Gi RWO ceph-rbd-ssd 36m pgcluster-example-3-wal Bound pvc-ea822c9c-7fd2-4c60-8bcf-a1b0fe7a9323 1Gi RWO ceph-rbd-ssd 36m
We currently have a default of 8GB for data and 1GB for WALs. We can see that from within a pod:
postgres@pgcluster-example-2:/srv/app$ df -h | grep rbd /dev/rbd1 974M 113M 846M 12% /var/lib/postgresql/wal /dev/rbd0 7.8G 30M 7.8G 1% /var/lib/postgresql/data
Now, let's pretend we're closing in to the 8GB limit and would like an extra 5G of space. We edit the pgcluster-example-2 PVC and request 13Gi of space instead of 8Gi
root@deploy1003:/home/brouberol/cloudnative-pg# kubectl get pvc -owide -n pgcluster-test NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE pgcluster-example-1 Bound pvc-e5643a97-5a7d-455e-ba35-4f9a4ac98b95 8Gi RWO ceph-rbd-ssd 41m Filesystem pgcluster-example-1-wal Bound pvc-a3abc301-976c-4539-9130-d6da8ab238dd 1Gi RWO ceph-rbd-ssd 41m Filesystem pgcluster-example-2 Bound pvc-c2f7ab6c-51b7-4276-bd81-9b3e063b96a2 13Gi RWO ceph-rbd-ssd 40m Filesystem pgcluster-example-2-wal Bound pvc-329d8497-c10e-4fb0-bbe5-60df79da7e82 1Gi RWO ceph-rbd-ssd 40m Filesystem pgcluster-example-3 Bound pvc-460fdc26-8021-4ded-9976-b44aa0051f7d 8Gi RWO ceph-rbd-ssd 39m Filesystem pgcluster-example-3-wal Bound pvc-ea822c9c-7fd2-4c60-8bcf-a1b0fe7a9323 1Gi RWO ceph-rbd-ssd 39m Filesystem
And just like that, we can see (after a couple of seconds) that the container "sees" the extra space, without having to restart the pod
postgres@pgcluster-example-2:/srv/app$ df -h | grep rbd /dev/rbd1 974M 113M 846M 12% /var/lib/postgresql/wal /dev/rbd0 13G 30M 13G 1% /var/lib/postgresql/data root@deploy1003:/home/brouberol/cloudnative-pg# kubectl get pod -n pgcluster-test NAME READY STATUS RESTARTS AGE pgcluster-example-1 1/1 Running 0 42m pgcluster-example-2 1/1 Running 0 41m pgcluster-example-3 1/1 Running 0 40m pgcluster-example-ping-test-ndr9f 0/1 Completed 0 42m pgcluster-example-pooler-rw-76b879bff5-4rm28 1/1 Running 0 42m pgcluster-example-pooler-rw-76b879bff5-gfqcw 1/1 Running 0 42m pgcluster-example-pooler-rw-76b879bff5-s6rjw 1/1 Running 0 42m
If we wanted to do this in a cleaner way, we'd change the cluster.storage.size configuration value:
root@deploy1003:/home/brouberol/cloudnative-pg# cat pgcluster-values.yaml fullnameOverride: "pgcluster-example" cluster: replicas: 3 version: 15 storage: size: 15Gi pooler: enabled: true
Let's update it to 15Gi, as we can't downsize it.
brouberol@deploy1003:~/cloudnative-pg$ helm template pgcluster-example --set 'cluster.version=15' -f /srv/deployment-charts/charts/cloudnative-pg-cluster/values.yaml -f /etc/helmfile-defaults/general-dse-k8s-eqiad.yaml -f pgcluster-values.yaml -n pgcluster-test /srv/deployment-charts/charts/cloudnative-pg-cluster/ > rendered-pgcluster.yaml root@deploy1003:/home/brouberol/cloudnative-pg# kubectl apply -n pgcluster-test -f rendered-pgcluster.yaml networkpolicy.crd.projectcalico.org/pgcluster-example-to-kubapi unchanged networkpolicy.networking.k8s.io/pgcluster-example-ingress-pooler created cluster.postgresql.cnpg.io/pgcluster-example configured imagecatalog.postgresql.cnpg.io/pgcluster-example-catalog unchanged pooler.postgresql.cnpg.io/pgcluster-example-pooler-rw unchanged job.batch/pgcluster-example-ping-test unchanged
Note that while the change is automatic, it isn't instantaneous:
root@deploy1003:/home/brouberol/cloudnative-pg# kubectl get pvc -owide -n pgcluster-test NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE pgcluster-example-1 Bound pvc-e5643a97-5a7d-455e-ba35-4f9a4ac98b95 8Gi RWO ceph-rbd-ssd 45m Filesystem pgcluster-example-1-wal Bound pvc-a3abc301-976c-4539-9130-d6da8ab238dd 1Gi RWO ceph-rbd-ssd 45m Filesystem pgcluster-example-2 Bound pvc-c2f7ab6c-51b7-4276-bd81-9b3e063b96a2 15Gi RWO ceph-rbd-ssd 44m Filesystem pgcluster-example-2-wal Bound pvc-329d8497-c10e-4fb0-bbe5-60df79da7e82 1Gi RWO ceph-rbd-ssd 44m Filesystem pgcluster-example-3 Bound pvc-460fdc26-8021-4ded-9976-b44aa0051f7d 8Gi RWO ceph-rbd-ssd 43m Filesystem pgcluster-example-3-wal Bound pvc-ea822c9c-7fd2-4c60-8bcf-a1b0fe7a9323 1Gi RWO ceph-rbd-ssd 43m Filesystem ... root@deploy1003:/home/brouberol/cloudnative-pg# kubectl get pvc -owide -n pgcluster-test NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE pgcluster-example-1 Bound pvc-e5643a97-5a7d-455e-ba35-4f9a4ac98b95 15Gi RWO ceph-rbd-ssd 46m Filesystem pgcluster-example-1-wal Bound pvc-a3abc301-976c-4539-9130-d6da8ab238dd 1Gi RWO ceph-rbd-ssd 46m Filesystem pgcluster-example-2 Bound pvc-c2f7ab6c-51b7-4276-bd81-9b3e063b96a2 15Gi RWO ceph-rbd-ssd 45m Filesystem pgcluster-example-2-wal Bound pvc-329d8497-c10e-4fb0-bbe5-60df79da7e82 1Gi RWO ceph-rbd-ssd 45m Filesystem pgcluster-example-3 Bound pvc-460fdc26-8021-4ded-9976-b44aa0051f7d 15Gi RWO ceph-rbd-ssd 45m Filesystem pgcluster-example-3-wal Bound pvc-ea822c9c-7fd2-4c60-8bcf-a1b0fe7a9323 1Gi RWO ceph-rbd-ssd 45m Filesystem