OpenSearch on K8s is backed by Ceph RBD storage.
As currently configured, there is replication both at the application layer (OpenSearch shards) and at the storage layer (Ceph PGs). This leads to unnecessary disk usage and degraded performance.
Creating this ticket to:
Re: emptyDir (local storage)
Most dse-k8s hosts have 500 GB disk (ref: sudo cumin A:dse-k8s-worker 'pvs')
pvs (12) dse-k8s-worker[1005-1011,1015-1019].eqiad.wmnet ----- OUTPUT of 'pvs' ----- PV VG Fmt Attr PSize PFree /dev/md0 vg0 lvm2 a-- 446.72g 89.35g lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert containerd vg0 -wi-ao---- <205.55g kubelet vg0 -wi-ao---- <123.89g root vg0 -wi-ao---- <27.94g
/var/lib/kubelet (where emptyDir storage "lives" on the k8s worker by default) is mostly empty:
122G 444M 115G 1% /var/lib/kubelet
/var/lib/containerd is mostly empty as well:
202G 43G 149G 23% /var/lib/containerd
You can also check the amount of available ephemeral storage on a worker node via kubectl describe node dse-k8s-worker1007.eqiad.wmnet from the deployment server if you have root access:
Allocatable:
ephemeral-storage: 117299717950117299717950 works out to ~109 GB, roughly matching what's shown by /var/lib/kubelet above.