| Raymond_Ndibe | |
| Oct 7 2024, 10:43 PM |
| F65618547: image.png | |
| Jul 23 2025, 8:26 AM |
| F57595919: Screenshot 2024-10-07 at 11.38.14 PM.png | |
| Oct 7 2024, 10:43 PM |
the object storage container harborstorage in the diagram below is empty (all files and folders have been deleted), but the space they occupy have not been freed (size is still listed as 8GB even when the container is empty)
After a brief discussion and debugging, David suggested that this might be because the garbage collection process in radosgw might not be working as expected
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | Raymond_Ndibe | T336668 [harbor] Create backups and/or replication | |||
| Open | Raymond_Ndibe | T356301 [harbor] Deploy with Helm | |||
| Resolved | Raymond_Ndibe | T350687 [harbor] Move harbor data to object storage service | |||
| Resolved | dcaro | T376673 [openstack object storage] deleted files still occupying space | |||
| Resolved | dcaro | T306820 [ceph] Upgrade to v16 | |||
| Resolved | dcaro | T309789 [ceph] Upgrade hosts to bullseye | |||
| Resolved | dcaro | T369026 [reimage,ceph] reimaging cloudcephosd hosts gets stuck in network configuration screen | |||
| Resolved | Andrew | T383817 partman vs cloudcephosd1012 | |||
| Open | dcaro | T396940 cloudcephosd1xxxx.private.eqiad.wikimedia.cloud | |||
| Open | dcaro | T398389 ceph-mon 16.2.15+ds-0+deb12u1 uses all the RAM |
This might be due to the rados gw nodes (cloudcontrols) running ceph v16, while the manager nodes (cloudcephmon) run ceph v15, so the garbar collecting process (that runs on the mrg) fails to list the newer buckets, similar upstream bug:
https://github.com/rook/rook/issues/10538#issuecomment-1397793407
For reference, some commands on how to get info from ceph and openstack:
# on the openstack side, you have containers and objects root@cloudcontrol2005-dev:~# OS_PROJECT_ID=testlabs openstack container show harbor_tests ... root@cloudcontrol2005-dev:~# OS_PROJECT_ID=testlabs openstack object list harbor_tests
# Note that this runs also on the cloudcontrol, as the cloudcephmons have the old version and will fail to get info of new buckets # for radosgw-admin, we have buckets (matches the containers) and the objects (matches the objects in openstack) root@cloudcontrol1005:~# radosgw-admin bucket list --bucket toolsbeta/harborstorage ... long json root@cloudcontrol1005:~# radosgw-admin bucket stats | jq '.[] | select(.bucket == "harborstorage")' ... another json
The space has been freed :), the radosgw-admin command work on the mons again, so this can be closed.
