| Michael | |
| Jul 27 2021, 4:18 PM |
| F34567948: image.png | |
| Jul 28 2021, 2:02 PM |
| F34567943: image.png | |
| Jul 28 2021, 1:58 PM |
The 20GB disk space of the wbregistry-01 instance of wikibase-registry has fully filled up. The vast majority is located in /var/lib/docker, followed by /var/cache/apt/ which makes up 10% (~2GB).
The container running:
there are several seemingly duplicate container.
It seems to have been so out of disk space, that it even couldn't send error emails :/
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T287492 wbregistry-01 is out of disk space | |||
| Resolved | Lucas_Werkmeister_WMDE | T287586 wbregistry-01 docker-compose.yml is not under source control | |||
| Open | None | T287587 wikibase registry didn't send an email when it ran out of disk space | |||
| Open | None | T287588 Figure out if wikibase-registry should move to wbstack or maybe use k8s natively |
Mentioned in SAL (#wikimedia-cloud) [2021-07-27T16:22:02Z] <wm-bot> sudo docker kill 36a5b061acfe stopping duplicate old docker container T287492
Mentioned in SAL (#wikimedia-cloud) [2021-07-27T16:22:37Z] <wm-bot> sudo docker kill 9e0309ff1120 stopping duplicate old docker container T287492
Mentioned in SAL (#wikimedia-cloud) [2021-07-27T16:22:43Z] <wm-bot> sudo docker kill dcd0b3d98c51 stopping duplicate old docker container T287492
Mentioned in SAL (#wikimedia-cloud) [2021-07-27T16:23:15Z] <wm-bot> sudo docker container prune removing stopped containers to free up space T287492
Mentioned in SAL (#wikimedia-cloud) [2021-07-27T16:23:23Z] <wm-bot> sudo docker image prune removing dangling docker images to free up space T287492
Was it that it couldn't send emails because it was out of space, or out of space because of all the emails it couldn't send? It has been reported that there was a general problem with email delivery for some time.
Apparently, most of the disk usage was concentrated in the docker directory, due to old jobs somehow dangling there. We will look into this more. We will also check for the emails and try to find a solution where this does not happen again.
For now, we're back to 11% (~2GB) free space.
Mentioned in SAL (#wikimedia-cloud) [2021-07-28T13:50:28Z] <wm-bot> sudo docker kill wikibase-registry_wikibase-update_run_1 kill container not in /root/wikibase-registry/docker-compose.yml - no idea where that came from. T287492
Mentioned in SAL (#wikimedia-cloud) [2021-07-28T13:53:04Z] <wm-bot> sudo docker container prune remove dangling container meta data. T287492
Mentioned in SAL (#wikimedia-cloud) [2021-07-28T13:53:32Z] <wm-bot> sudo docker image prune remove dangling images. Freed up 117 MB. T287492
After deleting the duplicate container, taking /root/wikibase-registry/docker-compose.yml as the source of truth for what container are supposed to run and running sudo docker container prune and sudo docker image prune, we're back to 11% free space:
This isn't great yet. Follow-up work will happen in sub-tasks.
Mentioned in SAL (#wikimedia-cloud) [2021-07-28T14:57:13Z] <wm-bot> sudo docker image prune --all removing all unused docker images. Freed up 5.14 GB. T287492