deployment-deploy03 /srv at 100% usage
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	TheresNoTime
	Jul 31 2023, 10:40 AM

Description

[samtar@deployment-deploy03 ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           799M   81M  718M  11% /run
/dev/sda1        20G  5.1G   14G  28% /
tmpfs           3.9G  4.0K  3.9G   1% /dev/shm
tmpfs           5.0M   20K  5.0M   1% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sdb         40G   38G     0 100% /srv
tmpfs           799M     0  799M   0% /run/user/0
tmpfs           799M     0  799M   0% /run/user/13367
tmpfs           799M     0  799M   0% /run/user/2947
tmpfs           799M     0  799M   0% /run/user/11634
tmpfs           799M     0  799M   0% /run/user/37672
tmpfs           799M     0  799M   0% /run/user/4816
tmpfs           799M     0  799M   0% /run/user/12744

[samtar@deployment-deploy03 ~]$ sudo ncdu /srv/

--- /srv -----------------------------------------------------------------------------------------------------------------
   13.4 GiB [##########] /deployment-charts
   12.3 GiB [######### ] /mediawiki-staging
   11.4 GiB [########  ] /deployment
  189.6 MiB [          ] /jenkins
   33.8 MiB [          ] /packages
  132.0 KiB [          ] /patches
   24.0 KiB [          ] /mwpresync
   20.0 KiB [          ] /mwbuilder
e  16.0 KiB [          ] /lost+found
   12.0 KiB [          ] /home
e   4.0 KiB [          ] /jenkins-workspace
@   0.0   B [          ]  mediawiki

Event Timeline

TheresNoTime created this task.Jul 31 2023, 10:40 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 31 2023, 10:40 AM

TheresNoTime added a subscriber: jnuche.Jul 31 2023, 10:41 AM

Deleted /srv/deployment-charts to free some space. .git dir there had grown to 14G.

The repo wasn't puppetized as far as I could see and hadn't gotten any new commits from the remote since May. Git metada seemed corrupt and the repo couldn't be gc'd or fetch new changes.

I checked with @Clement_Goubert and @Joe but none of us knew where the local checkout of the repo originated from. We considered it safe to delete.

After deletion:

jnuche@deployment-deploy03:/srv$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3,9G     0  3,9G   0% /dev
tmpfs           799M   81M  718M  11% /run
/dev/sda1        20G  5,1G   14G  28% /
tmpfs           3,9G  4,0K  3,9G   1% /dev/shm
tmpfs           5,0M   20K  5,0M   1% /run/lock
tmpfs           3,9G     0  3,9G   0% /sys/fs/cgroup
/dev/sdb         40G   24G   14G  65% /srv
tmpfs           799M     0  799M   0% /run/user/0
tmpfs           799M     0  799M   0% /run/user/13367
tmpfs           799M     0  799M   0% /run/user/2947
tmpfs           799M     0  799M   0% /run/user/11634
tmpfs           799M     0  799M   0% /run/user/37672
tmpfs           799M     0  799M   0% /run/user/4816
tmpfs           799M     0  799M   0% /run/user/12744

jnuche closed this task as Resolved.Jul 31 2023, 10:47 AM

jnuche claimed this task.

Mentioned in SAL (#wikimedia-releng) [2023-07-31T10:53:03Z] <jnuche> reenabled beta-code-update-eqiad job after freeing up some disk space. See: https://phabricator.wikimedia.org/T343114

jnuche added a project: Release-Engineering-Team (Escape Goats🐐).Jul 31 2023, 10:57 AM

jnuche moved this task from Backlog to Done on the Release-Engineering-Team (Escape Goats🐐) board.Jul 31 2023, 11:02 AM

deployment-deploy03 /srv at 100% usageClosed, ResolvedPublicActions

Description

Event Timeline

deployment-deploy03 /srv at 100% usage
Closed, ResolvedPublic
Actions