Page MenuHomePhabricator

deployment-tin has disk space issues
Closed, ResolvedPublic

Description

Monitoring alerts about deployment-tin disk space:

$ df -h / /srv
Filesystem                          Size  Used Avail Use% Mounted on
/dev/vda3                            19G   14G  4.0G  78% /
/dev/mapper/vd-second--local--disk   40G   29G  9.6G  75% /srv

Event Timeline

Mentioned in SAL (#wikimedia-releng) [2017-05-29T09:51:44Z] <hashar> deployment-tin: rm /var/lib/l10nupdate/caches/cache-master/*.json T166492

@thcipriani you have clones of mediawiki repositories on deployment-tin in /home/thcipriani/mwclonetest. I guess that can be safely deleted.

Mentioned in SAL (#wikimedia-releng) [2017-05-29T10:06:40Z] <hashar> deployment-tin rm -fR /usr/src/hhvm T166492

@thcipriani you have clones of mediawiki repositories on deployment-tin in /home/thcipriani/mwclonetest. I guess that can be safely deleted.

Removed. That should save 1.8GB.

/ looks better now, thank you!

$ df -h / /srv
Filesystem                          Size  Used Avail Use% Mounted on
/dev/vda3                            19G   12G  6.3G  65% /
/dev/mapper/vd-second--local--disk   40G   29G  9.4G  76% /srv

Next candidate is /srv/mediawiki/.git which is 8 GBytes. I am not sure why the .git directory ends up being synced to the deployment target. Most of that is taken by a 7.0G pack file:

$ ls -lh /srv/mediawiki/.git/objects/pack/pack-09e0a9019c69f9efcfda2463a1a582a6e7a454b8.pack
                               vvvv
-r--r--r-- 1 mwdeploy mwdeploy 7.0G
                               ^^^^
May 29 13:24 /srv/mediawiki/.git/objects/pack/pack-09e0a9019c69f9efcfda2463a1a582a6e7a454b8.pack

Spoke with Tyler about the 7.0G, that is a git repository that contains all the other repos. Meant for scap3 usage.

Lets get an instance with more disk.

hashar changed the task status from Stalled to Open.Jun 13 2017, 7:49 AM

m1.large would do it.

greg lowered the priority of this task from Medium to Low.Sep 18 2017, 4:57 PM
greg added a subscriber: greg.

Hit this last week as well, just deleted /srv/mw/.git

Not a dire need, but we should rebuild -tin with a larger disk in the not too distant future.

  • created instance deployment-deploy-01 with stretch and flavor x-large (T192561)
  • deployment-deploy1001 has been deleted by thcipriani
Dzahn claimed this task.

deployment-tin does not exist anymore and deployment-deploy01 and 02 have 60GB disk space each.