deployment-tin has disk space issues
Open, LowPublic

Description

Monitoring alerts about deployment-tin disk space:

$ df -h / /srv
Filesystem                          Size  Used Avail Use% Mounted on
/dev/vda3                            19G   14G  4.0G  78% /
/dev/mapper/vd-second--local--disk   40G   29G  9.6G  75% /srv
hashar created this task.May 29 2017, 9:50 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 29 2017, 9:50 AM

Mentioned in SAL (#wikimedia-releng) [2017-05-29T09:51:44Z] <hashar> deployment-tin: rm /var/lib/l10nupdate/caches/cache-master/*.json T166492

@thcipriani you have clones of mediawiki repositories on deployment-tin in /home/thcipriani/mwclonetest. I guess that can be safely deleted.

Mentioned in SAL (#wikimedia-releng) [2017-05-29T10:06:40Z] <hashar> deployment-tin rm -fR /usr/src/hhvm T166492

@thcipriani you have clones of mediawiki repositories on deployment-tin in /home/thcipriani/mwclonetest. I guess that can be safely deleted.

Removed. That should save 1.8GB.

/ looks better now, thank you!

$ df -h / /srv
Filesystem                          Size  Used Avail Use% Mounted on
/dev/vda3                            19G   12G  6.3G  65% /
/dev/mapper/vd-second--local--disk   40G   29G  9.4G  76% /srv

Next candidate is /srv/mediawiki/.git which is 8 GBytes. I am not sure why the .git directory ends up being synced to the deployment target. Most of that is taken by a 7.0G pack file:

$ ls -lh /srv/mediawiki/.git/objects/pack/pack-09e0a9019c69f9efcfda2463a1a582a6e7a454b8.pack
                               vvvv
-r--r--r-- 1 mwdeploy mwdeploy 7.0G
                               ^^^^
May 29 13:24 /srv/mediawiki/.git/objects/pack/pack-09e0a9019c69f9efcfda2463a1a582a6e7a454b8.pack

Spoke with Tyler about the 7.0G, that is a git repository that contains all the other repos. Meant for scap3 usage.

Lets get an instance with more disk.

hashar changed the task status from Open to Stalled.

We would need an OpenStack flavor with a larger disk. Filled as T167723: Create a new instance flavor for deployment-prep

hashar changed the task status from Stalled to Open.

m1.large would do it.

greg changed the status of subtask T167723: Create a new instance flavor for deployment-prep from Resolved to Invalid.Jun 13 2017, 6:45 PM
hashar triaged this task as Normal priority.Jun 16 2017, 11:00 AM
greg lowered the priority of this task from Normal to Low.Sep 18 2017, 4:57 PM
greg added a subscriber: greg.

Hit this last week as well, just deleted /srv/mw/.git

Not a dire need, but we should rebuild -tin with a larger disk in the not too distant future.