Page MenuHomePhabricator

deployment-tin has disk space issues
Closed, ResolvedPublic

Description

Monitoring alerts about deployment-tin disk space:

$ df -h / /srv
Filesystem                          Size  Used Avail Use% Mounted on
/dev/vda3                            19G   14G  4.0G  78% /
/dev/mapper/vd-second--local--disk   40G   29G  9.6G  75% /srv

Event Timeline

hashar created this task.May 29 2017, 9:50 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 29 2017, 9:50 AM

Mentioned in SAL (#wikimedia-releng) [2017-05-29T09:51:44Z] <hashar> deployment-tin: rm /var/lib/l10nupdate/caches/cache-master/*.json T166492

@thcipriani you have clones of mediawiki repositories on deployment-tin in /home/thcipriani/mwclonetest. I guess that can be safely deleted.

Mentioned in SAL (#wikimedia-releng) [2017-05-29T10:06:40Z] <hashar> deployment-tin rm -fR /usr/src/hhvm T166492

@thcipriani you have clones of mediawiki repositories on deployment-tin in /home/thcipriani/mwclonetest. I guess that can be safely deleted.

Removed. That should save 1.8GB.

/ looks better now, thank you!

$ df -h / /srv
Filesystem                          Size  Used Avail Use% Mounted on
/dev/vda3                            19G   12G  6.3G  65% /
/dev/mapper/vd-second--local--disk   40G   29G  9.4G  76% /srv

Next candidate is /srv/mediawiki/.git which is 8 GBytes. I am not sure why the .git directory ends up being synced to the deployment target. Most of that is taken by a 7.0G pack file:

$ ls -lh /srv/mediawiki/.git/objects/pack/pack-09e0a9019c69f9efcfda2463a1a582a6e7a454b8.pack
                               vvvv
-r--r--r-- 1 mwdeploy mwdeploy 7.0G
                               ^^^^
May 29 13:24 /srv/mediawiki/.git/objects/pack/pack-09e0a9019c69f9efcfda2463a1a582a6e7a454b8.pack

Spoke with Tyler about the 7.0G, that is a git repository that contains all the other repos. Meant for scap3 usage.

Lets get an instance with more disk.

hashar changed the task status from Open to Stalled.Jun 12 2017, 7:49 PM

We would need an OpenStack flavor with a larger disk. Filled as T167723: Create a new instance flavor for deployment-prep

hashar changed the task status from Stalled to Open.Jun 13 2017, 7:49 AM

m1.large would do it.

greg changed the status of subtask T167723: Create a new instance flavor for deployment-prep from Resolved to Invalid.Jun 13 2017, 6:45 PM
hashar triaged this task as Normal priority.Jun 16 2017, 11:00 AM
greg lowered the priority of this task from Normal to Low.Sep 18 2017, 4:57 PM
greg added a subscriber: greg.

Hit this last week as well, just deleted /srv/mw/.git

Not a dire need, but we should rebuild -tin with a larger disk in the not too distant future.

Dzahn added a subscriber: Dzahn.May 30 2018, 6:36 PM
  • created instance deployment-deploy-01 with stretch and flavor x-large (T192561)
  • deployment-deploy1001 has been deleted by thcipriani
Dzahn closed this task as Resolved.Jul 9 2019, 9:09 PM
Dzahn claimed this task.

deployment-tin does not exist anymore and deployment-deploy01 and 02 have 60GB disk space each.

Dzahn reassigned this task from Dzahn to thcipriani.Jul 9 2019, 9:10 PM