Page MenuHomePhabricator

mwdebug2001 and mwdebug2002 "/" almost full
Closed, ResolvedPublic

Description

mwdebug2001 is almost full:

root@mwdebug2001:/srv/mediawiki# df -hT /
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/vda1      ext4   39G   35G  2.0G  95% /
root@mwdebug2001:/srv# du -sh *
27G	mediawiki

Related Objects

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 3 2019, 1:50 PM

This is related to T218783

jcrespo renamed this task from mwdebug2001 "/" almost full to mwdebug2001 and mwdebug2002 "/" almost full.Apr 3 2019, 3:54 PM
fgiunchedi triaged this task as Normal priority.Apr 9 2019, 8:37 AM
Marostegui raised the priority of this task from Normal to Unbreak Now!.Apr 10 2019, 5:02 AM

Both mwdebug2001 and 2002 are now full:
T218783#5099940

Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptApr 10 2019, 5:02 AM
greg added a comment.Apr 10 2019, 5:23 AM

Since these are vms can a quick fix be to expand their disk? Since they're the oddballs of the mw fleet in that way. I know Tyler wants to fix the scap clean issue this week. (Sorry, my comment on the other task should have been here.)

Joe added a subscriber: Joe.Apr 10 2019, 5:23 AM

Also the last mediawiki train didn't deploy correctly to those servers and will never be able to unless we remove old versions.

Joe added a comment.Apr 10 2019, 5:38 AM

Since these are vms can a quick fix be to expand their disk? Since they're the oddballs of the mw fleet in that way. I know Tyler wants to fix the scap clean issue this week. (Sorry, my comment on the other task should have been here.)

It's not a quick fix expanding the disk of a ganeti VM requires quite some time, in particular in this case given it will imply a reimage.

Also I can't remove old php versions because any scap pull or scap sync will bring us back in this situation.

I'm opting for putting both servers out of the scap sync until the parent bug is resolved.

Mentioned in SAL (#wikimedia-operations) [2019-04-10T05:52:11Z] <_joe_> setting both mwdebug200{1,2} to pooled = inactive to remove them from scap dsh list and allow deployments, T219989

Joe lowered the priority of this task from Unbreak Now! to High.Apr 10 2019, 5:52 AM
fsero claimed this task.Apr 10 2019, 10:21 AM

Mentioned in SAL (#wikimedia-operations) [2019-04-10T10:25:26Z] <fsero> resizing disk on mwdebug2001 T219989

Mentioned in SAL (#wikimedia-operations) [2019-04-10T13:47:25Z] <fsero> resizing disk on mwdebug2002 T219989

Mentioned in SAL (#wikimedia-operations) [2019-04-10T15:01:05Z] <fsero> pooled back mwdebug200[1,2] T219989

fsero closed this task as Resolved.Apr 10 2019, 3:02 PM

mwdebug2001,2 disk has been increased and VMs reimaged and pooled back, so this should be good to go now.

While doing this i faced the same issue stated here https://phabricator.wikimedia.org/T178457 and this should be added to the puppet code