mwdebug2001 is almost full:
root@mwdebug2001:/srv/mediawiki# df -hT / Filesystem Type Size Used Avail Use% Mounted on /dev/vda1 ext4 39G 35G 2.0G 95% /
root@mwdebug2001:/srv# du -sh * 27G mediawiki
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • fsero | T219989 mwdebug2001 and mwdebug2002 "/" almost full | |||
| Open | None | T218783 `scap clean` failure: SSH Too many authentication failures: 7 |
Event Timeline
Comment Actions
Since these are vms can a quick fix be to expand their disk? Since they're the oddballs of the mw fleet in that way. I know Tyler wants to fix the scap clean issue this week. (Sorry, my comment on the other task should have been here.)
Comment Actions
Also the last mediawiki train didn't deploy correctly to those servers and will never be able to unless we remove old versions.
Comment Actions
It's not a quick fix expanding the disk of a ganeti VM requires quite some time, in particular in this case given it will imply a reimage.
Also I can't remove old php versions because any scap pull or scap sync will bring us back in this situation.
I'm opting for putting both servers out of the scap sync until the parent bug is resolved.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2019-04-10T05:52:11Z] <_joe_> setting both mwdebug200{1,2} to pooled = inactive to remove them from scap dsh list and allow deployments, T219989
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2019-04-10T10:25:26Z] <fsero> resizing disk on mwdebug2001 T219989
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2019-04-10T13:47:25Z] <fsero> resizing disk on mwdebug2002 T219989
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2019-04-10T15:01:05Z] <fsero> pooled back mwdebug200[1,2] T219989
Comment Actions
mwdebug2001,2 disk has been increased and VMs reimaged and pooled back, so this should be good to go now.
While doing this i faced the same issue stated here https://phabricator.wikimedia.org/T178457 and this should be added to the puppet code