For some reason, /srv is not using /dev/mapper/ruthenium--vg-tank (was ruthenium reinstalled recently?).
Because of that, ruthenium is going to run out of space and soon fail all operations. We can mount that and move /srv there, but it needs to stop using it.
Two question: can we have downtime and do this? Why did this happen, was this incorrectly setup on reinstall or something?
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Invalid | None | T161920 logrotate for visualdiff tests on Parsoid test host (scandium) | |||
| Resolved | jcrespo | T164915 ruthenium is going to run out of space on /srv and stop working |
There is stuff on /dev/mapper/ruthenium--vg-tank (unmounted)
afwiki cuwiki enwikivoyage fiwiki hiwiki itwikivoyage kowiki nlwiki shwiki viwiki arwiki cvwiki enwiktionary frwiki huwiki itwiktionary lbewiki nowiki svwiki warwiki bnwiki dewiki eswiki frwikisource hywiki jawiki lnwiki plwiki thwiki zhwiki cawiki elwiki eswikisource frwikivoyage iswiki kaawiki lost+found pnbwiki trwiki zh-yuewiki cebwiki enwiki eswikivoyage frwiktionary itwiki kawiki mlwiki ptwiki ukwiki ckbwiki enwikisource eswiktionary hewiki itwikisource knwiki mznwiki ruwiki uzwiki
This is what there is on /srv:
$ du --max-depth=1 -h 8.3G ./log 119M ./testreduce 2.8M ./uprightdiff 30G ./visualdiff 619M ./deployment 39G .
Waiting for @ssastry because this seems to be an application problem, (took 30 gb overnight), and fixing it without knowing why it is taking so much space could only make this worse.
/dev/mapper/ruthenium--vg-tank was supposed to be mounted at /srv/visualdiff/pngs ... i suppose on a recent reboot, it didn't get mounted .. i suppose it is missing some puppet code for that? But, I am okay with deleting everything in /srv/visualdiff/pngs and mounting the ruthenium--vg-tank volume there.
I will reset the test results in mysql and restart the tests. I have stopped testreduce-vd and testreduce-vd-client.
Mentioned in SAL (#wikimedia-operations) [2017-05-10T14:39:45Z] <jynus> disabling puppet to solve disk mount issues T164915
I have mounted /dev/mapper/ruthenium--vg-tank on /srv/visualdiff/pngs type ext4 (rw,relatime,data=ordered), deleted old /srv/visualdiff/pngs contents on the root partition, and added the entrie to fstab so it doesn't get lost on reboot. Whoever setup this server, forgot to configure the partitioning correctly.
Thanks for this @jcrespo I mounted that on March 31st as a stop gap because it was running out of disk (didn't have logrotate afair T161920) and should have edited fstab.
(SAL entry was: 18:27 mutante: ruthenium mounting /dev/mapper/ruthenium--vg-tank into /srv/visualdiff/pngs | deleted "mysql" and "dumps" data that was on previously unmounted partition , subbu checked that wasn't needed anymore .."