Page MenuHomePhabricator

ruthenium is going to run out of space on /srv and stop working
Closed, ResolvedPublic

Description

For some reason, /srv is not using /dev/mapper/ruthenium--vg-tank (was ruthenium reinstalled recently?).

Because of that, ruthenium is going to run out of space and soon fail all operations. We can mount that and move /srv there, but it needs to stop using it.

Two question: can we have downtime and do this? Why did this happen, was this incorrectly setup on reinstall or something?

https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&panelId=17&fullscreen&orgId=1&var-server=ruthenium&var-network=eth0&from=now-30d&to=now

Event Timeline

jcrespo created this task.May 10 2017, 8:46 AM

There is stuff on /dev/mapper/ruthenium--vg-tank (unmounted)

afwiki   cuwiki        enwikivoyage  fiwiki        hiwiki        itwikivoyage  kowiki      nlwiki   shwiki  viwiki
arwiki   cvwiki        enwiktionary  frwiki        huwiki        itwiktionary  lbewiki     nowiki   svwiki  warwiki
bnwiki   dewiki        eswiki        frwikisource  hywiki        jawiki        lnwiki      plwiki   thwiki  zhwiki
cawiki   elwiki        eswikisource  frwikivoyage  iswiki        kaawiki       lost+found  pnbwiki  trwiki  zh-yuewiki
cebwiki  enwiki        eswikivoyage  frwiktionary  itwiki        kawiki        mlwiki      ptwiki   ukwiki
ckbwiki  enwikisource  eswiktionary  hewiki        itwikisource  knwiki        mznwiki     ruwiki   uzwiki

This is what there is on /srv:

$ du --max-depth=1 -h
8.3G    ./log
119M    ./testreduce
2.8M    ./uprightdiff
30G     ./visualdiff
619M    ./deployment
39G     .

Waiting for @ssastry because this seems to be an application problem, (took 30 gb overnight), and fixing it without knowing why it is taking so much space could only make this worse.

/dev/mapper/ruthenium--vg-tank was supposed to be mounted at /srv/visualdiff/pngs ... i suppose on a recent reboot, it didn't get mounted .. i suppose it is missing some puppet code for that? But, I am okay with deleting everything in /srv/visualdiff/pngs and mounting the ruthenium--vg-tank volume there.

I will reset the test results in mysql and restart the tests. I have stopped testreduce-vd and testreduce-vd-client.

Mentioned in SAL (#wikimedia-operations) [2017-05-10T14:39:45Z] <jynus> disabling puppet to solve disk mount issues T164915

jcrespo closed this task as Resolved.May 10 2017, 2:52 PM
jcrespo claimed this task.

I have mounted /dev/mapper/ruthenium--vg-tank on /srv/visualdiff/pngs type ext4 (rw,relatime,data=ordered), deleted old /srv/visualdiff/pngs contents on the root partition, and added the entrie to fstab so it doesn't get lost on reboot. Whoever setup this server, forgot to configure the partitioning correctly.

Dzahn added a comment.May 10 2017, 4:44 PM

Thanks for this @jcrespo I mounted that on March 31st as a stop gap because it was running out of disk (didn't have logrotate afair T161920) and should have edited fstab.

(SAL entry was: 18:27 mutante: ruthenium mounting /dev/mapper/ruthenium--vg-tank into /srv/visualdiff/pngs | deleted "mysql" and "dumps" data that was on previously unmounted partition , subbu checked that wasn't needed anymore .."