Page MenuHomePhabricator

Get more disk space for InTense database
Open, LowestPublic

Description

We're quickly running out of disk space:

$ sudo du -shc /var/lib/mysql/
864M    /var/lib/mysql/
864M    total
$ df -h
Filesystem                                        Size  Used Avail Use% Mounted on
/dev/mapper/vd-var                                2.0G  1.2G  682M  63% /var

https://graphite.wmflabs.org/render/?title=ttmserver-mediawiki01+Disk+space+last+week&width=400&height=250&from=-1week&hideLegend=false&uniqueLegend=true&target=aliasByNode%28ttmserver.ttmserver-mediawiki01.diskspace.%2A.byte_avail.value%2C-3%2C-2%29

Proposed instances to spread the load and the 20 available cores: 8 for es, 1 for salt, 2-4 for mw+repolibrary, 2-4 for mysql. Need to verify if instances with 2-4 cores are big enough for the DB.

Event Timeline

Nemo_bis assigned this task to Nikerabbit.
Nemo_bis raised the priority of this task from to Medium.
Nemo_bis updated the task description. (Show Details)
Nemo_bis added a project: translatewiki.net.
Nemo_bis subscribed.

I don't really know how we could getting more disk space other than creating a new bigger instance or moving the files to some other location.

In theory an alternative is to change partitions, I guess. But yes, the only sensible thing is probably to create another instance for the DB.

Apply the role::labs::lvm::srv role and you shall magically get a lot of space in /srv

That was done (and /srv data recovered by Yuvi); when we have time, we "only" need to move the mysql data from /var, after checking the IO speed is reasonable.

Nemo_bis set Security to None.
Aklapper lowered the priority of this task from Medium to Lowest.Nov 27 2019, 1:18 PM

Lowering priority to reflect the reality that currently nobody is working on InTense (plus its test website is down).