Page MenuHomePhabricator

Bump disk space on testreduce1001
Closed, ResolvedPublic


df on testreduce1001 shows:

/dev/vda1       36893152 34766668    222716 100% /

Can the disk space be bumped to at least 50G? This gives us a bit more room in keeping around results of older test runs (it would be nice to keep around test results from the last 3 months) .

Event Timeline

Disk space can't be simply bumped up, but what we can offer is to create a new virtual hard disk and mount it into the file system.

Then you can use the new disk for mysql or whatever you like. But it doesn't involve resizing the file system or moving data around.

Would a second disk with 50G fresh GB, mounted on /srv/data or something work for you?

[ganeti1011:~] $ sudo gnt-instance modify --disk add:size=50G testreduce1001.eqiad.wmnet
Tue Mar 16 19:15:57 2021  - INFO: Waiting for instance testreduce1001.eqiad.wmnet to sync disks
Tue Mar 16 19:15:57 2021  - INFO: - device disk/1:  0.10% done, 2h 48m 1s remaining (estimated)

On it.. ^ but will take a while. I will create a filesystem on it and mount it once done. Will update the ticket.

@ssastry The disk has been created but for the VM to detect it we have to reboot the VM just like if it was a physical machine. Can I do that at any time or do you have some tests running right now?

mysql is running a recovery after a previous crash .. so, let us wait for it to complete before restarting.

I think something is broken with mysql on testreduce1001 at this point .. so, maybe just reboot the server and see if that fixes anything. If not, we can just wipe the db and recreate it. We'll just rerun a couple of tests from a known good version and take it from there.

So, yes, you can reboot the VM at your earliest convenience.

Mentioned in SAL (#wikimedia-operations) [2021-03-17T14:18:07Z] <jayme> rebooting restreduce1001 for T277580

Mentioned in SAL (#wikimedia-operations) [2021-03-17T19:05:54Z] <mutante> ganeti1011 - rebooting VM testreduce1001 on ganeti level for T277580

<jayme> rebooting restreduce1001 for T277580

The reboot needs to happen on Ganeti level, not from inside the VM. But once I did that the VM was not reachable via SSH anymore. Same thing happened that happened before, the NIC got renumbered and is suddenly "ens6" instead of "ens5" after adding the new disk.

So had to login via console, and replace the interface name in /etc/network/interfaces and reboot again and VM is back.

We now have a new /dev/vdb available.

so.. fdisk /dev/vdb to make a new partition table, then a primary partition.

Now we have /dev/vdb1 and can mkfs.ext4 /dev/vdb1 to put a filesystem on it.

Then mkdir /srv/data and mount /dev/vdb1 /srv/data and finally:

blkid /dev/vdb1 to get UUID and edit /etc/fstab like:

UUID=995df86e-4aca-45d2-80e4-c31bc84b8f69 /srv/data ext4 errors=remount-ro   0   1

to make it survive a reboot.

Dzahn claimed this task.

Rebooted a final time to confirm it stays mounted:

[testreduce1001:~] $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        36G   32G  1.8G  95% /
/dev/vdb1        49G   53M   47G   1% /srv/data

Here you go!

Change 673175 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] parsoid::testreduce: switch mysql data dir to /srv/data/mysql

Change 673175 merged by Dzahn:
[operations/puppet@production] parsoid::testreduce: switch mysql data dir to /srv/data/mysql

Mentioned in SAL (#wikimedia-operations) [2021-03-19T00:45:02Z] <mutante> testreduce1001 - stop mysql; rsyncing /var/lib/mysql to /srv/data/mysql (T277580)

Mentioned in SAL (#wikimedia-operations) [2021-03-19T19:50:47Z] <mutante> testreduce1001 - confirmed MariaDB @@datadir is /srv/data/mysql and deleting /var/lib/mysql (T277580)

rm -rf /var/lib/mysql/

root@testreduce1001:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        36G  4.0G   30G  12% /
/dev/vdb1        49G   29G   19G  61% /srv/data