Disk space can't be simply bumped up, but what we can offer is to create a new virtual hard disk and mount it into the file system.

Then you can use the new disk for mysql or whatever you like. But it doesn't involve resizing the file system or moving data around.

Would a second disk with 50G fresh GB, mounted on /srv/data or something work for you?

Sure.

And, thanks! :)

[ganeti1011:~] $ sudo gnt-instance modify --disk add:size=50G testreduce1001.eqiad.wmnet
Tue Mar 16 19:15:57 2021  - INFO: Waiting for instance testreduce1001.eqiad.wmnet to sync disks
Tue Mar 16 19:15:57 2021  - INFO: - device disk/1:  0.10% done, 2h 48m 1s remaining (estimated)

On it.. ^ but will take a while. I will create a filesystem on it and mount it once done. Will update the ticket.

@ssastry The disk has been created but for the VM to detect it we have to reboot the VM just like if it was a physical machine. Can I do that at any time or do you have some tests running right now?

mysql is running a recovery after a previous crash .. so, let us wait for it to complete before restarting.

I think something is broken with mysql on testreduce1001 at this point .. so, maybe just reboot the server and see if that fixes anything. If not, we can just wipe the db and recreate it. We'll just rerun a couple of tests from a known good version and take it from there.

So, yes, you can reboot the VM at your earliest convenience.

Mentioned in SAL (#wikimedia-operations) [2021-03-17T14:18:07Z] <jayme> rebooting restreduce1001 for T277580

Mentioned in SAL (#wikimedia-operations) [2021-03-17T19:05:54Z] <mutante> ganeti1011 - rebooting VM testreduce1001 on ganeti level for T277580

In T277580#6921570, @Stashbot wrote:

<jayme> rebooting restreduce1001 for T277580

The reboot needs to happen on Ganeti level, not from inside the VM. But once I did that the VM was not reachable via SSH anymore. Same thing happened that happened before, the NIC got renumbered and is suddenly "ens6" instead of "ens5" after adding the new disk.

So had to login via console, and replace the interface name in /etc/network/interfaces and reboot again and VM is back.

We now have a new /dev/vdb available.

so.. fdisk /dev/vdb to make a new partition table, then a primary partition.

Now we have /dev/vdb1 and can mkfs.ext4 /dev/vdb1 to put a filesystem on it.

Then mkdir /srv/data and mount /dev/vdb1 /srv/data and finally:

blkid /dev/vdb1 to get UUID and edit /etc/fstab like:

UUID=995df86e-4aca-45d2-80e4-c31bc84b8f69 /srv/data ext4 errors=remount-ro   0   1

to make it survive a reboot.

Change 673175 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] parsoid::testreduce: switch mysql data dir to /srv/data/mysql

https://gerrit.wikimedia.org/r/673175

Change 673175 merged by Dzahn:
[operations/puppet@production] parsoid::testreduce: switch mysql data dir to /srv/data/mysql

https://gerrit.wikimedia.org/r/673175

Mentioned in SAL (#wikimedia-operations) [2021-03-19T00:45:02Z] <mutante> testreduce1001 - stop mysql; rsyncing /var/lib/mysql to /srv/data/mysql (T277580)

Mentioned in SAL (#wikimedia-operations) [2021-03-19T19:50:47Z] <mutante> testreduce1001 - confirmed MariaDB @@datadir is /srv/data/mysql and deleting /var/lib/mysql (T277580)