| BTullis | |
| Sep 26 2022, 8:41 AM |
| F35534427: image.png | |
| Sep 26 2022, 8:41 AM |
We currently have an incident affecting matomo1002 and therefore putting piwik.wikimedia.org at risk.
A recent addition to Matomo in T315613 has caused usage to increase rapidly, which has caused an increase in the size of the MariaDB data directory: /var/lib/mysql
The simplest solution would be to add another virtual disk to matomo1002 of sufficient size and move /var/lib/mysql to this new disk.
I am adding an 80 GB virtual disk to matomo1002 with the following command:
btullis@ganeti1027:~$ sudo gnt-instance modify --disk add:size=80G matomo1002.eqiad.wmnet
The logs indicate that the drive is being pre-allocated now.
Mon Sep 26 08:56:04 2022 - INFO: Waiting for instance matomo1002.eqiad.wmnet to sync disks Mon Sep 26 08:56:04 2022 - INFO: - device disk/1: 0.10% done, 56m 21s remaining (estimated)
I will need to reboot matomo1002 once this is done, to pick up the new disk.
I have created and formatted a single partition on the new 80 GB disk.
btullis@matomo1002:~$ sudo fdisk /dev/vdb
Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xde70dc4e.
Command (m for help): p
Disk /dev/vdb: 80 GiB, 85899345920 bytes, 167772160 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xde70dc4e
Command (m for help): n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p):
Using default response p.
Partition number (1-4, default 1):
First sector (2048-167772159, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-167772159, default 167772159):
Created a new partition 1 of type 'Linux' and of size 80 GiB.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
btullis@matomo1002:~$ sudo mke2fs -t ext4 -m 0.5 /dev/vdb1
mke2fs 1.44.5 (15-Dec-2018)
Creating filesystem with 20971264 4k blocks and 5242880 inodes
Filesystem UUID: d1fae7f8-a9db-432b-ba52-c502bc5080c0
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000
Allocating group tables: done
Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: doneNow proceeding to mount this to a temporary location and carry out an initial sync, so that I can quickly swap the mount points over.
I've now performed a full sync of this data and switched over to use it.
btullis@matomo1002:/var/lib$ df -h Filesystem Size Used Avail Use% Mounted on udev 3.9G 0 3.9G 0% /dev tmpfs 798M 8.5M 790M 2% /run /dev/vda1 42G 12G 28G 30% / tmpfs 3.9G 4.0K 3.9G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup tmpfs 1.0G 0 1.0G 0% /var/lib/nginx tmpfs 798M 0 798M 0% /run/user/32741 /dev/vdb1 79G 24G 55G 30% /var/lib/mysql