| BTullis | |
| Wed, Dec 11, 10:41 AM |
| F57798665: image.png | |
| Wed, Dec 11, 10:41 AM |
We are running close to capacity on /var/lib/archiva on archiva1002 and the trend is still for the data size to increase.
We are planning to deprecate the Archiva service, but until then we need to find a way to support the current data sizes.
Restarting the archiva service removes some files that are in use, but these are re-created soon afterwards, as the graph shows.
It is a virtual server, but it does not currently use LVM. The current partition scheme has /var/lib/archiva on /dev/vdb1.
archiva1002 is a VM running under Ganeti.
btullis@ganeti1028:~$ sudo gnt-instance list archiva1002.wikimedia.org Instance Hypervisor OS Primary_node Status Memory archiva1002.wikimedia.org kvm debootstrap+default ganeti1035.eqiad.wmnet running 4.0G
It is hosted by ganeti1035.
btullis@ganeti1028:~$ sudo gnt-node list ganeti1035.eqiad.wmnet Node DTotal DFree MTotal MNode MFree Pinst Sinst ganeti1035.eqiad.wmnet 2.5T 2.0T 62.4G 14.6G 44.8G 3 2
That host has 2.5 TB of free disk.
The secondary node of archiva1002 is ganeti1023.
btullis@ganeti1028:~$ sudo gnt-instance list -o snodes archiva1002.wikimedia.org Secondary_Nodes ganeti1023.eqiad.wmnet
This host has 2.0 TB free.
btullis@ganeti1028:~$ sudo gnt-node list ganeti1023.eqiad.wmnet Node DTotal DFree MTotal MNode MFree Pinst Sinst ganeti1023.eqiad.wmnet 2.5T 2.0T 62.5G 17.4G 43.2G 4 2
It would seem that I have a few options.
Looking back at previous tasks, I see that we bought ourselves some headroom by deleting old artifacts in March 2022 in T304224: Archiva's disk partiton space is getting filled up
About 4 months later, I added the second disk in T313386: archiva1002 is running low on space left in the root partition
I'm going to add a new 300 GB disk, then copy the files and remove the old 200 GB one.
Following the commands from T313386#8099952, but this time, I will make it an LVS PV, so that I can extend it in future, should the need arise.
btullis@ganeti1028:~$ sudo gnt-instance modify --disk add:size=300g archiva1002.wikimedia.org
That completed successfully.
Wed Dec 11 14:07:39 2024 - INFO: Instance archiva1002.wikimedia.org's disks are in sync Modified instance archiva1002.wikimedia.org - disk/2 -> add:size=307200,mode=rw Please don't forget that most parameters take effect only at the next (re)start of the instance initiated by ganeti; restarting from within the instance will not be enough.
Icinga downtime and Alertmanager silence (ID=6e480811-2ad5-4916-8363-12ee14122326) set by btullis@cumin1002 for 1:00:00 on 1 host(s) and their services with reason: Adding new disk
archiva1002.wikimedia.org
The new disk has been added. I had to log in over the ganeti console and edit /etc/network/interfaces as per https://wikitech.wikimedia.org/wiki/Ganeti#Adding_a_disk but that's done now.
btullis@archiva1002:~$ sudo lshw -class disk
*-cdrom UNCLAIMED
description: SCSI CD-ROM
product: QEMU DVD-ROM
vendor: QEMU
physical id: 0.0.0
bus info: scsi@1:0.0.0
version: 2.5+
capabilities: removable
configuration: ansiversion=5
*-virtio1
description: Virtual I/O device
physical id: 0
bus info: virtio@1
logical name: /dev/vda
size: 220GiB (236GB)
capabilities: partitioned partitioned:dos
configuration: driver=virtio_blk logicalsectorsize=512 sectorsize=512 signature=df19b68e
*-virtio2
description: Virtual I/O device
physical id: 0
bus info: virtio@2
logical name: /dev/vdb
size: 220GiB (236GB)
capabilities: partitioned partitioned:dos
configuration: driver=virtio_blk logicalsectorsize=512 sectorsize=512 signature=85fcb6c1
*-virtio3
description: Virtual I/O device
physical id: 0
bus info: virtio@3
logical name: /dev/vdc
size: 300GiB (322GB)
configuration: driver=virtio_blk logicalsectorsize=512 sectorsize=512
btullis@archiva1002:~$Manually adding the lvm2 package.
btullis@archiva1002:~$ sudo apt install lvm2
Creating an LVM physical volume from /dev/vdc:
btullis@archiva1002:~$ sudo pvcreate /dev/vdc Physical volume "/dev/vdc" successfully created.
Creating a new LVM volume group:
btullis@archiva1002:~$ sudo vgcreate vg00 /dev/vdc Volume group "vg00" successfully created
Creating a new LVM volume with 100% of the available space:
btullis@archiva1002:~$ sudo lvcreate -n lv-archiva -l 100%FREE vg00 Logical volume "lv-archiva" created.
Making a new ext4 filesystem on this volume.
btullis@archiva1002:~$ sudo mke2fs -t ext4 -m 0 -L archiva /dev/vg00/lv-archiva mke2fs 1.46.2 (28-Feb-2021) Creating filesystem with 78642176 4k blocks and 19660800 inodes Filesystem UUID: fcaf78a4-51db-4070-9bd4-a028af74c362 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872, 71663616 Allocating group tables: done Writing inode tables: done Creating journal (262144 blocks): done Writing superblocks and filesystem accounting information: done
Mounting it to /mnt temporarily.
btullis@archiva1002:~$ sudo mount /dev/vg00/lv-archiva /mnt
I will now start an rsync operation in a screen session, copying from /var/lib/archiva to /mnt. This will copy the bulk of it, then I can do a final pass when I switch it over, to make sure that we are consistent.
Icinga downtime and Alertmanager silence (ID=71aebe12-454c-47a6-b562-8b5cbfd71c92) set by btullis@cumin1002 for 1:00:00 on 1 host(s) and their services with reason: Adding new disk
archiva1002.wikimedia.org
I have synced all of the contents and mounted the new 300 GB drive to /var/lib/archiva
Removed the 200 GB drive.
btullis@ganeti1028:~$ sudo gnt-instance shutdown archiva1002.wikimedia.org Waiting for job 2747554 for archiva1002.wikimedia.org ... btullis@ganeti1028:~$ sudo gnt-instance modify --disk 1:remove archiva1002.wikimedia.org Modified instance archiva1002.wikimedia.org - disk/1 -> remove Please don't forget that most parameters take effect only at the next (re)start of the instance initiated by ganeti; restarting from within the instance will not be enough. btullis@ganeti1028:~$ sudo gnt-instance start archiva1002.wikimedia.org Waiting for job 2747556 for archiva1002.wikimedia.org ...
This now looks better.
btullis@archiva1002:~$ df -h -t ext4 Filesystem Size Used Avail Use% Mounted on /dev/vda1 94G 40G 51G 44% / /dev/mapper/vg00-lv--archiva 295G 179G 117G 61% /var/lib/archiva