Page MenuHomePhabricator
Create Task
Maniphest T367514

archiva1002 has high disk usage
Closed, ResolvedPublic
Actions

Description

See https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=archiva1002&refresh=5m&from=now-30d&to=now, we have reached 95% usage of /var/lib/archiva.

ncdu does not show any huge offender, it's more a ||death by a thousand cuts|| kind of situation, where /var/lib/archiva/repositories contains many packages.

The host being a Ganeti VM, we could extend the disk, or adding a new, bigger one, and copying all the data to the new disk.

As the disk increase is very slow, we've muted the alert until Monday June 17th and we'll focus on it then,

Related Objects
Search...

Event Timeline

brouberol created this task.Jun 14 2024, 1:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 14 2024, 1:52 PM
brouberol added a comment.Jun 14 2024, 1:56 PM
--- /var/lib/archiva --------------------------
  118.6 GiB [##########] /repositories
   33.3 GiB [##        ] /data
    7.5 GiB [          ] /logs
  120.9 MiB [          ] /git-fat
   36.0 KiB [          ] /conf
e  16.0 KiB [          ] /lost+found
    8.0 KiB [          ] /groups
    4.0 KiB [          ]  derby.log
--- /var/lib/archiva/repositories --------------------
                         /..
   65.3 GiB [##########] /releases
   25.0 GiB [###       ] /mirror-maven-central
   22.5 GiB [###       ] /mirror-cloudera
    3.6 GiB [          ] /snapshots
    2.3 GiB [          ] /python
    2.5 MiB [          ] /analytics-old-uploads
    1.8 MiB [          ] /mirror-spark
   80.0
--- /var/lib/archiva/repositories/releases ----
                         /..
   55.9 GiB [######### ] /org
    7.0 GiB [#         ] /com
    2.0 GiB [          ] /.indexer
  128.5 MiB [          ] /ml
  104.8 MiB [          ] /article-recommender
  339.3 KiB [          ] /graphframes
  265.1 KiB [          ] /ua_parser
  240.1 KiB [          ] /commons-codec
  150.3 KiB [          ] /de
--- /var/lib/archiva/repositories/releases/org
                         /..
   32.5 GiB [##########] /wikidata
   23.2 GiB [#######   ] /wikimedia
  185.4 MiB [          ] /linkeddatafragments
   12.4 MiB [          ] /elasticsearch
    5.5 MiB [          ] /xbib
    5.5 MiB [          ] /apache
    1.0 MiB [          ] /codehaus
   38.2 KiB [          ] /slf4j
   12.3 KiB [          ] /jboss=
--- /var/lib/archiva/repositories/releases/org/wikidata/query/rdf -
                         /..
   15.9 GiB [##########] /service
    8.7 GiB [#####     ] /blazegraph-service
    2.0 GiB [#         ] /streaming-updater-producer
    1.6 GiB [#         ] /rdf-spark-tools
    1.3 GiB [          ] /tools
    1.3 GiB [          ] /streaming-updater-consumer
  887.7 MiB [          ] /mw-oauth-proxy
  325.9 MiB [          ] /streaming-updater
  194.7 MiB [          ] /jetty-logging
  152.7 MiB [          ] /flink-fs-swift
   65.1 MiB [          ] /blazegraph
   32.6 MiB [          ] /common
   11.2 MiB [          ] /testTools
    7.5 MiB [          ] /streaming-updater-common
    7.0 MiB [          ] /query-service-parent
   53.6 KiB [          ] /parent
BTullis claimed this task.Jun 17 2024, 9:20 AM
BTullis moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.05.27 - 2024.06.16) board.
BTullis triaged this task as High priority.Jun 17 2024, 11:04 AM
BTullis added a comment.EditedJun 17 2024, 12:29 PM

I have added 20 GB to the /dev/vdb device on ganeti1002 with:

btullis@ganeti1028:~$ sudo gnt-instance grow-disk --absolute archiva1002.wikimedia.org 1 220g
Mon Jun 17 11:56:30 2024 Growing disk 1 of instance 'archiva1002.wikimedia.org' by 20.0G to 220.0G
Mon Jun 17 12:20:46 2024  - INFO: Instance archiva1002.wikimedia.org's disks are in sync
btullis@ganeti1028:~$ sudo gnt-instance start archiva1002.wikimedia.org

I subsequently unmounted the /var/lib/archiva volume and updated the partition table:

btullis@archiva1002:~$ sudo fdisk /dev/vdb

Welcome to fdisk (util-linux 2.36.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/vdb: 220 GiB, 236223201280 bytes, 461373440 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x85fcb6c1

Device     Boot Start       End   Sectors  Size Id Type
/dev/vdb1        2048 419430399 419428352  200G 83 Linux

Command (m for help): d
Selected partition 1
Partition 1 has been deleted.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-461373439, default 2048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-461373439, default 461373439): 

Created a new partition 1 of type 'Linux' and of size 220 GiB.
Partition #1 contains a ext4 signature.

Do you want to remove the signature? [Y]es/[N]o: n

Command (m for help): p

Disk /dev/vdb: 220 GiB, 236223201280 bytes, 461373440 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x85fcb6c1

Device     Boot Start       End   Sectors  Size Id Type
/dev/vdb1        2048 461373439 461371392  220G 83 Linux

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

I checked the filesystem.

btullis@archiva1002:~$ sudo e2fsck -f /dev/vdb1
e2fsck 1.46.2 (28-Feb-2021)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/vdb1: 2485577/13107200 files (0.1% non-contiguous), 43069124/52428544 blocks

I then resized the filesystem.

btullis@archiva1002:~$ sudo resize2fs /dev/vdb1 
resize2fs 1.46.2 (28-Feb-2021)
Resizing the filesystem on /dev/vdb1 to 57671424 (4k) blocks.
The filesystem on /dev/vdb1 is now 57671424 (4k) blocks long.

Mounted the filesystem and checked the size

btullis@archiva1002:~$ sudo mount -a
btullis@archiva1002:~$ df -h /var/lib/archiva/
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb1       216G  161G   56G  75% /var/lib/archiva

Now showing 75% full.

BTullis closed this task as Resolved.Jun 17 2024, 12:35 PM
BTullis moved this task from In Progress to Done on the Data-Platform-SRE (2024.05.27 - 2024.06.16) board.
atsuko added a parent task: T367315: [Epic] Replace Archiva with Gitlab artifact repositories.May 1 2026, 11:43 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits