Page MenuHomePhabricator
Create Task
Maniphest T301967

toolsdb: evaluate storage usage by some tools
Open, LowPublic
Actions

Description

I just noticed this:

aborrero@clouddb1001:~$ sudo du -cks /srv/labsdb/data/* | sort -rn | head
2566525640	total
1151204360	/srv/labsdb/data/s51230__linkwatcher
151231476	/srv/labsdb/data/s51434__mixnmatch_p
129605716	/srv/labsdb/data/s53220__quickstatements_p
107163660	/srv/labsdb/data/ibdata1
100784876	/srv/labsdb/data/s53685__editgroups
63725868	/srv/labsdb/data/s51114__enwp10
60054664	/srv/labsdb/data/s53952__freebase_p
50662520	/srv/labsdb/data/s51499__wikiminiatlas
50048072	/srv/labsdb/data/s51698__yetkin

Apparently we have a few heavy users:

  • s51230__linkwatcher is using 1.1TB of data, which is about 32% of the total storage available on toolsdb (3.4TB as I write this).

See also: https://tool-db-usage.toolforge.org/

image.png (496×741 px, 57 KB)

Figures doesn't match, but the first few positions are still occupied by the same tools.

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolveddcaroT301951 toolsdb: full disk on clouddb1001 broke clouddb1002 (secondary) replication
 OpenNoneT301967 toolsdb: evaluate storage usage by some tools
 OpenNoneT291782 Migrate largest ToolsDB users to Trove
 ResolvedAndrewT292546 cloud NFS: figure out backups for cinder volumes
 Resolved• aborreroT293752 cloud ceph: refactor rbd client puppet profiles
 DuplicateNoneT294429 cinder-backups: figure out automation
 Resolved• aborreroT295584 eqiad: 2 VMs for cloudbackup-dev
 Resolved• aborreroT296413 cinder: get victoria point release in the bpo repo
 Resolved• aborreroT299708 network access to eqiad ceph cluster from cloudbackup2002
 ResolvedAndrewT339830 cinder-backup getting OOM-killed for large volumes
 ResolvedAndrewT344065 Replace cinder-backup process with backy2
 ResolvedAndrewT358855 Use cloudbackup100[12]-dev for cinder backup test/dev
 ResolvedAndrewT366071 'backy2 cleanup' not getting called properly on cloudbackup hosts
 ResolvedAndrewT323502 Move some of magnus's tools to Trove databases (was: Request increased quota for mix-n-match Toolforge tool)
 ResolvedAndrewT324984 Trove volume size limit of 31Gb
 ResolvedfnegriT326754 Clarify Trove and Toolsdb usage within WMCS
 ResolvedTBurmeisterT326854 Publish revised Trove user guide
 ResolvedfnegriT328691 [toolsdb] Migrate linkwatcher db to Trove
 ResolvedTheresNoTimeT334491 Update Linkwatcher tool to use new Trove database
 ResolvedfnegriT328693 [toolsdb] Migrate "s54518__mw" db to Trove
 OpenNoneT350862 [toolsdb] Migrate mixnmatch db to Trove
 OpenNoneT369177 [toolsdb] Migrate quickstatements db to Trove

Event Timeline

• aborrero created this task.Feb 17 2022, 11:24 AM
• aborrero updated the task description. (Show Details)Feb 17 2022, 11:28 AM
• aborrero added a subtask: T224154: Reduce size of linkwatcher db if at all possible.Feb 17 2022, 11:43 AM
• aborrero reopened subtask T224154: Reduce size of linkwatcher db if at all possible as Open.Feb 17 2022, 11:51 AM
• aborrero mentioned this in T224154: Reduce size of linkwatcher db if at all possible.
RhinosF1 subscribed.Feb 17 2022, 4:30 PM
bd808 moved this task from Backlog to ToolsDB on the Data-Services board.Mar 17 2022, 10:04 PM
fnegri mentioned this in T301949: ToolsDB upgrade => Bullseye, MariaDB 10.4.Oct 27 2022, 6:58 PM
TheresNoTime removed a subscriber: RhinosF1.Dec 15 2022, 11:35 PM
fnegri added a subtask: T291782: Migrate largest ToolsDB users to Trove.Jan 10 2023, 5:34 PM
fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 6:39 PM
fnegri moved this task from Kanban to Inbox on the cloud-services-team board.
fnegri removed a subtask: T224154: Reduce size of linkwatcher db if at all possible.Apr 28 2023, 4:37 PM
fnegri triaged this task as Low priority.Jul 3 2024, 2:38 PM
fnegri subscribed.

DB storage is tracked in the subtask T291782: Migrate largest ToolsDB users to Trove

NFS storage does not seem to be an immediate issue, but we should probably check if there are tools/users using a disproportionate amount of space, and maybe create an alert for that.

taavi removed a project: Data-Services.Oct 21 2024, 2:41 PM
fnegri mentioned this in T291782: Migrate largest ToolsDB users to Trove.Feb 12 2025, 5:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits