Page MenuHomePhabricator
Create Task
Maniphest T375443

categories migration: determine size of new WDQS categories dump
Closed, ResolvedPublic
Actions

Description

Per T374016 and T374967 , we have to migrate wdqs-categories to its own infrastructure. To that end, we'll need to figure out our disk space needs.

The current WDQS categories journal file is ~36 GB:

bking@cumin2002:~$ sudo cumin A:wdqs-public 'du -hcxs /srv/wdqs/categories.jnl'
...
----- OUTPUT of 'du -hcxs /srv/wdqs/categories.jnl' -----
36G     /srv/wdqs/categories.jnl
36G     total

However, a newly-imported dump is likely to be smaller, as Blazegraph (similar to other DBs) doesn't fully reclaim space. Creating this ticket to:

  • Do a full data reload from categories using this cookbook
  • Record results

RESULTS Total size 27200978944 (human readable: 26G)

Related Objects
Search...

Event Timeline

bking created this task.Sep 23 2024, 9:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 23 2024, 9:10 PM
bking moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2024.09.06 - 2024.09.27) board.Sep 23 2024, 9:10 PM
Maintenance_bot added a project: Wikidata.Sep 23 2024, 9:29 PM
bking added a comment.EditedSep 23 2024, 10:03 PM

I had a problem starting the categories reload cookbook, as we've already made some changes as part of the process of separating categories from the rest of the WDQS stack (T374009 links some of the changes).

Before I could get the cookbook to run, I had to load the new scripts from this CR . The right way to do this is via a scap deploy, but I one-off copied the scripts to /srv/deployment/wdqs/wdqs on wdqs2020 and started the cookbook from wdqs2020. This appears to have worked, and the cookbook is still running as I write this...

RKemper updated the task description. (Show Details)Sep 24 2024, 12:45 AM
RKemper subscribed.

Results are in on wdqs2020:

Total size 27200978944 (human readable: 26G)

Gehel triaged this task as Medium priority.Sep 24 2024, 2:29 PM
bking added a parent task: T375520: EPIC: WDQS categories migration.Sep 25 2024, 7:25 PM
bking closed this task as Resolved.Sep 26 2024, 8:11 PM
bking claimed this task.
bking moved this task from In Progress to Done on the Data-Platform-SRE (2024.09.06 - 2024.09.27) board.

For a bit more context:

Size of dumps download: ~1.5 GB
Size of newly-created journal file: 26 GB
Time to create journal file from dumps: ~90 minutes

All of these figures will be important as we decide the next steps.

I've added this info to the WDQS categories migration Google Doc and now I'm closing out this ticket.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits