Page MenuHomePhabricator
Create Task
Maniphest T375687

Test categories performance under Ganeti
Closed, ResolvedPublic
Actions

Description

Per pairing with @dcausse this morning, we think it would be wise to look into Ganeti as a potential option for the categories migration.

Creating this ticket to:

  • Investigate current wdqs-categories resource usage
  • (separate ticket) Assuming the resources are within Ganeti's capabilities, provision a VM with the appropriate resources
  • (separate ticket) Work through the Puppet code until we are confident we can deploy categories independently from the rest of the wdqs stack. Even if we don't end up migrating to Ganeti, this will force us to clean up and logically separate the Puppet code, which is an operational win in itself.

Details

SubjectRepoBranchLines +/-
wdqs.categories-reload: don't check hostoperations/cookbooksmaster+1 -2
wdqs-categories: use correct insetup roleoperations/puppetproduction+1 -1
wdqs-categories: introduce VM for testingoperations/puppetproduction+15 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

bking created this task.Wed, Sep 25, 7:43 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Sep 25, 7:43 PM
bking changed the task status from Open to In Progress.Wed, Sep 25, 7:43 PM
bking triaged this task as Medium priority.
bking edited projects, added Data-Platform-SRE (2024.09.06 - 2024.09.27); removed Data-Platform-SRE.
Maintenance_bot added a project: Wikidata.Wed, Sep 25, 8:30 PM
bking added a parent task: T375520: EPIC: WDQS categories migration.Thu, Sep 26, 8:14 PM
bking added a comment.Thu, Sep 26, 10:21 PM

I have a few panels at the bottom of the WDQS dashboard which should help us estimate the resource needs for Categories.

Gehel edited projects, added Data-Platform-SRE (2024.09.28 - 2024.10.18); removed Data-Platform-SRE (2024.09.06 - 2024.09.27).Fri, Sep 27, 1:00 PM
dr0ptp4kt moved this task from Incoming to Current work on the Wikidata-Query-Service board.Mon, Sep 30, 3:17 PM
dr0ptp4kt edited projects, added Discovery-Search (Current work); removed Wikidata-Query-Service.
bking added a subtask: T376079: eqiad: request 1 VM for wdqs-categories.Mon, Sep 30, 8:59 PM
bking updated the task description. (Show Details)Mon, Sep 30, 9:17 PM
bking added a comment.Mon, Sep 30, 9:21 PM

This dashboard has data about the individual blazegraph instances’ performance. Based on the data, categories’ resource usages are as follows:

gerritbot added a comment.Mon, Sep 30, 9:26 PM

Change #1076841 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs-categories: introduce VM for testing

https://gerrit.wikimedia.org/r/1076841

gerritbot added a project: Patch-For-Review.Mon, Sep 30, 9:26 PM
gerritbot added a comment.Wed, Oct 2, 2:20 PM

Change #1076841 merged by Bking:

[operations/puppet@production] wdqs-categories: introduce VM for testing

https://gerrit.wikimedia.org/r/1076841

Maintenance_bot removed a project: Patch-For-Review.Wed, Oct 2, 2:31 PM
gerritbot added a comment.Wed, Oct 2, 4:10 PM

Change #1077427 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs-categories: use correct insetup role

https://gerrit.wikimedia.org/r/1077427

gerritbot added a project: Patch-For-Review.Wed, Oct 2, 4:10 PM
gerritbot added a comment.Wed, Oct 2, 5:17 PM

Change #1077427 merged by Bking:

[operations/puppet@production] wdqs-categories: use correct insetup role

https://gerrit.wikimedia.org/r/1077427

Maintenance_bot removed a project: Patch-For-Review.Wed, Oct 2, 5:31 PM
Stashbot added a comment.Wed, Oct 2, 9:54 PM

Mentioned in SAL (#wikimedia-operations) [2024-10-02T21:54:58Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wdqs-categories1001.eqiad.wmnet with reason: T375687

Stashbot added a comment.Wed, Oct 2, 9:55 PM

Mentioned in SAL (#wikimedia-operations) [2024-10-02T21:55:13Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wdqs-categories1001.eqiad.wmnet with reason: T375687

gerritbot added a comment.Thu, Oct 3, 6:51 PM

Change #1077777 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/cookbooks@master] wdqs.categories-reload: don't check host

https://gerrit.wikimedia.org/r/1077777

gerritbot added a project: Patch-For-Review.Thu, Oct 3, 6:51 PM
gerritbot added a comment.Thu, Oct 3, 7:06 PM

Change #1077777 merged by Bking:

[operations/cookbooks@master] wdqs.categories-reload: don't check host

https://gerrit.wikimedia.org/r/1077777

Maintenance_bot removed a project: Patch-For-Review.Thu, Oct 3, 7:30 PM
bking closed subtask T376079: eqiad: request 1 VM for wdqs-categories as Resolved.Tue, Oct 8, 4:38 PM
bking updated the task description. (Show Details)Tue, Oct 8, 4:41 PM
bking closed this task as Resolved.EditedTue, Oct 8, 6:47 PM
bking claimed this task.
bking moved this task from Backlog - project to Needs Review on the Data-Platform-SRE (2024.09.28 - 2024.10.18) board.

I provisioned wdqs-categories1001 in T376079. After provisioning, I one-offed the host and loaded categories via /usr/local/bin/reloadCategories.sh wdqs . As demonstrated by this graph , the reload took ~2 hours. Post-reload, memory usage has been stable at ~10 GB. I think this evidence confirms that we can run categories in the Ganeti infrastructure if necessary. At this point, I'm ready to decom/destroy this VM and work on a migration in a future task.*

Moving to "Needs Review" so @dcausse has an opportunity to check out the VM, ask questions, run tests etc before we get rid of this VM.

  • Where we migrate is still an open question, this shouldn't be taken to mean we are definitely going to Ganeti.
dcausse added a comment.Tue, Oct 8, 7:56 PM
In T375687#10212080, @bking wrote:

I provisioned wdqs-categories1001 in T376079. After provisioning, I one-offed the host and loaded categories via /usr/local/bin/reloadCategories.sh wdqs . As demonstrated by this graph , the reload took ~2 hours. Post-reload, memory usage has been stable at ~10 GB. I think this evidence confirms that we can run categories in the Ganeti infrastructure if necessary. At this point, I'm ready to decom/destroy this VM and work on a migration in a future task.*

Moving to "Needs Review" so @dcausse has an opportunity to check out the VM, ask questions, run tests etc before we get rid of this VM.

  • Where we migrate is still an open question, this shouldn't be taken to mean we are definitely going to Ganeti.

Thanks for running the test, I can't access the VM but if the reload worked fine this is a very good indication that this should be enough.

brouberol moved this task from Needs Review to Done on the Data-Platform-SRE (2024.09.28 - 2024.10.18) board.Wed, Oct 9, 8:42 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits