Page MenuHomePhabricator
Create Task
Maniphest T301167

Service implementation for wdqs20[09,10,11,12]
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Procurement by dc-ops done in T291982; racking by dc-ops done in T294297

AC

Note

Details

SubjectRepoBranchLines +/-
wdqs: decom wdqs200[1-3]operations/puppetproduction+1 -7
Revert "wdqs: disable notifs on not-yet-in-service hosts"operations/puppetproduction+1 -5
wdqs: disable notifs on not-yet-in-service hostsoperations/puppetproduction+5 -1
wdqs data reload: clear out unused optionoperations/cookbooksmaster+2 -2
wdqs: Bring wdqs20[09,10,11,12] onlineoperations/puppetproduction+9 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

RKemper created this task.Feb 7 2022, 6:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 7 2022, 6:46 PM
Maintenance_bot added a project: Wikidata.Feb 7 2022, 7:45 PM
• MPhamWMF moved this task from needs triage to Current work on the Discovery-Search board.Feb 14 2022, 4:24 PM
• MPhamWMF edited projects, added Discovery-Search (Current work); removed Discovery-Search.
• MPhamWMF moved this task from Incoming to Current work on the Wikidata-Query-Service board.Feb 14 2022, 4:29 PM
bking subscribed.Feb 14 2022, 4:31 PM
• MPhamWMF set the point value for this task to 3.Feb 14 2022, 4:50 PM
• MPhamWMF moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.
bking mentioned this in T313539: Document Elastic host onboard process.Aug 9 2022, 6:13 PM
Gehel moved this task from Ready for Dev -- SWE to Prioritized on the Discovery-Search (Current work) board.Aug 22 2022, 3:40 PM
gerritbot added a comment.Nov 30 2022, 10:32 PM

Change 862369 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs: Bring wdqs20[09,10,11,12] online

https://gerrit.wikimedia.org/r/862369

gerritbot added a project: Patch-For-Review.Nov 30 2022, 10:32 PM
gerritbot added a comment.Dec 7 2022, 10:14 PM

Change 862369 merged by Bking:

[operations/puppet@production] wdqs: Bring wdqs20[09,10,11,12] online

https://gerrit.wikimedia.org/r/862369

Maintenance_bot removed a project: Patch-For-Review.Dec 7 2022, 10:30 PM
Stashbot added a comment.Dec 7 2022, 10:41 PM

Mentioned in SAL (#wikimedia-operations) [2022-12-07T22:41:40Z] <ryankemper> T301167 Downtimed wdqs20[09-12] for 7 days

RKemper moved this task from Prioritized to Blocked/Waiting on the Discovery-Search (Current work) board.Dec 7 2022, 10:51 PM

Waiting for data reload to complete on wdqs2009 (or another host) before we finish data transfers and pool these hosts

Stashbot added a comment.Dec 14 2022, 11:28 PM

Mentioned in SAL (#wikimedia-operations) [2022-12-14T23:28:29Z] <ryankemper> T301167 wdqs2011/2012 were not visible in pybal (oversight from when I added the other hosts with conftool last week). Fixed that, so now all of the new hosts are showing up properly.

Gehel added a subtask: T323096: WDQS Data Reload.Dec 19 2022, 4:27 PM
Gehel assigned this task to bking.Jan 5 2023, 2:37 PM
Gehel moved this task from Blocked/Waiting to In Progress on the Discovery-Search (Current work) board.
jcrespo mentioned this in T323096: WDQS Data Reload.Jan 11 2023, 10:58 AM
jcrespo subscribed.EditedJan 11 2023, 11:02 AM

One tip to avoid having people on call (like me) worrying about pending implementation services is to add the hiera key profile::monitoring::notifications_enabled: false. This is not promoted much because most people handle stateless services that are easy and fast to setup. If you, like me, handle stateful ones that can take a lot to load its data and be fully setup, this flag is useful until they are fully productionized. Hope this tip is found useful. Thank you and sorry for the previous ping!

gerritbot added a comment.Jan 17 2023, 8:00 PM

Change 881000 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: disable notifs on not-yet-in-service hosts

https://gerrit.wikimedia.org/r/881000

gerritbot added a project: Patch-For-Review.Jan 17 2023, 8:00 PM
RKemper added a comment.Jan 17 2023, 8:01 PM
In T301167#8516140, @jcrespo wrote:

One tip to avoid having people on call (like me) worrying about pending implementation services is to add the hiera key profile::monitoring::notifications_enabled: false. This is not promoted much because most people handle stateless services that are easy and fast to setup. If you, like me, handle stateful ones that can take a lot to load its data and be fully setup, this flag is useful until they are fully productionized. Hope this tip is found useful. Thank you and sorry for the previous ping!

@jcrespo Thanks for the suggestion, I wasn't aware of that flag! We'll definitely make use of that (https://gerrit.wikimedia.org/r/c/operations/puppet/+/881000)

gerritbot added a comment.Jan 17 2023, 8:01 PM

Change 881000 merged by Ryan Kemper:

[operations/puppet@production] wdqs: disable notifs on not-yet-in-service hosts

https://gerrit.wikimedia.org/r/881000

Gehel moved this task from In Progress to Blocked/Waiting on the Discovery-Search (Current work) board.Feb 6 2023, 4:31 PM
Maintenance_bot removed a project: Patch-For-Review.Feb 6 2023, 5:30 PM
gerritbot added a comment.Feb 13 2023, 2:13 PM

Change 888707 had a related patch set uploaded (by Bking; author: Bking):

[operations/cookbooks@master] wdqs data reload: clear out unused option

https://gerrit.wikimedia.org/r/888707

gerritbot added a project: Patch-For-Review.Feb 13 2023, 2:13 PM
gerritbot added a comment.Feb 13 2023, 2:21 PM

Change 888707 merged by Bking:

[operations/cookbooks@master] wdqs data reload: clear out unused option

https://gerrit.wikimedia.org/r/888707

Maintenance_bot removed a project: Patch-For-Review.Feb 13 2023, 2:31 PM
gerritbot added a comment.Feb 27 2023, 10:35 PM

Change 891626 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] [WIP] Revert "wdqs: disable notifs on not-yet-in-service hosts"

https://gerrit.wikimedia.org/r/891626

gerritbot added a project: Patch-For-Review.Feb 27 2023, 10:35 PM
RKemper updated the task description. (Show Details)Feb 27 2023, 10:36 PM
RKemper updated the task description. (Show Details)Feb 27 2023, 10:49 PM

Arbitrarily chosen wdqs2012 is happy as seen by our query test suite:

image.png (191×1 px, 53 KB)

All that should be left is decoming wdqs200[1-3]

RKemper added a comment.Feb 27 2023, 10:58 PM
In T301167#8650988, @RKemper wrote:

Arbitrarily chosen wdqs2012 is happy as seen by our query test suite:

image.png (191×1 px, 53 KB)

All that should be left is decoming wdqs200[1-3]

Actually, this only tested the wikidata journal. We still need to transfer the categories journal over before these hosts are fully happy.

RKemper updated the task description. (Show Details)Feb 27 2023, 10:58 PM
bking mentioned this in T330714: Document SRE steps for deploying a new WDQS (and WCQS) host.Feb 27 2023, 11:25 PM
gerritbot added a comment.Feb 28 2023, 7:20 PM

Change 891626 merged by Ryan Kemper:

[operations/puppet@production] Revert "wdqs: disable notifs on not-yet-in-service hosts"

https://gerrit.wikimedia.org/r/891626

Stashbot added a comment.Feb 28 2023, 7:21 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-28T19:21:16Z] <ryankemper> [WDQS] (The following was ~20 hours ago, forgot to press enter) T301167 Transferred /srv/wdqs/categories.jnl from wdqs2001 (in-service host) to wdqs20[09-12] (new hosts being brought into service)

Stashbot added a comment.Feb 28 2023, 7:21 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-28T19:21:51Z] <ryankemper> [WDQS] (Current time) T301167 Re-enabled icinga notifications for wdqs20[09-12]

RKemper updated the task description. (Show Details)Feb 28 2023, 7:24 PM
Maintenance_bot removed a project: Patch-For-Review.Feb 28 2023, 7:31 PM
Gehel moved this task from Blocked/Waiting to In Progress on the Discovery-Search (Current work) board.Mar 1 2023, 2:42 PM
gerritbot added a comment.Mar 2 2023, 10:14 PM

Change 893825 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: decom wdqs200[1-3]

https://gerrit.wikimedia.org/r/893825

gerritbot added a project: Patch-For-Review.Mar 2 2023, 10:14 PM
gerritbot added a comment.Mar 2 2023, 10:28 PM

Change 893825 merged by Ryan Kemper:

[operations/puppet@production] wdqs: decom wdqs200[1-3]

https://gerrit.wikimedia.org/r/893825

Maintenance_bot removed a project: Patch-For-Review.Mar 2 2023, 10:31 PM
RKemper added a comment.EditedMar 2 2023, 10:33 PM

Decom ticket for dc-ops: T331074

ops-monitoring-bot added a comment.Mar 2 2023, 11:39 PM

cookbooks.sre.hosts.decommission executed by ryankemper@cumin2002 for hosts: wdqs[2001-2003].codfw.wmnet

  • wdqs2001.codfw.wmnet (WARN)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • wdqs2002.codfw.wmnet (WARN)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • wdqs2003.codfw.wmnet (WARN)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
Gehel moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board.Mar 6 2023, 4:26 PM
Gehel closed subtask T323096: WDQS Data Reload as Resolved.Mar 10 2023, 2:05 PM
Gehel closed this task as Resolved.Mar 10 2023, 2:15 PM
RKemper updated the task description. (Show Details)Jul 13 2023, 5:35 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits