Page MenuHomePhabricator
Create Task
Maniphest T309810

Service implementation for elastic1[084-102].eqiad.wmnet
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

This is a ticket composed of:

elastic108[4-8] procured in T291655 and racked in T294152; these refresh hosts are replacing elastic10[48-52]

elastic1[089-102] procured in T297645 and racked in T299609; these are 14 new expansion hosts

TODO rest of ticket description, use https://phabricator.wikimedia.org/T300943 as a template

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
elastic: pick new canary hostoperations/puppetproduction+1 -1
elastic: decom elastic10[51-52]operations/puppetproduction+0 -6
elastic: decom elastic10[49-50]operations/puppetproduction+0 -6
elastic: decom elastic1048operations/puppetproduction+0 -3
elastic: fix bad copypasteoperations/puppetproduction+2 -2
elastic: allocate psi vs omega for new hostsoperations/puppetproduction+66 -3
elastic: racking info for new hostsoperations/puppetproduction+30 -0
Customize query in gerrit

Related Objects

Event Timeline

RKemper created this task.Jun 2 2022, 8:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 2 2022, 8:14 PM
• MPhamWMF triaged this task as High priority.Jun 6 2022, 3:33 PM
• MPhamWMF moved this task from needs triage to Ops / SRE on the Discovery-Search board.
bking claimed this task.Aug 9 2022, 5:55 PM
bking edited projects, added Discovery-Search (Current work); removed Discovery-Search.
bking added a comment.Aug 9 2022, 6:09 PM

Starting this work now, as we need more capacity for the Bullseye upgrades.

bking mentioned this in T313539: Document Elastic host onboard process.Aug 9 2022, 6:14 PM
gerritbot added a comment.Aug 10 2022, 6:22 PM

Change 822129 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] elastic: racking info for new hosts

https://gerrit.wikimedia.org/r/822129

gerritbot added a project: Patch-For-Review.Aug 10 2022, 6:22 PM
RKemper added a comment.Aug 10 2022, 6:28 PM

Here's the rack/row info for the new hosts, corresponding to https://gerrit.wikimedia.org/r/c/operations/puppet/+/822129/ pulled from Netbox:

elastic1084 A4
elastic1085 B7
elastic1086 B7
elastic1087 C7
elastic1088 C7

elastic1089 E1
elastic1090 E1

elastic1091 E2
elastic1092 E2

elastic1093 E3
elastic1094 E3
elastic1095 E3

elastic1096 F1
elastic1097 F1

elastic1098 F2
elastic1099 F2

elastic1100 F3
elastic1101 F3
elastic1102 F3
gerritbot added a comment.Aug 10 2022, 7:19 PM

Change 822129 merged by Ryan Kemper:

[operations/puppet@production] elastic: racking info for new hosts

https://gerrit.wikimedia.org/r/822129

Maintenance_bot removed a project: Patch-For-Review.Aug 10 2022, 7:30 PM
gerritbot added a comment.Aug 10 2022, 8:03 PM

Change 822169 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] elastic: allocate psi vs omega for new hosts

https://gerrit.wikimedia.org/r/822169

gerritbot added a project: Patch-For-Review.Aug 10 2022, 8:03 PM
RKemper added a comment.Aug 10 2022, 8:31 PM
host        row assigned_small_cluster

elastic1084 A4  Psi
elastic1085 B7  Psi
elastic1086 B7  Psi
elastic1087 C7  Psi
elastic1088 C7  Psi

elastic1089 E1  Omega
elastic1090 E1  Psi  

elastic1091 E2  Omega
elastic1092 E2  Psi  

elastic1093 E3  Omega
elastic1094 E3  Omega
elastic1095 E3  Psi  

elastic1096 F1  Omega
elastic1097 F1  Psi  

elastic1098 F2  Omega
elastic1099 F2  Psi  

elastic1100 F3  Omega
elastic1101 F3  Psi  
elastic1102 F3  Psi

Note that elastic108[4-8] are replacing elastic10[48-52] which are psi hosts, thus why that whole block was assigned to psi.

gerritbot added a comment.Aug 10 2022, 8:42 PM

Change 822169 merged by Ryan Kemper:

[operations/puppet@production] elastic: allocate psi vs omega for new hosts

https://gerrit.wikimedia.org/r/822169

Stashbot added a comment.Aug 10 2022, 9:09 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-10T21:09:30Z] <bking@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on elastic[1101-1102].eqiad.wmnet with reason: T309810

Stashbot added a comment.Aug 10 2022, 9:09 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-10T21:09:44Z] <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on elastic[1101-1102].eqiad.wmnet with reason: T309810

Stashbot added a comment.Aug 10 2022, 9:10 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-10T21:10:16Z] <bking@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on 16 hosts with reason: T309810

Stashbot added a comment.Aug 10 2022, 9:10 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-10T21:10:39Z] <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 16 hosts with reason: T309810

gerritbot added a comment.Aug 10 2022, 9:16 PM

Change 822173 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] elastic: fix bad copypaste

https://gerrit.wikimedia.org/r/822173

gerritbot added a comment.Aug 10 2022, 9:18 PM

Change 822173 merged by Ryan Kemper:

[operations/puppet@production] elastic: fix bad copypaste

https://gerrit.wikimedia.org/r/822173

Stashbot added a comment.Aug 11 2022, 4:29 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-11T16:29:37Z] <bking@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic[1100-1102].eqiad.wmnet with reason: T309810

Stashbot added a comment.Aug 11 2022, 4:29 PM

Mentioned in SAL (#wikimedia-operations) [2022-08-11T16:29:52Z] <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic[1100-1102].eqiad.wmnet with reason: T309810

bking mentioned this in T315038: Overlay VRF / VXLAN traffic failure between lsw1-f2-eqiad and lsw1-f3-eqiad.Aug 11 2022, 5:59 PM
bking subscribed.Aug 15 2022, 3:24 PM
• MPhamWMF set the point value for this task to 5.Aug 15 2022, 3:40 PM
• MPhamWMF moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.
gerritbot added a comment.Aug 16 2022, 9:37 PM

Change 823747 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elastic: decom elastic1048

https://gerrit.wikimedia.org/r/823747

gerritbot added a comment.Aug 16 2022, 9:41 PM

Change 823747 merged by Bking:

[operations/puppet@production] elastic: decom elastic1048

https://gerrit.wikimedia.org/r/823747

ops-monitoring-bot added a comment.Aug 16 2022, 9:53 PM

cookbooks.sre.hosts.decommission executed by bking@cumin1001 for hosts: elastic1048.eqiad.wmnet

  • elastic1048.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Icinga/Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Aug 17 2022, 12:48 AM

Change 823771 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] elastic: decom elastic10[49-50]

https://gerrit.wikimedia.org/r/823771

gerritbot added a comment.Aug 17 2022, 12:51 AM

Change 823771 merged by Ryan Kemper:

[operations/puppet@production] elastic: decom elastic10[49-50]

https://gerrit.wikimedia.org/r/823771

ops-monitoring-bot added a comment.Aug 17 2022, 2:07 AM

cookbooks.sre.hosts.decommission executed by ryankemper@cumin1001 for hosts: elastic[1049-1050].eqiad.wmnet

  • elastic1049.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Icinga/Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • elastic1050.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Icinga/Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Aug 17 2022, 2:10 AM

Change 823786 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] elastic: decom elastic10[51-52]

https://gerrit.wikimedia.org/r/823786

gerritbot added a comment.Aug 17 2022, 2:15 AM

Change 823786 merged by Ryan Kemper:

[operations/puppet@production] elastic: decom elastic10[51-52]

https://gerrit.wikimedia.org/r/823786

gerritbot added a comment.Aug 17 2022, 2:20 AM

Change 823788 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] elastic: pick new canary host

https://gerrit.wikimedia.org/r/823788

gerritbot added a comment.Aug 17 2022, 2:32 AM

Change 823788 merged by Ryan Kemper:

[operations/puppet@production] elastic: pick new canary host

https://gerrit.wikimedia.org/r/823788

ops-monitoring-bot added a comment.Aug 17 2022, 2:58 AM

cookbooks.sre.hosts.decommission executed by ryankemper@cumin1001 for hosts: elastic[1051-1052].eqiad.wmnet

  • elastic1051.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Icinga/Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • elastic1052.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Icinga/Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board.Aug 22 2022, 3:20 PM
RKemper added a comment.Aug 23 2022, 7:46 PM

Remembered I still need to create dc-ops decom ticket for the 5 eqiad elastic refresh hosts

RKemper added a comment.Aug 31 2022, 12:20 AM
In T309810#8179500, @RKemper wrote:

Remembered I still need to create dc-ops decom ticket for the 5 eqiad elastic refresh hosts

Created this decom task: https://phabricator.wikimedia.org/T316728

Gehel closed this task as Resolved.Sep 26 2022, 9:52 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits