Page MenuHomePhabricator
Create Task
Maniphest T335281

Drain and then decommission ms-be10[40-43]
Closed, ResolvedPublic
Actions

Description

These are being replaced by ms-be107[2-5] cf T335279

Details

SubjectRepoBranchLines +/-
hiera: remove ms-be104[0-3] from profile::swift::storagehostsoperations/puppetproduction+0 -4
swift: remove ms-be104[0-3] from the ringsoperations/puppetproduction+0 -8
swift: add new nodes, drain old nodes from the ringsoperations/puppetproduction+26 -8
Customize query in gerrit

Related Objects
Search...

Event Timeline

MatthewVernon created this task.Apr 24 2023, 10:58 AM
gerritbot added a comment.Apr 25 2023, 10:41 AM

Change 911779 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: add new nodes, drain old nodes from the rings

https://gerrit.wikimedia.org/r/911779

gerritbot added a project: Patch-For-Review.Apr 25 2023, 10:41 AM
gerritbot added a comment.Apr 26 2023, 7:37 AM

Change 911779 merged by MVernon:

[operations/puppet@production] swift: add new nodes, drain old nodes from the rings

https://gerrit.wikimedia.org/r/911779

Stashbot mentioned this in T335279: Bring ms-be107[2-5] into the rings.Apr 26 2023, 7:39 AM
Stashbot mentioned this in T335278: Bring ms-be207[0-3] into the rings.

Mentioned in SAL (#wikimedia-operations) [2023-04-26T07:39:24Z] <Emperor> start to load new swift backends, drain old ones T335278 T335279 T335280 T335281

Stashbot mentioned this in T335280: Drain and then decommission ms-be20[40-43].Apr 26 2023, 7:39 AM
Maintenance_bot removed a project: Patch-For-Review.Apr 26 2023, 8:10 AM
MatthewVernon added a subtask: T336778: Urgent: disk failed in ms-be1063.May 17 2023, 9:20 AM
Jclark-ctr closed subtask T336778: Urgent: disk failed in ms-be1063 as Resolved.May 17 2023, 2:12 PM
gerritbot added a comment.Jun 13 2023, 9:18 AM

Change 929642 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: remove ms-be104[0-3] from the rings

https://gerrit.wikimedia.org/r/929642

gerritbot added a project: Patch-For-Review.Jun 13 2023, 9:18 AM
MatthewVernon added a comment.Jun 13 2023, 9:18 AM

These are all at 0 weight in the rings, so can now be removed from the rings ready to decommission.

gerritbot added a comment.Jun 13 2023, 9:22 AM

Change 929642 merged by MVernon:

[operations/puppet@production] swift: remove ms-be104[0-3] from the rings

https://gerrit.wikimedia.org/r/929642

Maintenance_bot removed a project: Patch-For-Review.Jun 13 2023, 9:30 AM
gerritbot added a comment.Jun 13 2023, 3:19 PM

Change 929730 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] hiera: remove ms-be104[0-3] from profile::swift::storagehosts

https://gerrit.wikimedia.org/r/929730

gerritbot added a project: Patch-For-Review.Jun 13 2023, 3:19 PM
gerritbot added a comment.Jun 14 2023, 7:59 AM

Change 929730 merged by MVernon:

[operations/puppet@production] hiera: remove ms-be104[0-3] from profile::swift::storagehosts

https://gerrit.wikimedia.org/r/929730

Maintenance_bot removed a project: Patch-For-Review.Jun 14 2023, 8:10 AM
ops-monitoring-bot added a comment.Jun 14 2023, 10:30 AM

cookbooks.sre.hosts.decommission executed by mvernon@cumin1001 for hosts: ms-be[1040-1043].eqiad.wmnet

  • ms-be1040.eqiad.wmnet (WARN)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be1041.eqiad.wmnet (WARN)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be1042.eqiad.wmnet (WARN)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • ms-be1043.eqiad.wmnet (WARN)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
MatthewVernon closed this task as Resolved.Jun 14 2023, 10:32 AM

Nodes decommissioned, handover to DC-Ops in T339100.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits