Page MenuHomePhabricator
Create Task
Maniphest T384001

decommission db2131.codfw.wmnet
Closed, ResolvedPublicRequest
Actions

Description

This task will track the decommission-hardware of server db2131.codfw.wmnet

With the launch of updates to the decom cookbook, the majority of these steps can be handled by the service owners directly. The DC Ops team only gets involved once the system has been fully removed from service and powered down by the decommission cookbook.

db2131

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (likely done by script)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal, and run homer.
  • - remove all remaining puppet references and all host entries in the puppet repo
  • - reassign task from service owner to no owner and ensure the site project (ops-sitename depending on site of server) is assigned.

End service owner steps / Begin DC-Ops team steps:

  • - system disks removed (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: set netbox state to 'inventory' and hostname to asset tag

Details

SubjectRepoBranchLines +/-
orchestrator.conf: Add Federicooperations/puppetproduction+1 -0
site.pp: Remove db2131operations/puppetproduction+1 -1
db2131.yaml: remove fileoperations/puppetproduction+0 -4
instances.yaml: Remove db2131operations/puppetproduction+0 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

FCeratto-WMF created this task.Jan 17 2025, 12:48 PM
Marostegui moved this task from Triage to In progress on the DBA board.Jan 20 2025, 9:11 AM
FCeratto-WMF added a parent task: T374623: Decommission db21[21-40].Jan 20 2025, 11:27 AM
FCeratto-WMF added a comment.EditedJan 20 2025, 11:31 AM

Starting decommissioning.
Runbook:

  • Depool the host:
sudo dbctl instance db2131 depool
sudo dbctl config commit -m "Depool db2131 T384001"
  • Make puppet patch to remove it from dbctl
  • Merge the puppet patch
  • Run dbctl commit:
sudo dbctl config commit -m "Remove db2131 from dbctl T384001"
  • Make puppet patch cleaning the rest up (DO NOT merge)
  • Run the decommissioning script:
sudo cookbook sre.hosts.decommission -t T384001 db2131.codfw.wmnet
  • Merge the puppet patch
  • Remove from zacillo
    • In IRC: !log Removing db2131 from zarcillo T384001
    • In cumin: sudo db-mysql db1215 -A zarcillo
set binlog_format='ROW';
delete from servers where hostname="db2131";
delete from instances where name like 'db2131%';
delete from section_instances where instance like 'db2131%';
  • Remove host from orchestrator
  • Update the task and hand it over to dcops
FCeratto-WMF changed the task status from Open to In Progress.Jan 20 2025, 11:33 AM
FCeratto-WMF mentioned this in T374623: Decommission db21[21-40].
Stashbot added a comment.Jan 20 2025, 11:41 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-20T11:41:55Z] <federico3> depooling db2131 as per T384001

Stashbot added a comment.Jan 20 2025, 11:43 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-20T11:43:07Z] <fceratto@cumin1002> dbctl commit (dc=all): 'Depool db2131 T384001', diff saved to https://phabricator.wikimedia.org/P72153 and previous config saved to /var/cache/conftool/dbconfig/20250120-114306-fceratto.json

ops-monitoring-bot added a comment.Jan 20 2025, 11:48 AM

Icinga downtime and Alertmanager silence (ID=4afc4f81-4fa5-47a8-a2f2-53901aebc3ae) set by fceratto@cumin1002 for 1:00:00 on 1 host(s) and their services with reason: Downtime db2131

db2131.codfw.wmnet
gerritbot added a comment.Jan 20 2025, 11:59 AM

Change #1112727 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] instances.yaml: Remove db2131

https://gerrit.wikimedia.org/r/1112727

gerritbot added a project: Patch-For-Review.Jan 20 2025, 11:59 AM
gerritbot added a comment.Jan 20 2025, 12:06 PM

Change #1112727 merged by Federico Ceratto:

[operations/puppet@production] instances.yaml: Remove db2131

https://gerrit.wikimedia.org/r/1112727

Stashbot added a comment.Jan 20 2025, 12:13 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-20T12:13:19Z] <fceratto@cumin1002> dbctl commit (dc=all): 'Remove db2131 from dbctl T384001', diff saved to https://phabricator.wikimedia.org/P72154 and previous config saved to /var/cache/conftool/dbconfig/20250120-121318-fceratto.json

gerritbot added a comment.Jan 20 2025, 12:18 PM

Change #1112741 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] db2131.yaml: remove file

https://gerrit.wikimedia.org/r/1112741

gerritbot added a comment.Jan 20 2025, 12:24 PM

Change #1112745 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] site.pp: Remove db2131

https://gerrit.wikimedia.org/r/1112745

gerritbot added a comment.Jan 20 2025, 12:26 PM

Change #1112741 merged by Federico Ceratto:

[operations/puppet@production] db2131.yaml: remove file

https://gerrit.wikimedia.org/r/1112741

gerritbot added a comment.Jan 20 2025, 12:26 PM

Change #1112745 merged by Federico Ceratto:

[operations/puppet@production] site.pp: Remove db2131

https://gerrit.wikimedia.org/r/1112745

Maintenance_bot removed a project: Patch-For-Review.Jan 20 2025, 12:30 PM
FCeratto-WMF added a comment.EditedJan 20 2025, 12:46 PM

The decommissioning script found unmerged changes for another host:
3.193.10.in-addr.arpa -> removing "71 1H IN PTR wmf6766.mgmt.codfw.wmnet."
codfw.wmnet -> "71 1H IN PTR wmf6766.mgmt.codfw.wmne"
mgmt.codfw.wmnet -> "wmf6766 1H IN A 10.193.3.7"
..and few other files always with the same host.

(pinging @Volans FYI)

FCeratto-WMF added a subscriber: Volans.Jan 20 2025, 12:47 PM
ops-monitoring-bot added a comment.Jan 20 2025, 12:49 PM

cookbooks.sre.hosts.decommission executed by fceratto@cumin1002 for hosts: db2131.codfw.wmnet

  • db2131.codfw.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Failed to power off, manual intervention required: Remote IPMI for db2131.mgmt.codfw.wmnet failed (exit=1): b''
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

Stashbot added a comment.Jan 20 2025, 12:55 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-20T12:55:46Z] <federico3> Removing db2131 from zarcillo T384001

gerritbot added a comment.Jan 20 2025, 12:58 PM

Change #1112752 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] orchestrator.conf: Add Federico

https://gerrit.wikimedia.org/r/1112752

gerritbot added a project: Patch-For-Review.Jan 20 2025, 12:58 PM
gerritbot added a comment.Jan 20 2025, 1:03 PM

Change #1112752 merged by Marostegui:

[operations/puppet@production] orchestrator.conf: Add Federico

https://gerrit.wikimedia.org/r/1112752

Maintenance_bot removed a project: Patch-For-Review.Jan 20 2025, 1:30 PM
FCeratto-WMF removed FCeratto-WMF as the assignee of this task.Jan 20 2025, 1:38 PM
FCeratto-WMF updated the task description. (Show Details)
FCeratto-WMF updated the task description. (Show Details)
FCeratto-WMF added a project: ops-codfw.Jan 20 2025, 1:41 PM
Restricted Application added a project: DC-Ops. · View Herald TranscriptJan 20 2025, 1:41 PM
FCeratto-WMF added a subscriber: elukey.Jan 20 2025, 1:42 PM

This is ready for DCOps. The host shutdown step in the decommissioning failed but I'm advised by @elukey that the decommissioning can proceed.

Marostegui moved this task from In progress to Done on the DBA board.Jan 20 2025, 1:51 PM
Maintenance_bot added a project: SRE.Jan 20 2025, 2:29 PM
Jhancock.wm moved this task from Backlog to Decommission on the ops-codfw board.Jan 21 2025, 5:26 AM
Jhancock.wm closed this task as Resolved.Jan 22 2025, 5:30 PM
Jhancock.wm claimed this task.
Jhancock.wm updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits