Page MenuHomePhabricator
Create Task
Maniphest T255406

decommission dbproxy1008.eqiad.wmnet
Closed, ResolvedPublicRequest
Actions

Description

This task will track the decommission-hardware of server dbproxy1008.eqiad.wmnet.

With the launch of updates to the decom cookbook, the majority of these steps can be handled by the service owners directly. The DC Ops team only gets involved once the system has been fully removed from service and powered down by the decommission cookbook.

dbproxy1008

Steps for service owner:

  • - Remove grants T231280 https://gerrit.wikimedia.org/r/607407
  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. https://gerrit.wikimedia.org/r/606066
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system) recommended to ensure services offline but not 100% required as long as the decom script is IMMEDIATELY run below.
  • - login to cumin host and run the decom cookbook: cookbook sre.hosts.decommission <host fqdn> -t <phab task>. This does: bootloader wipe, host power down, netbox update to decommissioning status, puppet node clean, puppet node deactivate, debmonitor removal.
  • - remove all remaining puppet references (include role::spare) and all host entries in the puppet repo
  • - remove ALL dns entries except the asset tag mgmt entries.
  • - reassign task from service owner to DC ops team member depending on site of servee.

End service owner steps / Begin DC-Ops team steps:

  • - disable switch port / set to asset tag if host isn't being unracked / remove from switch if being unracked.
  • - system disks wiped (by onsite)
  • - determine system age, under 5 years are reclaimed to spare, over 5 years are decommissioned.
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and set state to offline
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
Missed the asset tag for dbproxy1008 mgmt removaloperations/dnsmaster+0 -1
Removing mgmt dns entries for dbproxy1008operations/dnsmaster+0 -3
check-microcode.py: Remove dbproxy1008operations/puppetproduction+5 -6
templates: Remove dbproxy1008 production entriesoperations/dnsmaster+0 -2
mariadb: Remove dbproxy1008operations/puppetproduction+1 -14
report_users: Remove dbproxy1008operations/softwaremaster+3 -3
production-m3.sql: Remove grants for dbproxy1008operations/puppetproduction+0 -45
dbproxy1008: Disable notificationsoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Jun 15 2020, 9:29 AM
Marostegui moved this task from Triage to Pending comment on the DBA board.

dbproxy1008 is no longer the active m3-master per https://gerrit.wikimedia.org/r/#/c/operations/dns/+/605531/
Let's give it a few days before starting its decommissioning process.

Marostegui added a parent task: T231280: Remove grants for the old dbproxy hosts from the misc databases.Jun 15 2020, 9:30 AM
Marostegui mentioned this in T231280: Remove grants for the old dbproxy hosts from the misc databases.
Marostegui updated the task description. (Show Details)
Marostegui mentioned this in T255408: Upgrade dbproxyXXXX to Buster.Jun 15 2020, 9:42 AM
gerritbot added a comment.Jun 17 2020, 4:59 AM

Change 606066 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1008: Disable notifications

https://gerrit.wikimedia.org/r/606066

gerritbot added a project: Patch-For-Review.Jun 17 2020, 4:59 AM

Change 606066 merged by Marostegui:
[operations/puppet@production] dbproxy1008: Disable notifications

https://gerrit.wikimedia.org/r/606066

Marostegui updated the task description. (Show Details)Jun 17 2020, 4:59 AM
Maintenance_bot removed a project: Patch-For-Review.Jun 17 2020, 5:10 AM
Stashbot added a comment.Jun 22 2020, 5:43 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-22T05:43:55Z] <marostegui> Stop haproxy on dbproxy1008 - T255406

Marostegui updated the task description. (Show Details)Jun 22 2020, 5:46 AM
• Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Jun 22 2020, 5:00 PM
Marostegui moved this task from Pending comment to In progress on the DBA board.Jun 24 2020, 5:13 AM
Stashbot added a comment.Jun 24 2020, 5:14 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-24T05:14:56Z] <marostegui> Remove grants from dbproxy1008 - T231280 T255406

Marostegui added a comment.Jun 24 2020, 5:17 AM

dbproxy1008 grants removed from m3 (and also checked all the other mX sections):

root@cumin2001:/home/marostegui# ./section m3 | while read host port; do echo $host; mysql.py -h$host:$port -e "show grants for 'haproxy'@'10.64.32.157';"; done
db2134.codfw.wmnet
ERROR 1141 (42000) at line 1: There is no such grant defined for user 'haproxy' on host '10.64.32.157'
db2078.codfw.wmnet
ERROR 1141 (42000) at line 1: There is no such grant defined for user 'haproxy' on host '10.64.32.157'
db1128.eqiad.wmnet
ERROR 1141 (42000) at line 1: There is no such grant defined for user 'haproxy' on host '10.64.32.157'
db1117.eqiad.wmnet
ERROR 1141 (42000) at line 1: There is no such grant defined for user 'haproxy' on host '10.64.32.157'
gerritbot added a comment.Jun 24 2020, 5:19 AM

Change 607407 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] production-m3.sql: Remove grants for dbproxy1008

https://gerrit.wikimedia.org/r/607407

gerritbot added a project: Patch-For-Review.Jun 24 2020, 5:19 AM
gerritbot added a comment.Jun 24 2020, 5:27 AM

Change 607407 merged by Marostegui:
[operations/puppet@production] production-m3.sql: Remove grants for dbproxy1008

https://gerrit.wikimedia.org/r/607407

Marostegui updated the task description. (Show Details)Jun 24 2020, 5:28 AM
gerritbot added a comment.Jun 24 2020, 5:30 AM

Change 607408 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] report_users: Remove dbproxy1008

https://gerrit.wikimedia.org/r/607408

gerritbot added a comment.Jun 24 2020, 5:32 AM

Change 607409 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove dbproxy1008

https://gerrit.wikimedia.org/r/607409

gerritbot added a comment.Jun 24 2020, 5:34 AM

Change 607408 merged by Marostegui:
[operations/software@master] report_users: Remove dbproxy1008

https://gerrit.wikimedia.org/r/607408

ops-monitoring-bot added a comment.Jun 24 2020, 5:34 AM

cookbooks.sre.hosts.decommission executed by marostegui@cumin2001 for hosts: dbproxy1008.eqiad.wmnet

  • dbproxy1008.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Jun 24 2020, 5:34 AM

Change 607409 merged by Marostegui:
[operations/puppet@production] mariadb: Remove dbproxy1008

https://gerrit.wikimedia.org/r/607409

Marostegui updated the task description. (Show Details)Jun 24 2020, 5:35 AM
gerritbot added a comment.Jun 24 2020, 5:37 AM

Change 607410 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] templates: Remove dbproxy1008 production entries

https://gerrit.wikimedia.org/r/607410

gerritbot added a comment.Jun 24 2020, 5:41 AM

Change 607411 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] check-microcode.py: Remove dbproxy1008

https://gerrit.wikimedia.org/r/607411

gerritbot added a comment.Jun 24 2020, 5:41 AM

Change 607410 merged by Marostegui:
[operations/dns@master] templates: Remove dbproxy1008 production entries

https://gerrit.wikimedia.org/r/607410

Marostegui updated the task description. (Show Details)Jun 24 2020, 5:55 AM
Marostegui updated the task description. (Show Details)Jun 24 2020, 6:01 AM
Marostegui removed projects: Patch-For-Review, DBA.
Marostegui added a project: DC-Ops.
Marostegui edited subscribers, added: wiki_willy; removed: Marostegui.

This is ready for DC-Ops.

gerritbot added a comment.Jun 24 2020, 7:05 AM

Change 607411 merged by Muehlenhoff:
[operations/puppet@production] check-microcode.py: Remove dbproxy1008

https://gerrit.wikimedia.org/r/607411

Marostegui reassigned this task from Marostegui to wiki_willy.Jun 24 2020, 7:16 AM
Marostegui updated the task description. (Show Details)
Marostegui subscribed.
Marostegui moved this task from Backlog to pending onsite steps (eqiad) on the decommission-hardware board.Jun 24 2020, 9:58 AM
wiki_willy reassigned this task from wiki_willy to Jclark-ctr.Jun 26 2020, 6:37 PM
Jclark-ctr updated the task description. (Show Details)Jul 20 2020, 10:36 PM
• Cmjohnson claimed this task.Aug 4 2020, 5:51 PM
• Cmjohnson added a subscriber: Jclark-ctr.
gerritbot added a comment.Aug 10 2020, 6:15 PM

Change 619338 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Removing mgmt dns entries for dbproxy1008

https://gerrit.wikimedia.org/r/619338

gerritbot added a project: Patch-For-Review.Aug 10 2020, 6:15 PM
gerritbot added a comment.Aug 10 2020, 6:17 PM

Change 619338 merged by Cmjohnson:
[operations/dns@master] Removing mgmt dns entries for dbproxy1008

https://gerrit.wikimedia.org/r/619338

gerritbot added a comment.Aug 10 2020, 6:26 PM

Change 619343 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Missed the asset tag for dbproxy1008 mgmt removal

https://gerrit.wikimedia.org/r/619343

gerritbot added a comment.Aug 10 2020, 6:27 PM

Change 619343 merged by Cmjohnson:
[operations/dns@master] Missed the asset tag for dbproxy1008 mgmt removal

https://gerrit.wikimedia.org/r/619343

• Cmjohnson closed this task as Resolved.Aug 10 2020, 6:28 PM
• Cmjohnson updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits