Page MenuHomePhabricator

Decommission db1001
Closed, ResolvedPublic

Description

Wait until db1065 is in place

Decommission Checklist

  • - all system services confirmed offline from production use - should be done by DBA team:
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration - should be done by DBA team
  • - any service group puppet/heira/dsh config removed - should be done by DBA team
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.) - should be done by DBA team: https://gerrit.wikimedia.org/r/#/c/421216/

START NON-INTERRUPPTABLE STEPS - please assign to @RobH for the non-interrupt steps

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

Related Gerrit Patches:
operations/dns : masterRemoving mgmt dns for db1001
operations/dns : masterdecom old db systems db1001, db1011, db1016
operations/puppet : productiondecom of db1001, db1011, and db1016
operations/software : masterm1.hosts: Remove db1001
operations/puppet : productionmariadb: Remove db1001
operations/puppet : productiondbproxy100{1,6}: Change standby host

Event Timeline

Marostegui triaged this task as Medium priority.Mar 21 2018, 10:46 AM
Marostegui created this task.
Marostegui moved this task from Triage to In progress on the DBA board.

db1065 is now replicating in m1, let's wait 24h before going to decommission this host

Change 420991 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy100{1,6}: Change standby host

https://gerrit.wikimedia.org/r/420991

Change 420991 merged by Marostegui:
[operations/puppet@production] dbproxy100{1,6}: Change standby host

https://gerrit.wikimedia.org/r/420991

Marostegui updated the task description. (Show Details)Mar 22 2018, 6:24 AM

Mentioned in SAL (#wikimedia-operations) [2018-03-22T06:25:06Z] <marostegui> Stop MySQL on db1001 to get ready to decommission it - T190262

Mentioned in SAL (#wikimedia-operations) [2018-03-22T06:25:53Z] <marostegui> Remove db1001 from tendril - T190262

Change 421216 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove db1001

https://gerrit.wikimedia.org/r/421216

Change 421217 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] m1.hosts: Remove db1001

https://gerrit.wikimedia.org/r/421217

Change 421216 merged by Marostegui:
[operations/puppet@production] mariadb: Remove db1001

https://gerrit.wikimedia.org/r/421216

Change 421217 merged by jenkins-bot:
[operations/software@master] m1.hosts: Remove db1001

https://gerrit.wikimedia.org/r/421217

Marostegui assigned this task to RobH.Mar 22 2018, 6:36 AM
Marostegui updated the task description. (Show Details)
Marostegui moved this task from In progress to Done on the DBA board.

This host is now ready for DC Ops steps.
Assigning it to @RobH

Restricted Application added a project: Operations. · View Herald TranscriptMar 22 2018, 6:38 AM

Change 421574 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom of db1001, db1011, and db1016

https://gerrit.wikimedia.org/r/421574

Change 421578 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom old db systems db1001, db1011, db1016

https://gerrit.wikimedia.org/r/421578

Change 421574 merged by RobH:
[operations/puppet@production] decom of db1001, db1011, and db1016

https://gerrit.wikimedia.org/r/421574

Change 421578 merged by RobH:
[operations/dns@master] decom old db systems db1001, db1011, db1016

https://gerrit.wikimedia.org/r/421578

RobH reassigned this task from RobH to Cmjohnson.Mar 23 2018, 5:26 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)

Change 422452 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Removing mgmt dns for db1001

https://gerrit.wikimedia.org/r/422452

Change 422452 merged by Cmjohnson:
[operations/dns@master] Removing mgmt dns for db1001

https://gerrit.wikimedia.org/r/422452

Cmjohnson updated the task description. (Show Details)Mar 28 2018, 6:25 PM
Cmjohnson closed this task as Resolved.Apr 3 2018, 6:33 PM