Page MenuHomePhabricator

Decommission db1016
Closed, ResolvedPublic

Description

db1016 has been failed over db1063.
Wait a few days and proceed to decommission it
db1016 data has been copied over to db1065

Decommission Checklist

  • - all system services confirmed offline from production use - should be done by DBA team
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration - should be done by DBA team
  • - any service group puppet/heira/dsh config removed - should be done by DBA team
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.) - should be done by DBA team: https://gerrit.wikimedia.org/r/#/c/420979/4/manifests/site.pp (pending a puppet run - puppet currently disabled on that host)

START NON-INTERRUPPTABLE STEPS - please assign to @RobH for the non-interrupt steps

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

Related Gerrit Patches:
operations/dns : masterRemoving mgmt dns from db1016
operations/software : masterm1.hosts: Remove db1016
operations/puppet : productionmariadb: Move db1065 to misc

Related Objects

Event Timeline

Marostegui triaged this task as Medium priority.Mar 20 2018, 4:22 PM
Marostegui created this task.
Marostegui moved this task from Triage to In progress on the DBA board.

Change 420979 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1065 to misc

https://gerrit.wikimedia.org/r/420979

Change 420979 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1065 to misc

https://gerrit.wikimedia.org/r/420979

Marostegui updated the task description. (Show Details)Mar 21 2018, 10:34 AM
Marostegui updated the task description. (Show Details)

db1016 data has been copied over to db1065

Marostegui updated the task description. (Show Details)Mar 21 2018, 11:43 AM

db1016 removed from tendril (only 1 host, ofc)

Change 421214 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] m1.hosts: Remove db1016

https://gerrit.wikimedia.org/r/421214

Marostegui updated the task description. (Show Details)Mar 22 2018, 6:20 AM

Change 421214 merged by jenkins-bot:
[operations/software@master] m1.hosts: Remove db1016

https://gerrit.wikimedia.org/r/421214

Marostegui assigned this task to RobH.Mar 22 2018, 6:21 AM
Marostegui moved this task from In progress to Done on the DBA board.

This host is now ready for DC Ops steps.
Assigning it to @RobH

Restricted Application added a project: Operations. · View Herald TranscriptMar 22 2018, 6:21 AM
RobH reassigned this task from RobH to Cmjohnson.Mar 23 2018, 5:26 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)

Change 422459 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Removing mgmt dns from db1016

https://gerrit.wikimedia.org/r/422459

Change 422459 merged by Cmjohnson:
[operations/dns@master] Removing mgmt dns from db1016

https://gerrit.wikimedia.org/r/422459

Cmjohnson closed this task as Resolved.Apr 3 2018, 6:36 PM
Cmjohnson updated the task description. (Show Details)
  • ge-2/0/0 {
  • description db1001;
  • disable;
  • }
  • ge-2/0/10 {
  • description db1011;
  • disable;
  • }
  • ge-2/0/15 {
  • description db1016;
  • disable;
  • }