Page MenuHomePhabricator

Decommission db1016
Closed, ResolvedPublic

Description

db1016 has been failed over db1063.
Wait a few days and proceed to decommission it
db1016 data has been copied over to db1065

Decommission Checklist

  • - all system services confirmed offline from production use - should be done by DBA team
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration - should be done by DBA team
  • - any service group puppet/heira/dsh config removed - should be done by DBA team
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.) - should be done by DBA team: https://gerrit.wikimedia.org/r/#/c/420979/4/manifests/site.pp (pending a puppet run - puppet currently disabled on that host)

START NON-INTERRUPPTABLE STEPS - please assign to @RobH for the non-interrupt steps

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone
StalledNone
OpenNone
Resolvedjcrespo
OpenNone
OpenNone
OpenNone
ResolvedNone
Resolvedjcrespo
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
Resolvedjcrespo
ResolvedMarostegui
ResolvedRobH
ResolvedAndrew
ResolvedCmjohnson
Resolvedjcrespo
ResolvedCmjohnson
ResolvedCmjohnson
Resolvedjcrespo
ResolvedCmjohnson

Event Timeline

Marostegui triaged this task as Medium priority.Mar 20 2018, 4:22 PM
Marostegui created this task.
Marostegui moved this task from Triage to In progress on the DBA board.

Change 420979 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1065 to misc

https://gerrit.wikimedia.org/r/420979

Change 420979 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1065 to misc

https://gerrit.wikimedia.org/r/420979

Marostegui updated the task description. (Show Details)Mar 21 2018, 10:34 AM
Marostegui updated the task description. (Show Details)

db1016 data has been copied over to db1065

Marostegui updated the task description. (Show Details)Mar 21 2018, 11:43 AM

db1016 removed from tendril (only 1 host, ofc)

Change 421214 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] m1.hosts: Remove db1016

https://gerrit.wikimedia.org/r/421214

Marostegui updated the task description. (Show Details)Mar 22 2018, 6:20 AM

Change 421214 merged by jenkins-bot:
[operations/software@master] m1.hosts: Remove db1016

https://gerrit.wikimedia.org/r/421214

Marostegui assigned this task to RobH.Mar 22 2018, 6:21 AM
Marostegui moved this task from In progress to Done on the DBA board.

This host is now ready for DC Ops steps.
Assigning it to @RobH

Restricted Application added a project: Operations. · View Herald TranscriptMar 22 2018, 6:21 AM
RobH reassigned this task from RobH to Cmjohnson.Mar 23 2018, 5:26 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)

Change 422459 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Removing mgmt dns from db1016

https://gerrit.wikimedia.org/r/422459

Change 422459 merged by Cmjohnson:
[operations/dns@master] Removing mgmt dns from db1016

https://gerrit.wikimedia.org/r/422459

Cmjohnson closed this task as Resolved.Apr 3 2018, 6:36 PM
Cmjohnson updated the task description. (Show Details)
  • ge-2/0/0 {
  • description db1001;
  • disable;
  • }
  • ge-2/0/10 {
  • description db1011;
  • disable;
  • }
  • ge-2/0/15 {
  • description db1016;
  • disable;
  • }