Page MenuHomePhabricator

Decommission db1056
Closed, ResolvedPublic

Description

db1056 has migrated its functionality to other host, decom or reuse its parts (but it is too old to be used as a reliable db host).

  • - all system services confirmed offline from production use: Removed from mediawiki-config: https://gerrit.wikimedia.org/r/#/c/430599/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.):
  • Host set to spare: https://gerrit.wikimedia.org/r/430884

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal) asw2-c-eqiad:ge-3/0/11
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key

END NON-INTERRUPPTABLE STEPS

  • - mark disk #10 as non-usable, it has smart errors. - try to wipe, if it won't degauss it.
  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Event Timeline

jcrespo triaged this task as Medium priority.May 3 2018, 2:03 PM
jcrespo created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 3 2018, 2:03 PM
jcrespo claimed this task.May 3 2018, 2:05 PM
jcrespo moved this task from Triage to In progress on the DBA board.

Change 430596 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] maridb: Depool db1056 for decommissioning

https://gerrit.wikimedia.org/r/430596

Change 430596 merged by jenkins-bot:
[operations/mediawiki-config@master] maridb: Depool db1056 for decommissioning

https://gerrit.wikimedia.org/r/430596

Change 430599 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Remove mediawiki references to db1056

https://gerrit.wikimedia.org/r/430599

Change 430599 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Remove mediawiki references to db1056

https://gerrit.wikimedia.org/r/430599

Change 430884 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Decommission db1056

https://gerrit.wikimedia.org/r/430884

Change 430885 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software@master] dbhosts: Remove db1056 for decom

https://gerrit.wikimedia.org/r/430885

Change 430885 merged by Jcrespo:
[operations/software@master] dbhosts: Remove db1056 for decom

https://gerrit.wikimedia.org/r/430885

Change 430884 merged by Jcrespo:
[operations/puppet@production] mariadb: Decommission db1056

https://gerrit.wikimedia.org/r/430884

jcrespo updated the task description. (Show Details)

We will wait until Monday and then send it to dc-ops.

jcrespo reassigned this task from jcrespo to RobH.May 8 2018, 6:46 AM
jcrespo added a project: decommission-hardware.
jcrespo updated the task description. (Show Details)
jcrespo moved this task from In progress to Done on the DBA board.

This is ready for dc-ops. Robh you may want to update the template here? Is there a "last version" somewhere?

jcrespo lowered the priority of this task from Medium to Low.May 8 2018, 6:47 AM
Restricted Application added a project: Operations. · View Herald TranscriptJun 8 2018, 2:14 PM
RobH updated the task description. (Show Details)Jun 8 2018, 3:11 PM
Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Jun 11 2018, 3:39 PM
Marostegui updated the task description. (Show Details)Jun 12 2018, 10:35 AM
jcrespo raised the priority of this task from Low to Medium.Jun 13 2018, 9:06 AM

Not low anymore, based on my proposal of 1 server movement.

Vvjjkkii renamed this task from Decommission db1056 to rpdaaaaaaa.Jul 1 2018, 1:12 AM
Vvjjkkii removed RobH as the assignee of this task.
Vvjjkkii raised the priority of this task from Medium to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii edited subscribers, added: RobH; removed: gerritbot, Aklapper.
Marostegui renamed this task from rpdaaaaaaa to Decommission db1056.Jul 1 2018, 6:25 PM
Marostegui assigned this task to Cmjohnson.
Marostegui lowered the priority of this task from High to Medium.
Marostegui updated the task description. (Show Details)
Marostegui added a subscriber: Aklapper.
RobH removed Cmjohnson as the assignee of this task.Jul 18 2018, 6:02 PM
RobH updated the task description. (Show Details)Jul 20 2018, 6:24 PM
RobH updated the task description. (Show Details)Jul 20 2018, 6:33 PM

Change 447094 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom of db1056

https://gerrit.wikimedia.org/r/447094

Change 447095 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom db1056 production dns entries

https://gerrit.wikimedia.org/r/447095

Change 447094 merged by RobH:
[operations/puppet@production] decom of db1056

https://gerrit.wikimedia.org/r/447094

Change 447095 merged by RobH:
[operations/dns@master] decom db1056 production dns entries

https://gerrit.wikimedia.org/r/447095

RobH assigned this task to Cmjohnson.Jul 20 2018, 6:39 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
RobH moved this task from Backlog to pending onsite steps (eqiad) on the decommission-hardware board.
Cmjohnson closed this task as Resolved.Aug 7 2018, 5:06 PM
Cmjohnson updated the task description. (Show Details)