Page MenuHomePhabricator

Decommission db1055
Closed, ResolvedPublic

Description

db1055 has been cloned away and can be decommissioned

  • Clone db1064 from db1055

Decommission Checklist

  • - all system services confirmed offline from production use - should be done by DBA team
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration - should be done by DBA team
  • - any service group puppet/heira/dsh config removed - should be done by DBA team
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.) - should be done by DBA team:

START NON-INTERRUPPTABLE STEPS - please assign to @RobH for the non-interrupt steps

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal) asw-c-eqiad:ge-2/0/13
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Event Timeline

Marostegui triaged this task as Medium priority.May 8 2018, 6:13 AM
Marostegui created this task.

Change 431722 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool db1055

https://gerrit.wikimedia.org/r/431722

Change 431722 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool db1055

https://gerrit.wikimedia.org/r/431722

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['db1064.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805081013_jynus_12330.log.

Change 431724 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Move db1064 from s4 to x1

https://gerrit.wikimedia.org/r/431724

Change 431724 merged by Jcrespo:
[operations/puppet@production] mariadb: Move db1064 from s4 to x1

https://gerrit.wikimedia.org/r/431724

Change 431734 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Pool db1064 into x1 with low load

https://gerrit.wikimedia.org/r/431734

Change 431735 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Remove references to db1055, to be decom

https://gerrit.wikimedia.org/r/431735

Change 431734 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Pool db1064 into x1 with low load

https://gerrit.wikimedia.org/r/431734

Change 431740 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariab: Fully pool db1064

https://gerrit.wikimedia.org/r/431740

Change 431735 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Remove references to db1055, to be decom

https://gerrit.wikimedia.org/r/431735

Change 431747 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software@master] dbhosts: Promote db1069 as master, remove db1055

https://gerrit.wikimedia.org/r/431747

Change 431747 merged by Jcrespo:
[operations/software@master] dbhosts: Remove db1055

https://gerrit.wikimedia.org/r/431747

Change 431748 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Set db1055 as spare before decommission

https://gerrit.wikimedia.org/r/431748

Change 431748 merged by Jcrespo:
[operations/puppet@production] mariadb: Set db1055 as spare before decommission

https://gerrit.wikimedia.org/r/431748

This is ready to be decommed, just in case we will wait a few days before sending it to DC-Ops

Change 431740 merged by jenkins-bot:
[operations/mediawiki-config@master] mariab: Fully pool db1064

https://gerrit.wikimedia.org/r/431740

jcrespo moved this task from In progress to Done on the DBA board.
jcrespo edited projects, added decommission-hardware, ops-eqiad; removed Patch-For-Review.
jcrespo subscribed.

@RobH this can now proceed as usual, db1055 is a spare with notifications disabled. Please note that in case of reusing its parts, megaraid,1,megaraid,10,megaraid,4 disks are degraded and should not be reused.

jcrespo lowered the priority of this task from Medium to Low.May 10 2018, 5:14 PM
jcrespo updated the task description. (Show Details)
Vvjjkkii renamed this task from Decommission db1055 to 5edaaaaaaa.Jul 1 2018, 1:11 AM
Vvjjkkii removed RobH as the assignee of this task.
Vvjjkkii raised the priority of this task from Low to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: gerritbot, Aklapper.
Marostegui renamed this task from 5edaaaaaaa to Decommission db1055.Jul 1 2018, 6:32 PM
Marostegui assigned this task to RobH.
Marostegui lowered the priority of this task from High to Medium.
Marostegui updated the task description. (Show Details)
CommunityTechBot lowered the priority of this task from Medium to Low.Jul 5 2018, 6:38 PM
CommunityTechBot updated the task description. (Show Details)

Change 447090 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom db1055 production dns entries

https://gerrit.wikimedia.org/r/447090

Change 447090 merged by RobH:
[operations/dns@master] decom db1055 production dns entries

https://gerrit.wikimedia.org/r/447090

Change 447091 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom of db1055

https://gerrit.wikimedia.org/r/447091

Change 447091 merged by RobH:
[operations/puppet@production] decom of db1055

https://gerrit.wikimedia.org/r/447091

RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
RobH moved this task from Backlog to pending onsite steps (eqiad) on the decommission-hardware board.
Cmjohnson updated the task description. (Show Details)