Page MenuHomePhabricator
Create Task
Maniphest T209738

decom einsteinium
Closed, ResolvedPublic
Actions

Description

einsteinium used to be the active Icinga server but has now been replaced by icinga1001. start to decom it (next week)

https://netbox.wikimedia.org/dcim/devices/1592/

This task will track the decommission-hardware of server einsteinium.wikimedia.org

The first 5 steps should be completed by the service owner that is returning the server to DC-ops (for reclaim to spare or decommissioning, dependent on server configuration and age.)

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system)
  • - remove einsteinium IP from the AQL whitelist
  • - unassign any owner from this task, check off completed steps.

Steps for DC-Ops:
The following steps cannot be interrupted, as it will leave the system in an unfinished state.
Start non-interrupt steps:

  • - disable puppet on host
  • - power down host
  • - update status in netbox (inventory for decom, planned for spare)
  • - disable switch port
  • - switch port assignment noted on this task (for later removal) - asw2-d-eqiad:ge-3/0/7
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate (handled by wmf-decommission-host)
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key (handled by wmf-decommission-host)

End non-interrupt steps.

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox status with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

ProjectBranchLines +/-Subject
operations/dnsmaster+0 -4remove mgmt entries for einsteinium
operations/puppetproduction+0 -6decom einsteinium
operations/dnsmaster+1 -4decom einsteinium production dns entries
operations/puppetproduction+2 -9decom einsteinium remove from netboot and DHCP
operations/puppetproduction+113 -230icinga: remove jessie support
operations/puppetproduction+1 -5profile::icinga: stop using mysql module, rm jessie support
operations/dnsmaster+0 -1remove icinga-old.wikimedia.org
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedDzahnT202782 upgrade icinga server to stretch and replace einsteinium
 ResolvedJclark-ctrT209738 decom einsteinium

Event Timeline

Dzahn created this task.Nov 16 2018, 11:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2018, 11:54 PM
Dzahn added a parent task: T202782: upgrade icinga server to stretch and replace einsteinium.Nov 16 2018, 11:55 PM
Dzahn changed the task status from Open to Stalled.Nov 16 2018, 11:57 PM
Dzahn triaged this task as Medium priority.
Dzahn updated the task description. (Show Details)
Dzahn added projects: SRE, decommission-hardware, Icinga, observability.

changed netbox status from Active to Staged

https://netbox.wikimedia.org/dcim/devices/1592/

Dzahn claimed this task.Nov 17 2018, 12:00 AM
Dzahn updated the task description. (Show Details)
Dzahn mentioned this in T202782: upgrade icinga server to stretch and replace einsteinium.
gerritbot added a subscriber: gerritbot.Nov 17 2018, 12:03 AM

Change 473278 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: remove einsteinium as an alerting_host

https://gerrit.wikimedia.org/r/473278

gerritbot added a project: Patch-For-Review.Nov 17 2018, 12:03 AM

Change 473276 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: remove jessie support

https://gerrit.wikimedia.org/r/473276

gerritbot added a comment.Nov 17 2018, 12:12 AM

Change 474390 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] decom einsteinium remove from netboot and DHCP

https://gerrit.wikimedia.org/r/474390

gerritbot added a comment.Nov 17 2018, 12:36 AM

Change 474392 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] remove icinga-old.wikimedia.org

https://gerrit.wikimedia.org/r/474392

Dzahn moved this task from Inbox to In progress on the observability board.Nov 26 2018, 4:01 PM
gerritbot added a comment.Nov 26 2018, 6:02 PM

Change 474392 merged by Dzahn:
[operations/dns@master] remove icinga-old.wikimedia.org

https://gerrit.wikimedia.org/r/474392

Dzahn mentioned this in T207050: Migrate most standard public TLS certificates to CertCentral issuance.Nov 26 2018, 6:04 PM
Dzahn added a comment.Nov 26 2018, 6:24 PM
  • removed alerting_host role from einsteinium
  • removed einsteinium from network::constants
  • removed from whitelisted hosts on lists server
  • removed from nagios-nrpe-server (NRPE) allowed hosts
  • removed as "icinga partner" in Hiera

https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/473278/

Dzahn updated the task description. (Show Details)Nov 26 2018, 6:30 PM
Stashbot mentioned this in T208824: rename tegmen to icinga2001 and reinstall it with stretch.Nov 26 2018, 6:46 PM
Stashbot added a subscriber: Stashbot.

Mentioned in SAL (#wikimedia-operations) [2018-11-26T18:46:54Z] <mutante> removed allowed sender addresses from AQL (mail2SMS gateway) portal: @einsteinium @tegmen addresses T208824 T209738

Stashbot added a comment.Nov 26 2018, 6:50 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-26T18:50:46Z] <mutante> stopping icinga service on einsteinium, is a role(spare) now T209738

gerritbot added a comment.Nov 26 2018, 9:43 PM

Change 475876 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] profile::icinga: stop using mysql module, rm jessie support

https://gerrit.wikimedia.org/r/475876

gerritbot added a comment.Nov 26 2018, 9:50 PM

Change 475876 merged by Dzahn:
[operations/puppet@production] profile::icinga: stop using mysql module, rm jessie support

https://gerrit.wikimedia.org/r/475876

gerritbot added a comment.Nov 27 2018, 12:44 AM

Change 473276 merged by Dzahn:
[operations/puppet@production] icinga: remove jessie support

https://gerrit.wikimedia.org/r/473276

gerritbot added a comment.Nov 27 2018, 12:55 AM

Change 474390 merged by Dzahn:
[operations/puppet@production] decom einsteinium remove from netboot and DHCP

https://gerrit.wikimedia.org/r/474390

Dzahn changed the task status from Stalled to Open.Nov 27 2018, 12:58 AM
Dzahn removed Dzahn as the assignee of this task.
Dzahn removed a project: Patch-For-Review.
Dzahn updated the task description. (Show Details)
GTirloni added a subscriber: GTirloni.Nov 27 2018, 1:28 PM

Just a heads up that einsteinium is still running Icinga and contacting servers:

Nov 27 13:25:11 labstore1004 nrpe[18896]: Host 208.80.155.119 is not allowed to talk to us!
Nov 27 13:25:24 labstore1004 nrpe[18984]: Host 208.80.155.119 is not allowed to talk to us!
Nov 27 13:25:24 labstore1004 nrpe[18986]: Host 208.80.155.119 is not allowed to talk to us!
Stashbot added a comment.Nov 27 2018, 3:35 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-27T15:35:17Z] <mutante> einsteinium - stopped icinga, stopped nsca, stopped rsyncd, killall -u icinga, killall -u nagios ... T209738

Dzahn added a comment.Nov 27 2018, 3:37 PM

@GTirloni Thanks! I was pretty sure i had already told it to stop the service.. but it wouldn't listen, i had to insist and killed everything ^

Stashbot added a comment.Nov 27 2018, 4:17 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-27T16:17:14Z] <mutante> einsteinium - apt-get remove --purge icinga nsca; apt-get autoremove ; apt-get remove --purge icinga-doc icinga-common icinga-cgi-bin icinga-cgi; apt-get remove --purge monitoring-plugin* ; rm /etc/rsync.d/frag-icinga* T209738

fgiunchedi moved this task from In progress to Externally blocked on the observability board.Dec 10 2018, 4:23 PM
RobH claimed this task.Dec 14 2018, 10:16 PM
RobH removed projects: observability, Icinga.
RobH moved this task from Backlog to Ready for Decommission on the decommission-hardware board.Dec 14 2018, 10:22 PM
gerritbot added a comment.Dec 14 2018, 10:24 PM

Change 479857 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom einsteinium production dns entries

https://gerrit.wikimedia.org/r/479857

gerritbot added a project: Patch-For-Review.Dec 14 2018, 10:24 PM

Change 479858 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom einsteinium

https://gerrit.wikimedia.org/r/479858

ops-monitoring-bot added a subscriber: ops-monitoring-bot.Dec 14 2018, 10:27 PM

wmf-decommission-host was executed by robh for einsteinium.wikimedia.org and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
RobH reassigned this task from RobH to Cmjohnson.Dec 14 2018, 10:28 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
RobH updated the task description. (Show Details)
RobH added a project: ops-eqiad.
RobH moved this task from Ready for Decommission to pending onsite steps (eqiad) on the decommission-hardware board.
RobH added a subscriber: RobH.

ready for disk wipe and remainder of steps

RobH moved this task from Backlog to Decommission on the ops-eqiad board.Dec 14 2018, 10:31 PM
gerritbot added a comment.Dec 14 2018, 10:42 PM

Change 479857 merged by RobH:
[operations/dns@master] decom einsteinium production dns entries

https://gerrit.wikimedia.org/r/479857

gerritbot added a comment.Dec 14 2018, 10:42 PM

Change 479858 merged by RobH:
[operations/puppet@production] decom einsteinium

https://gerrit.wikimedia.org/r/479858

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:41 PM
GTirloni removed a subscriber: GTirloni.Mar 21 2019, 9:06 PM
Jclark-ctr claimed this task.Oct 15 2019, 9:00 PM
Jclark-ctr added a subscriber: Cmjohnson.
Jclark-ctr updated the task description. (Show Details)Nov 1 2019, 10:38 PM
Papaul added a subscriber: Papaul.Nov 8 2019, 5:43 PM
papaul@asw2-d-eqiad# show | compare 
[edit interfaces]
-   ge-3/0/7 {
-       description einsteinium;
-       enable;
-   }
Papaul updated the task description. (Show Details)Nov 8 2019, 5:44 PM
gerritbot added a comment.Nov 8 2019, 5:47 PM

Change 549898 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] remove mgmt entries for einsteinium

https://gerrit.wikimedia.org/r/549898

gerritbot added a project: Patch-For-Review.Nov 8 2019, 5:47 PM
gerritbot added a comment.Nov 8 2019, 5:50 PM

Change 549898 merged by Dzahn:
[operations/dns@master] remove mgmt entries for einsteinium

https://gerrit.wikimedia.org/r/549898

Dzahn updated the task description. (Show Details)Nov 8 2019, 5:51 PM
Dzahn removed a project: Patch-For-Review.
Papaul closed this task as Resolved.Nov 8 2019, 7:45 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL