Page MenuHomePhabricator
Create Task
Maniphest T209642

Remove labnodepool1001.eqiad.wmnet
Closed, ResolvedPublic
Actions

Description

The Nodepool service is being phased out. It is running on labnodepool1001.eqiad.wmnet which is in the WMCS support LAN. There are firewall rules between production (contint1001 / contint2001) and the WMCS network.

The service can be dropped at anytime, it is no more being used.

START NON-INTERRUPPTABLE STEPS

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox status with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Remove Hiera file obsoleted by nodepool removalMuehlenhoffoperations/puppetproduction+0 -5
decom labnodepool1001 prod dnsRobHoperations/dnsmaster+1 -2
Clean up remaining nodepool/labnodepool1001 refsAndrew Bogottoperations/puppetproduction+1 -31
Make labnodepool1001.eqiad.wmnet a spare systemHasharoperations/puppetproduction+2 -626
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedhasharT209361 Phase out Nodepool from production
 ResolvedJclark-ctrT209642 Remove labnodepool1001.eqiad.wmnet

Event Timeline

hashar created this task.Nov 15 2018, 8:42 PM
gerritbot added a comment.Nov 15 2018, 8:50 PM

Change 473838 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] Make labnodepool1001.eqiad.wmnet a spare system

https://gerrit.wikimedia.org/r/473838

gerritbot added a project: Patch-For-Review.Nov 15 2018, 8:50 PM
hashar updated the task description. (Show Details)Nov 15 2018, 8:56 PM
hashar mentioned this in T209361: Phase out Nodepool from production.
Stashbot subscribed.Nov 15 2018, 9:05 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-15T21:05:38Z] <hashar> Stopped nodepool on labnodepool1001.eqiad.wmnet . Service is no more used. T209361 T209642

hashar updated the task description. (Show Details)Nov 15 2018, 9:28 PM
hashar updated the task description. (Show Details)
hashar removed hashar as the assignee of this task.EditedNov 15 2018, 9:31 PM

CI no more relies on the service that is hosted on labnodepool1001.eqiad.wmnet.

I have manually stopped the nodepool service on the host and disabled the two Icinga checks related to it.

The system can be wiped entirely, Release-Engineering-Team has no need for backups (unless ops need it for security/forensic or whatever reasons).

cloud-services-team might be interested in the machine since it is in the labs support network (to provide a service to WMCS instances).

• aborrero edited projects, added: cloud-services-team (Kanban); removed: cloud-services-team.Nov 16 2018, 9:00 AM
• aborrero subscribed.
In T209642#4752043, @hashar wrote:

cloud-services-team might be interested in the machine since it is in the labs support network (to provide a service to WMCS instances).

We could discuss in our team meeting. Do we have info on specs and expiration dates for the HW?

• aborrero moved this task from Inbox to Needs discussion on the cloud-services-team (Kanban) board.Nov 16 2018, 9:00 AM
MoritzMuehlenhoff added a project: decommission-hardware.Nov 16 2018, 9:06 AM
ArielGlenn triaged this task as Medium priority.Nov 16 2018, 11:45 AM
Dzahn subscribed.Nov 16 2018, 2:33 PM
In T209642#4753118, @aborrero wrote:

Do we have info on specs and expiration dates for the HW?

https://racktables.wikimedia.org/index.php?page=object&tab=default&object_id=1206

HW warranty expiration: 2014-01-27

HW type: Dell PowerEdge R610

faidon added subscribers: mark, faidon.Nov 16 2018, 2:39 PM

This specific HW is /very/ old and is already overdue for decomissioning (by 3 years no less).

But more generally, (re)allocation of hardware does not work like that. If there are any needs (budgeted or unbudgeted) feel free to submit a hardware-requests task, or in case of an odd ask, to reach out to me or @mark :)

greg moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.Nov 19 2018, 11:26 PM
gerritbot added a comment.Nov 26 2018, 10:58 PM

Change 473838 merged by Andrew Bogott:
[operations/puppet@production] Make labnodepool1001.eqiad.wmnet a spare system

https://gerrit.wikimedia.org/r/473838

Andrew claimed this task.Nov 27 2018, 4:41 PM
Andrew moved this task from Needs discussion to Doing on the cloud-services-team (Kanban) board.
Dzahn unsubscribed.Nov 27 2018, 4:49 PM
Andrew reassigned this task from Andrew to RobH.Nov 27 2018, 5:40 PM
Andrew edited projects, added: ops-eqiad, DC-Ops; removed: cloud-services-team (Kanban), Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard).
Andrew subscribed.
RobH moved this task from Backlog to Decommission on the ops-eqiad board.Dec 12 2018, 11:34 PM
RobH moved this task from Backlog to Ready for Decommission on the decommission-hardware board.Dec 13 2018, 6:24 PM
RobH added a comment.Dec 14 2018, 9:59 PM

labnodepool1001 asw2-b-eqiad ge-3/0/18

ops-monitoring-bot subscribed.Dec 14 2018, 10:02 PM

wmf-decommission-host was executed by robh for labnodepool1001.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
RobH removed a project: Patch-For-Review.Dec 14 2018, 10:07 PM
RobH updated the task description. (Show Details)
RobH updated the task description. (Show Details)
gerritbot added a comment.Dec 14 2018, 10:10 PM

Change 479855 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Clean up remaining nodepool/labnodepool1001 refs

https://gerrit.wikimedia.org/r/479855

gerritbot added a project: Patch-For-Review.Dec 14 2018, 10:10 PM

Change 479855 merged by Andrew Bogott:
[operations/puppet@production] Clean up remaining nodepool/labnodepool1001 refs

https://gerrit.wikimedia.org/r/479855

gerritbot added a comment.Dec 14 2018, 10:19 PM

Change 479856 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom labnodepool1001 prod dns

https://gerrit.wikimedia.org/r/479856

RobH updated the task description. (Show Details)Dec 14 2018, 10:20 PM
gerritbot added a comment.Dec 14 2018, 10:20 PM

Change 479856 merged by RobH:
[operations/dns@master] decom labnodepool1001 prod dns

https://gerrit.wikimedia.org/r/479856

RobH reassigned this task from RobH to • Cmjohnson.Dec 14 2018, 10:21 PM
RobH updated the task description. (Show Details)
RobH removed a project: Patch-For-Review.
RobH moved this task from Ready for Decommission to pending onsite steps (eqiad) on the decommission-hardware board.
RobH subscribed.

This is ready for disk wipe and remainder of steps to decom the system.

gerritbot added a comment.Dec 19 2018, 9:59 AM

Change 480713 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove Hiera file obsoleted by nodepool removal

https://gerrit.wikimedia.org/r/480713

gerritbot added a project: Patch-For-Review.Dec 19 2018, 9:59 AM
gerritbot added a comment.Dec 19 2018, 10:07 AM

Change 480713 merged by Muehlenhoff:
[operations/puppet@production] Remove Hiera file obsoleted by nodepool removal

https://gerrit.wikimedia.org/r/480713

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:41 PM
• aborrero added a comment.Jan 30 2019, 4:17 PM

There is a netbox entry for this host: https://netbox.wikimedia.org/dcim/devices/1638/ CC T214499

We may want to delete? I'm not sure about the policy.

Maintenance_bot removed a project: Patch-For-Review.May 22 2019, 3:46 PM
Jclark-ctr claimed this task.Oct 15 2019, 9:02 PM
Jclark-ctr added a subscriber: • Cmjohnson.
Jclark-ctr updated the task description. (Show Details)Nov 28 2019, 12:07 AM
Papaul subscribed.Feb 29 2020, 12:35 AM
papaul@asw2-b-eqiad# show | compare
[edit interfaces]
-   ge-3/0/18 {
-       description labnodepool1001;
-   }
Papaul updated the task description. (Show Details)Feb 29 2020, 12:35 AM
RobH unsubscribed.Feb 29 2020, 12:52 AM
Papaul closed this task as Resolved.Feb 29 2020, 1:05 AM
Papaul updated the task description. (Show Details)

This is done

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits