Page MenuHomePhabricator
Create Task
Maniphest T209361

Phase out Nodepool from production
Closed, ResolvedPublic
Actions

Assigned To
hashar
Authored By
hashar
Nov 13 2018, 2:22 PM
Tags
Referenced Files
None
Subscribers
Aklapper
gerritbot
hashar
Krenair
Stashbot
Tokens
"Love" token, awarded by Jdforrester-WMF."Barnstar" token, awarded by greg."Yellow Medal" token, awarded by thcipriani."Love" token, awarded by chasemp.

Description

All CI jobs have been migrated off of Nodepool (T190097) This task is to decommission Nodepool from the infrastructure.

WMCS

  • delete leftover instances
  • delete images and snapshots
  • purge custom disk images from WMCS infrastructure
  • delete contintcloud WMCS project - T209644
  • disable nodepoolmanager user from LDAP - T217064
  • clean up router/firewall rules between production and wmcs - T215173

Prod / puppet

See also server decom T209642

  • archive operations/debs/nodepool
  • remove nodepool package from apt.wikimedia.org
  • Kept around per operations
  • clean up puppet
  • drop sudo rules
  • clean up passwords/tokens from private repository done via T212230
  • remove firewall rules on contint1001/contint2001 (ferm should clean them up though)
  • drop database nodepooldb and user 'nodepool'@'10.64.16.155' on MySQL database (production-m5) - T212230

Misc

  • Delete Grafana boards
  • Purge Graphite metrics - T215172
  • Archive wiki documentation

Jenkins

Details

SubjectRepoBranchLines +/-
jenkins: stop purging nodepool agents config historyoperations/puppetproduction+0 -10
contint: remove unused classesoperations/puppetproduction+3 -233
admin: remove CI sudo rule for "nodepool"operations/puppetproduction+1 -6
cumin: remove nodepool from misc-releng groupoperations/puppetproduction+1 -1
nodepool is gone, no need to assign a clusteroperations/puppetproduction+1 -1
ci: stop monitoring zmq on Jenkinsoperations/puppetproduction+0 -12
nodepool: labtestservices2003 is not used for testingoperations/puppetproduction+0 -18
Clean JJB configintegration/configmaster+0 -106
Clean Zuul configintegration/configmaster+0 -19
Remove Nodepool / diskimage-builder materialintegration/configmaster+1 -807
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a project: Release-Engineering-Team (Kanban). · View Herald TranscriptNov 13 2018, 2:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
hashar added a subtask: T190097: Migrate all CI jobs from Nodepool, deprecate its use.Nov 13 2018, 2:23 PM
hashar updated the task description. (Show Details)Nov 13 2018, 2:29 PM
• chasemp awarded a token.Nov 13 2018, 2:30 PM
fgiunchedi mentioned this in T88997: Improve graphite failover.Nov 14 2018, 9:17 AM
hashar closed subtask T190097: Migrate all CI jobs from Nodepool, deprecate its use as Resolved.Nov 15 2018, 7:27 PM

I have migrated the last jobs still using Nodepool. Ready to phase out Nodepool :-]

gerritbot subscribed.Nov 15 2018, 7:32 PM

Change 473202 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Remove Nodepool / diskimage-builder material

https://gerrit.wikimedia.org/r/473202

gerritbot added a project: Patch-For-Review.Nov 15 2018, 7:32 PM
gerritbot added a comment.Nov 15 2018, 7:35 PM

Change 473202 merged by jenkins-bot:
[integration/config@master] Remove Nodepool / diskimage-builder material

https://gerrit.wikimedia.org/r/473202

gerritbot added a comment.Nov 15 2018, 7:40 PM

Change 473824 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Clean Zuul config

https://gerrit.wikimedia.org/r/473824

gerritbot added a comment.Nov 15 2018, 7:45 PM

Change 473827 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Clean JJB config

https://gerrit.wikimedia.org/r/473827

gerritbot added a comment.Nov 15 2018, 7:56 PM

Change 473824 merged by jenkins-bot:
[integration/config@master] Clean Zuul config

https://gerrit.wikimedia.org/r/473824

gerritbot added a comment.Nov 15 2018, 7:59 PM

Change 473827 merged by jenkins-bot:
[integration/config@master] Clean JJB config

https://gerrit.wikimedia.org/r/473827

hashar updated the task description. (Show Details)Nov 15 2018, 8:07 PM
thcipriani awarded a token.Nov 15 2018, 8:08 PM
gerritbot added a comment.Nov 15 2018, 8:12 PM

Change 473834 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] nodepool: labtestservices2003 is not used for testing

https://gerrit.wikimedia.org/r/473834

hashar removed a subtask: T190097: Migrate all CI jobs from Nodepool, deprecate its use.Nov 15 2018, 8:38 PM
hashar updated the task description. (Show Details)
hashar updated the task description. (Show Details)Nov 15 2018, 8:57 PM
gerritbot added a comment.Nov 15 2018, 9:02 PM

Change 473846 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] ci: stop monitoring zmq on Jenkins

https://gerrit.wikimedia.org/r/473846

hashar updated the task description. (Show Details)Nov 15 2018, 9:03 PM
Stashbot subscribed.Nov 15 2018, 9:05 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-15T21:05:38Z] <hashar> Stopped nodepool on labnodepool1001.eqiad.wmnet . Service is no more used. T209361 T209642

Stashbot mentioned this in T209642: Remove labnodepool1001.eqiad.wmnet.Nov 15 2018, 9:05 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-15T21:06:58Z] <hashar> Deleting Nodepool instances on contintcloud T209361

Krenair subscribed.Nov 15 2018, 9:07 PM
hashar updated the task description. (Show Details)Nov 15 2018, 9:12 PM
hashar mentioned this in T209644: Delete contintcloud WMCS project.Nov 15 2018, 9:17 PM
hashar updated the task description. (Show Details)
aborrero edited projects, added cloud-services-team (Kanban); removed cloud-services-team.Nov 16 2018, 8:57 AM
greg awarded a token.Nov 16 2018, 5:21 PM
hashar updated the task description. (Show Details)Nov 16 2018, 5:35 PM
greg moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.Nov 16 2018, 8:44 PM
gerritbot added a comment.Nov 16 2018, 10:35 PM

Change 473834 merged by Andrew Bogott:
[operations/puppet@production] nodepool: labtestservices2003 is not used for testing

https://gerrit.wikimedia.org/r/473834

Stashbot added a comment.Nov 23 2018, 11:09 AM

Mentioned in SAL (#wikimedia-releng) [2018-11-23T11:09:22Z] <hashar> Jenkins: removing plugins "Single Use Slave" and "Event Publisher (via ZMQ PUB SUB)". Were used for Nodepool | T209361

hashar updated the task description. (Show Details)Nov 23 2018, 11:09 AM
greg triaged this task as Medium priority.Nov 27 2018, 9:13 PM
gerritbot added a comment.Nov 28 2018, 4:07 PM

Change 473846 merged by Andrew Bogott:
[operations/puppet@production] ci: stop monitoring zmq on Jenkins

https://gerrit.wikimedia.org/r/473846

hashar updated the task description. (Show Details)Dec 15 2018, 9:07 AM
hashar updated the task description. (Show Details)
hashar updated the task description. (Show Details)Dec 18 2018, 4:14 PM
hashar updated the task description. (Show Details)Dec 18 2018, 4:18 PM
gerritbot added a comment.Dec 18 2018, 5:34 PM

Change 480546 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] admin: remove CI sudo rule for "nodepool"

https://gerrit.wikimedia.org/r/480546

gerritbot added a comment.Dec 18 2018, 10:30 PM

Change 480659 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] nodepool is gone, no need to assign a cluster

https://gerrit.wikimedia.org/r/480659

gerritbot added a comment.Dec 18 2018, 10:32 PM

Change 480659 merged by Andrew Bogott:
[operations/puppet@production] nodepool is gone, no need to assign a cluster

https://gerrit.wikimedia.org/r/480659

gerritbot added a comment.Dec 18 2018, 10:35 PM

Change 480663 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] cumin: remove nodepool from misc-releng group

https://gerrit.wikimedia.org/r/480663

gerritbot added a comment.Dec 18 2018, 10:48 PM

Change 480663 merged by CRusnov:
[operations/puppet@production] cumin: remove nodepool from misc-releng group

https://gerrit.wikimedia.org/r/480663

Marostegui closed subtask T212230: [DBA] remove nodepooldb on production-m5 and nodepool user as Resolved.Dec 19 2018, 7:38 AM
gerritbot added a comment.Dec 19 2018, 9:46 AM

Change 480546 merged by Muehlenhoff:
[operations/puppet@production] admin: remove CI sudo rule for "nodepool"

https://gerrit.wikimedia.org/r/480546

hashar updated the task description. (Show Details)Dec 19 2018, 9:49 AM
hashar updated the task description. (Show Details)Dec 19 2018, 10:00 AM
Jdforrester-WMF awarded a token.Dec 19 2018, 11:35 AM
gerritbot added a comment.Dec 21 2018, 7:34 PM

Change 481201 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] contint: remove unused classes

https://gerrit.wikimedia.org/r/481201

hashar moved this task from Backlog to In progress on the Continuous-Integration-Infrastructure (shipyard) board.Dec 31 2018, 3:08 PM
gerritbot added a comment.Jan 2 2019, 5:44 PM

Change 481201 merged by Dzahn:
[operations/puppet@production] contint: remove unused classes

https://gerrit.wikimedia.org/r/481201

Krinkle removed a project: Patch-For-Review.Jan 21 2019, 11:26 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:40 PM
hashar updated the task description. (Show Details)Feb 4 2019, 3:18 PM
hashar updated the task description. (Show Details)Feb 4 2019, 3:21 PM
hashar updated the task description. (Show Details)Feb 4 2019, 3:25 PM
Stashbot added a comment.Feb 4 2019, 3:25 PM

Mentioned in SAL (#wikimedia-releng) [2019-02-04T15:25:43Z] <hashar> removed Jenkins user "nodepoolmanager" as well as related authorizations | T209361

hashar updated the task description. (Show Details)Feb 4 2019, 3:30 PM
hashar updated the task description. (Show Details)Feb 4 2019, 3:33 PM
hashar moved this task from In-progress to Done (within RelEng) on the Release-Engineering-Team (Kanban) board.
hashar moved this task from Done (within RelEng) to Blocked (externally) on the Release-Engineering-Team (Kanban) board.
hashar removed projects: cloud-services-team (Kanban), SRE.

I have filled sub tasks for the other teams to act on :)

hashar changed the task status from Open to Stalled.Feb 4 2019, 3:35 PM
gerritbot added a comment.Feb 5 2019, 10:26 AM

Change 488019 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] jenkins: stop purging nodepool agents config history

https://gerrit.wikimedia.org/r/488019

gerritbot added a project: Patch-For-Review.Feb 5 2019, 10:26 AM
hashar closed subtask T215173: Clean labnodepool1001.eqiad.wmnet from firewall / router as Resolved.Feb 5 2019, 12:54 PM
hashar updated the task description. (Show Details)Feb 5 2019, 1:01 PM
fgiunchedi closed subtask T215172: Remove graphite data for nodepool as Resolved.Feb 5 2019, 2:29 PM
gerritbot added a comment.Feb 5 2019, 6:01 PM

Change 488019 merged by Dzahn:
[operations/puppet@production] jenkins: stop purging nodepool agents config history

https://gerrit.wikimedia.org/r/488019

Stashbot added a comment.Feb 5 2019, 6:17 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-05T18:17:39Z] <mutante> contint1001/contint2001 -manually deleting crontab lines unpuppetized in gerrit:488019 (T209361)

hashar updated the task description. (Show Details)Feb 5 2019, 8:03 PM

I caught up with some more cleanup. Last thing to do is to disable the nodepoolmanager user in LDAP. I am not quite sure how to do that though.

hashar updated the task description. (Show Details)Feb 25 2019, 5:52 PM
In T209361#4929205, @hashar wrote:

I caught up with some more cleanup. Last thing to do is to disable the nodepoolmanager user in LDAP. I am not quite sure how to do that though.

Filled as sub task T217064

RobH closed subtask T217064: Disable nodepoolmanager user in LDAP as Resolved.Feb 26 2019, 3:59 PM
hashar closed this task as Resolved.Feb 26 2019, 10:45 PM
hashar updated the task description. (Show Details)

All clean up tasks conducted.

The remaining one is to dispose of the hardware which is T209642.

Papaul closed subtask T209642: Remove labnodepool1001.eqiad.wmnet as Resolved.Feb 29 2020, 1:06 AM
Maintenance_bot removed a project: Patch-For-Review.Feb 29 2020, 1:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL