Page MenuHomePhabricator
Create Task
Maniphest T95757

Phase out gallium.wikimedia.org
Closed, ResolvedPublic
Actions

Description

gallium.wikimedia.org is years old and running Precise. We are migrating the CI services it hosts to contint1001.wikimedia.org.

Contacts

  • Antoine “hashar” Musso
  • Tyler “thcipriani” Cipriani

TL;DR:

  • merge patches for puppet cleanup
  • install zuul/jenkins on contint1001 in stopped state
  • restore contint1001 from gallium backup
  • on migration window: stop CI on gallium, start on contint1001
  • switch backend in misc varnish
  • refine/tune
  • done

Proposed window

  • Tuesday Oct 25 9am mountain - 15:00 UTC - 8am PST

Migration plan is on Google Doc https://docs.google.com/document/d/1xOcXkQA9gJaLAeyA6pePUJPZmV62RFU3KapGg8LCJ_A/edit# (requires WMF Google account).

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Apr 10 2015, 8:24 PM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added a project: Continuous-Integration-Infrastructure.
hashar subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 10 2015, 8:24 PM
RobH closed subtask T95760: eqiad: (1) allocate server to migrate Zuul server to as Resolved.Apr 13 2015, 9:35 PM
Krinkle triaged this task as Medium priority.Apr 14 2015, 1:39 AM
Krinkle moved this task from Untriaged to Backlog on the Continuous-Integration-Infrastructure board.
Krinkle set Security to None.
hashar added a subtask: T95959: install/setup/deploy cobalt as replacement for gallium.Oct 6 2015, 11:45 AM
hashar changed the task status from Open to Stalled.Oct 28 2015, 2:15 PM
hashar added a comment.Dec 9 2015, 4:49 PM

So this need to happen. Precise is definitely legacy and we need to migrate straight up to Jessie. There is a bunch of challenges though since part of what is running on gallium is unpuppetized (Jenkins) and it is mixing web publishing with the Zuul scheduler.

Would need @hashar (me) to write down what is running on the machine and fill a bunch of sub tasks.

hashar added a parent task: T123525: reduce amount of remaining Ubuntu 12.04 (precise) systems in production.Jan 15 2016, 11:07 AM
hashar mentioned this in T123525: reduce amount of remaining Ubuntu 12.04 (precise) systems in production.
greg added a parent task: T124121: [keyresult] Migrate Jenkins to Jessie (gallium -> cobalt).Jan 20 2016, 12:18 AM
hashar mentioned this in T127504: doc.wikimedia.org should be running PHP 5.5+, not 5.3 -> demos etc. don't work.Feb 19 2016, 8:16 PM
Ricordisamoa subscribed.Feb 23 2016, 10:40 AM
Dzahn mentioned this in T133150: Move gallium to an internal host?.Apr 20 2016, 3:06 PM
hashar added a subtask: T133150: Move gallium to an internal host?.Apr 20 2016, 4:03 PM
thcipriani subscribed.Apr 21 2016, 3:28 PM
• mmodell edited projects, added Continuous-Integration-Infrastructure (phase-out-gallium); removed Continuous-Integration-Infrastructure.Apr 21 2016, 3:35 PM
hashar added a comment.Apr 21 2016, 3:51 PM

We have created a subproject to Continuous-Integration-Infrastructure to track and organize the various tasks: https://phabricator.wikimedia.org/project/view/1966/

hashar moved this task from Backlog to Meta tasks on the Continuous-Integration-Infrastructure (phase-out-gallium) board.Apr 21 2016, 4:04 PM
hashar added a subtask: T133300: Target architecture without gallium.wikimedia.org.Apr 25 2016, 9:23 PM
Paladox subscribed.Apr 26 2016, 12:00 PM
fgiunchedi changed the status of subtask T133150: Move gallium to an internal host? from Open to Stalled.Apr 28 2016, 9:55 AM
Paladox raised the priority of this task from Medium to Needs Triage.Jun 8 2016, 1:30 PM
Paladox added a subtask: T137265: / on gallium is read only, breaking jenkins.

I doint know what priority to change it to so changing to triage for someone to choose a priority but T137265 is the task that is migrating to the new server.

Paladox changed the task status from Stalled to Open.Jun 8 2016, 1:33 PM
Paladox triaged this task as Medium priority.
jayvdb subscribed.Jun 8 2016, 1:35 PM
hashar closed subtask T137265: / on gallium is read only, breaking jenkins as Resolved.Jun 8 2016, 7:24 PM
hashar closed subtask T95959: install/setup/deploy cobalt as replacement for gallium as Resolved.Jun 8 2016, 8:45 PM
hashar removed a subtask: T133150: Move gallium to an internal host?.Jun 10 2016, 8:26 AM
greg raised the priority of this task from Medium to High.Jun 21 2016, 9:26 PM
greg added a project: releng-201617-q1.
greg mentioned this in T124121: [keyresult] Migrate Jenkins to Jessie (gallium -> cobalt).Jul 6 2016, 9:24 PM
greg added a project: Wikimedia-Incident.Jul 27 2016, 10:41 PM
greg moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.Jul 27 2016, 10:45 PM
hashar closed subtask T133300: Target architecture without gallium.wikimedia.org as Resolved.Sep 12 2016, 3:15 PM
hashar updated the task description. (Show Details)Oct 4 2016, 3:26 PM
hashar updated the task description. (Show Details)
hashar added a comment.Oct 4 2016, 3:30 PM

@thcipriani and I have overhauled this task. The task details highlight the migration overview.

https://docs.google.com/document/d/1xOcXkQA9gJaLAeyA6pePUJPZmV62RFU3KapGg8LCJ_A/edit# has the whole detailed migration plan, specially prerequisites that should be done beforehand.

Dzahn subscribed.Oct 4 2016, 10:54 PM

https://gerrit.wikimedia.org/r/313579
https://gerrit.wikimedia.org/r/313581

merged, ran puppet on gallium, ganglia files removed

hashar added a comment.Oct 26 2016, 7:49 PM

The migration to contint1001 is scheduled for Thursday November 3rd at 9:00am PST / 16:00 UTC / 17:00 CET.

https://wikitech.wikimedia.org/w/index.php?title=Deployments&oldid=930567#Thursday.2C.C2.A0November.C2.A003

gerritbot subscribed.Oct 26 2016, 10:42 PM

Change 318216 had a related patch set uploaded (by Dzahn):
remove gallium from site.pp, installserver

https://gerrit.wikimedia.org/r/318216

gerritbot added a project: Patch-For-Review.Oct 26 2016, 10:42 PM
gerritbot added a comment.Oct 26 2016, 10:45 PM

Change 318217 had a related patch set uploaded (by Dzahn):
contint: remove gallium conditional from contint::master_dir

https://gerrit.wikimedia.org/r/318217

gerritbot added a comment.Oct 26 2016, 10:47 PM

Change 318218 had a related patch set uploaded (by Dzahn):
nodepool: switch gallium to contint1001

https://gerrit.wikimedia.org/r/318218

gerritbot added a comment.Oct 26 2016, 10:49 PM

Change 318218 abandoned by Dzahn:
nodepool: switch gallium to contint1001

Reason:
duplicate of https://gerrit.wikimedia.org/r/#/c/313599/1

https://gerrit.wikimedia.org/r/318218

gerritbot added a comment.Oct 26 2016, 10:51 PM

Change 313599 had a related patch set uploaded (by Hashar):
nodepool: point to Jenkins on contint1001

https://gerrit.wikimedia.org/r/313599

gerritbot added a comment.Oct 27 2016, 1:02 AM

Change 318245 had a related patch set uploaded (by Dzahn):
contint: remove gallium from firewall::labs

https://gerrit.wikimedia.org/r/318245

gerritbot added a comment.Oct 27 2016, 1:08 AM

Change 318246 had a related patch set uploaded (by Dzahn):
cache::misc: switch gallium to contint1001

https://gerrit.wikimedia.org/r/318246

gerritbot added a comment.Oct 27 2016, 1:10 AM

Change 318247 had a related patch set uploaded (by Dzahn):
contint: rm gallium from ferm rules in zuul::merger

https://gerrit.wikimedia.org/r/318247

gerritbot added a comment.Oct 27 2016, 1:16 AM

Change 318248 had a related patch set uploaded (by Dzahn):
deployment-prep/integration: stop downgrading sshd MAC and KEX

https://gerrit.wikimedia.org/r/318248

gerritbot added a comment.Oct 27 2016, 1:27 AM

Change 318249 had a related patch set uploaded (by Dzahn):
switch zuul CNAME from gallium to contint1001

https://gerrit.wikimedia.org/r/318249

gerritbot added a comment.Oct 27 2016, 1:28 AM

Change 318250 had a related patch set uploaded (by Dzahn):
remove gallium.wikimedia.org, keep gallium.mgmt

https://gerrit.wikimedia.org/r/318250

gerritbot added a comment.Oct 27 2016, 1:35 AM

Change 318252 had a related patch set uploaded (by Dzahn):
zuul::merger: switch gearman server to contint1001

https://gerrit.wikimedia.org/r/318252

gerritbot added a comment.Oct 27 2016, 6:10 AM

Change 318248 abandoned by Hashar:
deployment-prep/integration: stop downgrading sshd MAC and KEX

Reason:
The issue is in Jenkins itself, not Java :D T100509

https://gerrit.wikimedia.org/r/318248

Ladsgroup subscribed.Oct 28 2016, 8:40 AM
tom29739 subscribed.Oct 28 2016, 11:04 AM
Dzahn added a comment.Nov 2 2016, 12:53 AM

IPv6 for contint1001 while at it:

https://gerrit.wikimedia.org/r/#/c/316040/
https://gerrit.wikimedia.org/r/#/c/319258/

contint1001:~] $ host contint1001.wikimedia.org
contint1001.wikimedia.org has address 208.80.154.17
contint1001.wikimedia.org has IPv6 address 2620:0:861:1:208:80:154:17

gerritbot added a comment.Nov 3 2016, 12:37 PM

Change 319557 had a related patch set uploaded (by Hashar):
Announce CI maintenance

https://gerrit.wikimedia.org/r/319557

gerritbot added a comment.Nov 3 2016, 12:38 PM

Change 319557 merged by jenkins-bot:
Announce CI maintenance

https://gerrit.wikimedia.org/r/319557

gerritbot added a comment.Nov 3 2016, 4:03 PM

Change 319584 had a related patch set uploaded (by Hashar):
contint: enable zuul::server on contint1001

https://gerrit.wikimedia.org/r/319584

gerritbot added a comment.Nov 3 2016, 4:03 PM

Change 319584 merged by Dzahn:
contint: enable zuul::server on contint1001

https://gerrit.wikimedia.org/r/319584

gerritbot added a comment.Nov 3 2016, 4:08 PM

Change 318252 merged by Dzahn:
zuul::merger: switch gearman server to contint1001

https://gerrit.wikimedia.org/r/318252

gerritbot added a comment.Nov 3 2016, 4:10 PM

Change 313599 merged by Dzahn:
nodepool: point to Jenkins on contint1001

https://gerrit.wikimedia.org/r/313599

gerritbot added a comment.Nov 3 2016, 4:14 PM

Change 318246 merged by Dzahn:
cache::misc: switch gallium to contint1001

https://gerrit.wikimedia.org/r/318246

gerritbot added a comment.Nov 3 2016, 4:16 PM

Change 318247 merged by Dzahn:
contint: rm gallium from ferm rules in zuul::merger

https://gerrit.wikimedia.org/r/318247

gerritbot added a comment.Nov 3 2016, 4:20 PM

Change 318249 merged by Dzahn:
switch zuul CNAME from gallium to contint1001

https://gerrit.wikimedia.org/r/318249

Dzahn added a comment.Nov 3 2016, 4:52 PM

https://gerrit.wikimedia.org/r/#/c/319604/

gerritbot added a comment.Nov 3 2016, 5:23 PM

Change 319619 had a related patch set uploaded (by Dzahn):
integration.wm: update Apache config to 2.4

https://gerrit.wikimedia.org/r/319619

gerritbot added a comment.Nov 3 2016, 5:24 PM

Change 319619 merged by Dzahn:
integration.wm: update Apache config to 2.4

https://gerrit.wikimedia.org/r/319619

gerritbot added a comment.Nov 3 2016, 5:52 PM

Change 319627 had a related patch set uploaded (by Dzahn):
contint: fix Apache config of doc.wikimedia.org

https://gerrit.wikimedia.org/r/319627

gerritbot added a comment.Nov 3 2016, 5:55 PM

Change 319627 merged by Dzahn:
contint: fix Apache config of doc.wikimedia.org

https://gerrit.wikimedia.org/r/319627

Dzahn claimed this task.Nov 3 2016, 7:18 PM
hashar added a comment.Nov 3 2016, 7:47 PM

The services have been migrated to contint1001 successfully a couple hours ago.

gerritbot added a comment.Nov 3 2016, 7:55 PM

Change 318245 merged by Dzahn:
contint: remove gallium from firewall::labs

https://gerrit.wikimedia.org/r/318245

gerritbot added a comment.Nov 3 2016, 8:21 PM

Change 318217 merged by Dzahn:
contint: remove gallium conditional from contint::master_dir

https://gerrit.wikimedia.org/r/318217

gerritbot added a comment.Nov 7 2016, 10:08 PM

Change 318216 merged by Dzahn:
remove gallium from site.pp, installserver

https://gerrit.wikimedia.org/r/318216

Stashbot subscribed.Nov 8 2016, 9:40 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-08T21:40:37Z] <mutante> gallium, ex-CI server, shutdown -h now (the contents of your home dir have been copied to contint1001 in /home/gallium-home/) (T95757)

hashar added a comment.Nov 8 2016, 9:46 PM

Thank you @Dzahn :] It is all done from my point of view.

gerritbot added a comment.Nov 9 2016, 3:43 AM

Change 318250 merged by Dzahn:
remove gallium.wikimedia.org, keep gallium.mgmt

https://gerrit.wikimedia.org/r/318250

Stashbot added a comment.Nov 9 2016, 3:44 AM

Mentioned in SAL (#wikimedia-operations) [2016-11-09T03:44:11Z] <mutante> gallium.wikimedia.org removed from DNS (T95757)

Dzahn created subtask T150316: decom gallium (data center).Nov 9 2016, 3:50 AM
Dzahn mentioned this in T150316: decom gallium (data center).
Dzahn closed this task as Resolved.Nov 9 2016, 3:53 AM

all done on my side as well now :) created subtask for datacenter ops to take care of it (wipe disks, physical decom, racktables....recycling ..)

Stashbot added a comment.Dec 16 2016, 1:45 PM

Mentioned in SAL (#wikimedia-releng) [2016-12-16T13:45:38Z] <hashar> integration / contintcloud : remove security rules of labs projects that allowed gallium (phased out) T95757

• Cmjohnson closed subtask T150316: decom gallium (data center) as Resolved.May 3 2017, 2:08 PM
Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:50 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL