Page MenuHomePhabricator

Move all Wikimedia CI (WMCS integration project) instances from jessie to stretch
Closed, ResolvedPublic

Description

T94916: Re-create ci slaves (April 2015) || T252071: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye

The end of life of Debian Jessie is approaching in 2020 and we need to move to Debian Buster (or Stretch) before that date.

All instances in the integration project need to upgrade as soon as possible. Instances not upgraded by 2019-12-31 may be subject to deletion unless prior arrangements for an extended deadline has been approved by the Cloud VPS administration team.

Remaining Debian Jessie instances (live report):


Listed administrator is Release-Engineering-Team

See also:

More info on current project instances is available via openstack browser

Details

Related Objects

Event Timeline

StrikerBot created this task.

I could do puppetmaster and cumin but don't know anything about the other instances.

Edit: Actually I checked the admin list, apparently I am only a member here so can't really.

The Cloud Services team would like to have Debian Jessie systems replaced with Debian Buster (or Stretch if necessary) before 2019-12-31. Please do respond of this task with comments if you know you will not be able to meet that target date or have additional questions about what to do or how to do it. Ideally you will create new instances in your Cloud VPS project, test them, and then migrate any final state data to the new instances before deleting the old Jessie instances. If you need more quota space in your project to create new instances in parallel with your existing instances please create a quota request task describing the increase you need and referencing this deprecation task.

Release-Engineering-Team talked about this at our team meeting this morning. The machines that are still pending migration have more complications than we can resolve by the end of the year.

We want to make this a project for Release-Engineering-Team for Q3.

Assigning to @hashar as the owner of our CI infra to break this task down a bit.

We still need some Jessie instances. Notably due to a long tail of Jenkins jobs that still have to be migrated and a CPU performance regression I have yet to reproduce using VMs on my local machine.

The breakdown of migrations is in the Continuous-Integration-Infrastructure subproject #phase-out-jessie

hashar removed hashar as the assignee of this task.Mar 5 2020, 2:10 PM

Mentioned in SAL (#wikimedia-releng) [2020-03-05T23:11:47Z] <Krenair> T236576 Shut down integration-puppetmaster01 in favour of integration-puppetmaster-02. /root and /home archives can be found in /root on the new system. All integration instances moved to use the new master. Old master to be deleted on 12th March in the absence of objections

I noticed this got unassigned around the time I picked up the puppetmaster work and began to look at cumin (which it turns out is blocked by part of T245114). I've completed the puppetmaster bit and will do cumin when possible. The rest still needs assignment or subtasks completed.

hashar updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-releng) [2020-04-29T16:03:01Z] <James_F> Removed integration-slave-jessie-1002 and integration-slave-jessie-1004 from Jenkins for T236576

Mentioned in SAL (#wikimedia-releng) [2020-04-29T16:03:51Z] <James_F> Shut down integration-slave-jessie-1002 and integration-slave-jessie-1004 in Horizon for T236576

Mentioned in SAL (#wikimedia-releng) [2020-05-01T18:41:50Z] <James_F> Deleting integration-slave-jessie-1002 and integration-slave-jessie-1004 instances from WMCS for T236576

Is this task still valid and should stay open? Asking as the Due Date on this task says December 2019. If this task is done, please resolve it (via Add Action...Change Status in the dropdown menu). If it is not done, please remove or reset the Due Date (via Edit Task). Thanks!

@Aklapper: It's still valid and should stay open, the due date is correctly set in the past. It's not clear to me whether December 2019 is accurate or if it should actually be some time in April 2020.
Edit: The same probably goes for most of the rest of the jessie deprecation tasks you just left this comment on.

Jdforrester-WMF renamed this task from "integration" Cloud VPS project jessie deprecation to Move all Wikimedia CI (WMCS integration project) instances from jessie to stretch.May 6 2020, 8:48 PM
Jdforrester-WMF updated the task description. (Show Details)

Re-scoping to just "get rid of jessie" and moving the cumin/buster blocker to T252071.

Mentioned in SAL (#wikimedia-releng) [2020-05-06T20:53:58Z] <James_F> Created integration-cumin-01 instance in WMCS based on stretch for final part of T236576

Change 594804 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] Point fabric at new, stretch-based CI cumin host

https://gerrit.wikimedia.org/r/594804

Mentioned in SAL (#wikimedia-releng) [2020-05-07T15:12:25Z] <hashar> Armed keyholder on integration-cumin-01 using key from integration-puppetmaster-02:/var/lib/git/labs/private/files/ssh/ # T236576

Change 594804 merged by jenkins-bot:
[integration/config@master] Point fabric at new, stretch-based CI cumin host

https://gerrit.wikimedia.org/r/594804

Mentioned in SAL (#wikimedia-releng) [2020-05-07T16:15:05Z] <James_F> Shutting off integration-cumin for T236576

Had to fight a bit with keyholder. The cumin master has to be listed in hiera cumin_masters list and keyholder had to be restarted to take in account the cumin configuration. integration-cumin-01 works now!

Mentioned in SAL (#wikimedia-releng) [2020-05-07T16:20:44Z] <James_F> Deleted integration-cumin.integration.eqiad.wmflabs for last part of T236576

Jdforrester-WMF updated the task description. (Show Details)