Page MenuHomePhabricator

Move all Wikimedia CI (WMCS integration project) instances from jessie to stretch
Closed, ResolvedPublic

Description

T94916: Re-create ci slaves (April 2015) || T252071: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster

The end of life of Debian Jessie is approaching in 2020 and we need to move to Debian Buster (or Stretch) before that date.

All instances in the integration project need to upgrade as soon as possible. Instances not upgraded by 2019-12-31 may be subject to deletion unless prior arrangements for an extended deadline has been approved by the Cloud VPS administration team.

Remaining Debian Jessie instances (live report):


Listed administrator is Release-Engineering-Team

See also:

More info on current project instances is available via openstack browser

Details

Due Date
Dec 31 2019, 11:59 PM

Related Objects

Event Timeline

StrikerBot triaged this task as Medium priority.Oct 26 2019, 12:32 AM
StrikerBot created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 26 2019, 12:32 AM
Krinkle removed a subscriber: Krinkle.Oct 26 2019, 2:28 AM
Krenair added a subscriber: Krenair.EditedOct 27 2019, 12:35 PM

I could do puppetmaster and cumin but don't know anything about the other instances.

Edit: Actually I checked the admin list, apparently I am only a member here so can't really.

hashar updated the task description. (Show Details)Oct 28 2019, 10:52 AM
hashar updated the task description. (Show Details)Oct 28 2019, 11:14 AM
bd808 added a comment.Dec 4 2019, 5:50 PM

The Cloud Services team would like to have Debian Jessie systems replaced with Debian Buster (or Stretch if necessary) before 2019-12-31. Please do respond of this task with comments if you know you will not be able to meet that target date or have additional questions about what to do or how to do it. Ideally you will create new instances in your Cloud VPS project, test them, and then migrate any final state data to the new instances before deleting the old Jessie instances. If you need more quota space in your project to create new instances in parallel with your existing instances please create a quota request task describing the increase you need and referencing this deprecation task.

thcipriani assigned this task to hashar.Dec 4 2019, 11:14 PM

Release-Engineering-Team talked about this at our team meeting this morning. The machines that are still pending migration have more complications than we can resolve by the end of the year.

We want to make this a project for Release-Engineering-Team for Q3.

Assigning to @hashar as the owner of our CI infra to break this task down a bit.

hashar updated the task description. (Show Details)Dec 10 2019, 2:15 PM

We still need some Jessie instances. Notably due to a long tail of Jenkins jobs that still have to be migrated and a CPU performance regression I have yet to reproduce using VMs on my local machine.

The breakdown of migrations is in the Continuous-Integration-Infrastructure subproject #phase-out-jessie

Jdforrester-WMF added a subscriber: Jdforrester-WMF.

Given that this "has" to happen by end-Q3, putting into our Q3 board.

hashar removed hashar as the assignee of this task.Mar 5 2020, 2:10 PM

Mentioned in SAL (#wikimedia-releng) [2020-03-05T23:11:47Z] <Krenair> T236576 Shut down integration-puppetmaster01 in favour of integration-puppetmaster-02. /root and /home archives can be found in /root on the new system. All integration instances moved to use the new master. Old master to be deleted on 12th March in the absence of objections

Krenair updated the task description. (Show Details)Mar 12 2020, 10:44 PM

I noticed this got unassigned around the time I picked up the puppetmaster work and began to look at cumin (which it turns out is blocked by part of T245114). I've completed the puppetmaster bit and will do cumin when possible. The rest still needs assignment or subtasks completed.

hashar updated the task description. (Show Details)Apr 17 2020, 4:28 PM
hashar updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-releng) [2020-04-29T16:03:01Z] <James_F> Removed integration-slave-jessie-1002 and integration-slave-jessie-1004 from Jenkins for T236576

Mentioned in SAL (#wikimedia-releng) [2020-04-29T16:03:51Z] <James_F> Shut down integration-slave-jessie-1002 and integration-slave-jessie-1004 in Horizon for T236576

Mentioned in SAL (#wikimedia-releng) [2020-05-01T18:41:50Z] <James_F> Deleting integration-slave-jessie-1002 and integration-slave-jessie-1004 instances from WMCS for T236576

Is this task still valid and should stay open? Asking as the Due Date on this task says December 2019. If this task is done, please resolve it (via Add Action...Change Status in the dropdown menu). If it is not done, please remove or reset the Due Date (via Edit Task). Thanks!

Krenair added a comment.EditedMay 3 2020, 4:06 PM

@Aklapper: It's still valid and should stay open, the due date is correctly set in the past. It's not clear to me whether December 2019 is accurate or if it should actually be some time in April 2020.
Edit: The same probably goes for most of the rest of the jessie deprecation tasks you just left this comment on.

Dzahn removed a subscriber: Dzahn.May 5 2020, 8:50 AM
hashar updated the task description. (Show Details)May 5 2020, 12:39 PM
Jdforrester-WMF renamed this task from "integration" Cloud VPS project jessie deprecation to Move all Wikimedia CI (WMCS integration project) instances from jessie to stretch.May 6 2020, 8:48 PM
Jdforrester-WMF updated the task description. (Show Details)

Re-scoping to just "get rid of jessie" and moving the cumin/buster blocker to T252071.

Mentioned in SAL (#wikimedia-releng) [2020-05-06T20:53:58Z] <James_F> Created integration-cumin-01 instance in WMCS based on stretch for final part of T236576

Change 594804 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] Point fabric at new, stretch-based CI cumin host

https://gerrit.wikimedia.org/r/594804

Mentioned in SAL (#wikimedia-releng) [2020-05-07T15:12:25Z] <hashar> Armed keyholder on integration-cumin-01 using key from integration-puppetmaster-02:/var/lib/git/labs/private/files/ssh/ # T236576

Change 594804 merged by jenkins-bot:
[integration/config@master] Point fabric at new, stretch-based CI cumin host

https://gerrit.wikimedia.org/r/594804

Mentioned in SAL (#wikimedia-releng) [2020-05-07T16:15:05Z] <James_F> Shutting off integration-cumin for T236576

hashar added a comment.May 7 2020, 4:15 PM

Had to fight a bit with keyholder. The cumin master has to be listed in hiera cumin_masters list and keyholder had to be restarted to take in account the cumin configuration. integration-cumin-01 works now!

Mentioned in SAL (#wikimedia-releng) [2020-05-07T16:20:44Z] <James_F> Deleted integration-cumin.integration.eqiad.wmflabs for last part of T236576

Jdforrester-WMF closed this task as Resolved.May 7 2020, 4:20 PM
Jdforrester-WMF updated the task description. (Show Details)
hashar added a comment.May 7 2020, 4:21 PM

Thank you so much @Jdforrester-WMF :]

antoine-approve