Page MenuHomePhabricator
Create Task
Maniphest T327742

Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm
Open, Needs TriagePublic
Actions

Description

< T278641: Migrate deployment-prep away from Debian Stretch to Buster/Bullseye | NOTYETCREATED >

With Stretch mostly removed now it's time to start removing Buster from deployment-prep, by either migrating services to newer Debian versions or by removing unused services entirely.

Tracking task for production migrations: T291916: Tracking task for Bullseye migrations in production

Instances to migrate

(live report)

  • deployment-acme-chief03 (replaced by deployment-acme-chief05)
  • deployment-acme-chief04 (replaced by deployment-acme-chief06)
  • deployment-cache-text07 (replaced by deployment-cache-text08)
  • deployment-cache-upload07 (replaced by deployment-cache-upload08)
  • deployment-cumin T361380
  • deployment-deploy03
  • deployment-docker-api-gateway01
  • deployment-docker-changeprop01
  • deployment-docker-cpjobqueue01
  • deployment-docker-mobileapps01
  • deployment-docker-proton01
  • deployment-echostore02 T361383
  • deployment-etcd02
  • deployment-eventlog08
  • deployment-ircd02
  • deployment-jobrunner04
  • deployment-kafka-jumbo-[5, 8-9] T361382
  • deployment-kafka-logging01 T361382
  • deployment-kafka-main-[5-6] T361382
  • deployment-maps-master01 T361381
  • deployment-mediawiki[11-12] T361387
  • deployment-memc[08-10] T361384
  • deployment-mwlog01
  • deployment-mwmaint02
  • deployment-ores02 T361385
  • deployment-parsoid12 T361386
  • deployment-poolcounter06
  • deployment-puppetdb[03-04] replaced with deployment-puppetdb05
  • deployment-puppetmaster04 replaced with deployment-puppetserver-1
  • deployment-push-notifications01
  • deployment-restbase04
  • deployment-sessionstore04
  • deployment-shellbox
  • deployment-snapshot03
  • deployment-urldownloader03
  • deployment-xhgui03

Details

SubjectRepoBranchLines +/-
[beta] Update wgCdnServersNoPurge to remove unused cache serversoperations/mediawiki-configmaster+0 -4
hiera: Removed unused cache instances from deployment-prepoperations/puppetproduction+0 -3
hiera: Added new bullseye instance for cache-text in deployment-prepoperations/puppetproduction+3 -0
[beta] Update wgCdnServersNoPurge for new cache serveroperations/mediawiki-configmaster+2 -0
hiera: Added new bullseye instance for cache-upload in deployment-prepoperations/puppetproduction+1 -1
[beta] Update wgCdnServersNoPurgeoperations/mediawiki-configmaster+2 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

taavi created this task.Jan 24 2023, 8:17 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 24 2023, 8:17 AM
taavi mentioned this in T278641: Migrate deployment-prep away from Debian Stretch to Buster/Bullseye.Jan 24 2023, 8:19 AM
Peachey88 added a project: Cloud-VPS (Debian Buster Deprecation).Jan 29 2023, 9:31 AM
taavi moved this task from Unsorted to VPS projects on the Cloud-VPS (Debian Buster Deprecation) board.Jan 29 2023, 10:18 AM
Vgutierrez updated the task description. (Show Details)Jun 16 2023, 1:39 PM
Vgutierrez updated the task description. (Show Details)Jun 16 2023, 1:48 PM
gerritbot added a comment.Jun 20 2023, 8:45 AM

Change 931550 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/mediawiki-config@master] [beta] Update wgCdnServersNoPurge

https://gerrit.wikimedia.org/r/931550

gerritbot added a project: Patch-For-Review.Jun 20 2023, 8:45 AM
TheresNoTime subscribed.Jun 20 2023, 9:38 AM
gerritbot added a comment.Jun 20 2023, 2:40 PM

Change 931550 merged by jenkins-bot:

[operations/mediawiki-config@master] [beta] Update wgCdnServersNoPurge

https://gerrit.wikimedia.org/r/931550

Maintenance_bot removed a project: Patch-For-Review.Jun 20 2023, 3:10 PM
gerritbot added a comment.Jun 22 2023, 10:00 AM

Change 932189 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: Added new bullseye instance for cache-upload in deployment-prep

https://gerrit.wikimedia.org/r/932189

gerritbot added a project: Patch-For-Review.Jun 22 2023, 10:00 AM
gerritbot added a comment.Jun 22 2023, 10:53 AM

Change 932189 merged by Fabfur:

[operations/puppet@production] hiera: Added new bullseye instance for cache-upload in deployment-prep

https://gerrit.wikimedia.org/r/932189

Stashbot added a comment.Jun 22 2023, 11:02 AM

Mentioned in SAL (#wikimedia-cloud) [2023-06-22T11:02:17Z] <fabfur> switch floating IP from deployment-cache-upload07 to deployment-cache-upload08 (bullseye upgrade: T327742)

Maintenance_bot removed a project: Patch-For-Review.Jun 22 2023, 11:10 AM
Fabfur updated the task description. (Show Details)Jun 23 2023, 9:12 AM
gerritbot added a comment.Jun 23 2023, 9:17 AM

Change 932380 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/mediawiki-config@master] [beta] Update wgCdnServersNoPurge for new cache server

https://gerrit.wikimedia.org/r/932380

gerritbot added a project: Patch-For-Review.Jun 23 2023, 9:17 AM
gerritbot added a comment.Jun 26 2023, 9:43 AM

Change 932380 merged by jenkins-bot:

[operations/mediawiki-config@master] [beta] Update wgCdnServersNoPurge for new cache server

https://gerrit.wikimedia.org/r/932380

Maintenance_bot removed a project: Patch-For-Review.Jun 26 2023, 10:10 AM
gerritbot added a comment.Jun 26 2023, 11:40 AM

Change 933085 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: Added new bullseye instance for cache-text in deployment-prep

https://gerrit.wikimedia.org/r/933085

gerritbot added a project: Patch-For-Review.Jun 26 2023, 11:40 AM
gerritbot added a comment.Jun 26 2023, 11:45 AM

Change 933085 merged by Fabfur:

[operations/puppet@production] hiera: Added new bullseye instance for cache-text in deployment-prep

https://gerrit.wikimedia.org/r/933085

Maintenance_bot removed a project: Patch-For-Review.Jun 26 2023, 12:10 PM
Stashbot added a comment.Jun 26 2023, 12:14 PM

Mentioned in SAL (#wikimedia-cloud) [2023-06-26T12:13:59Z] <fabfur> switch floating IP from deployment-cache-text07 to deployment-cache-text08 (bullseye upgrade: T327742)

Stashbot added a comment.Jun 26 2023, 12:41 PM

Mentioned in SAL (#wikimedia-cloud) [2023-06-26T12:41:49Z] <fabfur> switch floating IP from deployment-cache-text07 to deployment-cache-text08 (bullseye upgrade: T327742) (fix sec group)

Fabfur updated the task description. (Show Details)Jun 26 2023, 12:50 PM
gerritbot added a comment.Jun 27 2023, 11:31 AM

Change 933436 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: Removed unused cache instances from deployment-prep

https://gerrit.wikimedia.org/r/933436

gerritbot added a project: Patch-For-Review.Jun 27 2023, 11:31 AM
gerritbot added a comment.Jun 27 2023, 1:56 PM

Change 933463 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/mediawiki-config@master] [beta] Update wgCdnServersNoPurge to remove unused cache servers

https://gerrit.wikimedia.org/r/933463

gerritbot added a comment.Jun 27 2023, 1:58 PM

Change 933436 merged by Fabfur:

[operations/puppet@production] hiera: Removed unused cache instances from deployment-prep

https://gerrit.wikimedia.org/r/933436

gerritbot added a comment.Jun 28 2023, 9:52 AM

Change 933463 merged by jenkins-bot:

[operations/mediawiki-config@master] [beta] Update wgCdnServersNoPurge to remove unused cache servers

https://gerrit.wikimedia.org/r/933463

Stashbot added a comment.Jun 28 2023, 9:54 AM

Mentioned in SAL (#wikimedia-cloud) [2023-06-28T09:54:41Z] <fabfur> removed (text|upload) instance references from wgCdnServersNoPurge (T327742)

Maintenance_bot removed a project: Patch-For-Review.Jun 28 2023, 10:10 AM
Daimona mentioned this in T345566: error.log is not rotated in beta.Sep 5 2023, 12:17 AM
Andrew updated the task description. (Show Details)Mar 29 2024, 4:32 PM
Andrew updated the task description. (Show Details)
Andrew updated the task description. (Show Details)Mar 29 2024, 4:42 PM
Andrew updated the task description. (Show Details)Mar 29 2024, 4:47 PM
Andrew updated the task description. (Show Details)Mar 29 2024, 4:57 PM
thcipriani subscribed.Mar 29 2024, 4:57 PM

We talked about this a bit in the Release-Engineering-Team team meeting on Wednesday—we discussed whether we had the ability to help. And I don't think we're well-positioned to do all this work as this encompasses a clone of most of Wikimedia's production footprint (where SRE has the most expertise).

I think one path forward here would be for multiple teams pitch-in to fix the instances they know how to fix.

Looking at the instance list, I do wonder if they're needed in beta (echostore?). A place to start might be a bunch of subtasks verifying:

  1. Needed for beta? (i.e., critical to some team's workflow, if so: who?)
  2. Ported to Bullseye in production? If not, any foreseeable blockers?
  3. Can you help?

Other notes:

  • Quota: deployment-prep project is currently at its quota for vCPU, so it'd need some extra headroom to spin up new Bullseye instances before shutting down the "working" ones.
  • Production debian packages: T291916: Tracking task for Bullseye migrations in production may be a blocker in some cases deployment-prep depends on the same Debian packages used for production, and if they're not yet packaged for Bullseye, then there's no way to get a Bullseye image to run using production puppet code.
Andrew updated the task description. (Show Details)Mar 29 2024, 5:05 PM
thcipriani added a comment.Mar 29 2024, 5:52 PM
In T327742#9673367, @thcipriani wrote:
  • Quota: deployment-prep project is currently at its quota for vCPU, so it'd need some extra headroom to spin up new Bullseye instances before shutting down the "working" ones.

The average instance on deployment-prep is ~ 2vCPU, 20GB disk, 4GB ram.

deployment-prep is almost at RAM quota and at CPU quota. Bumping those up to allow 5 new instance should allow folks to at least get started. So 10vCPU/20GB ram? Possible? It'll probably possible to reclaim some of that post-migration.

Andrew updated the task description. (Show Details)Mar 29 2024, 6:02 PM
Andrew updated the task description. (Show Details)Mar 29 2024, 6:07 PM
Andrew updated the task description. (Show Details)
thcipriani mentioned this in T361477: requests to increase quotas deployment-prep.Apr 1 2024, 3:36 PM
In T327742#9673515, @thcipriani wrote:
In T327742#9673367, @thcipriani wrote:
  • Quota: deployment-prep project is currently at its quota for vCPU, so it'd need some extra headroom to spin up new Bullseye instances before shutting down the "working" ones.

The average instance on deployment-prep is ~ 2vCPU, 20GB disk, 4GB ram.

deployment-prep is almost at RAM quota and at CPU quota. Bumping those up to allow 5 new instance should allow folks to at least get started. So 10vCPU/20GB ram? Possible? It'll probably possible to reclaim some of that post-migration.

Added as subtask: T361477: requests to increase quotas deployment-prep

rook closed subtask T361477: requests to increase quotas deployment-prep as Resolved.Apr 1 2024, 7:45 PM
elukey mentioned this in T360595: beta-scap-sync-world fails: logstash_checker.py: KeyError: 'aggregations'.Apr 2 2024, 12:57 PM
Andrew closed subtask T361385: Replace deployment-ores02 as Resolved.Apr 2 2024, 1:19 PM
elukey subscribed.Mon, May 20, 3:50 PM

I have asked something on IRC's wikimedia-cloud about a way to speed up the process. In the puppet repo we have dist-upgrade.sh, a script that should be able to do all the safe steps to upgrade the OS in place. If it is ok to use it, we'd avoid the re-creation of new VMs with the issue of copying stateful data around (for example, Kafka/databases/etc..). The only downside would be for the Cloud team, that would need to fix the VM's metadata (like reported OS) after the dist-upgrade.

Andrew closed subtask T361536: Launching new bullseye deployment-prep instances fails, no sudo access as Resolved.Fri, Jun 14, 5:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL