Page MenuHomePhabricator
Create Task
Maniphest T362518

Deprecate buster-backports
Open, In Progress, HighPublic
Actions

Description

The buster-backports repo is no longer available. This is preventing us from building any images that include his repo on apt-update failure

Get:1 http://security.debian.org buster/updates InRelease [34.8 kB]
Hit:2 http://apt.wikimedia.org/wikimedia buster-wikimedia InRelease
Hit:3 http://mirrors.wikimedia.org/debian buster InRelease
Hit:4 http://mirrors.wikimedia.org/debian buster-updates InRelease
Ign:5 http://mirrors.wikimedia.org/debian buster-backports InRelease
Err:6 http://mirrors.wikimedia.org/debian buster-backports Release
  404  Not Found [IP: 208.80.154.74 8080]
Get:7 http://security.debian.org buster/updates/main amd64 Packages [595 kB]
Reading package lists...
E: The repository 'http://mirrors.wikimedia.org/debian buster-backports Release' no longer has a Release file.The command '/bin/sh -c apt-get update && apt-get install -y php7.4-tideways php7.4-ldap' returned a non-zero code: 100
make[1]: *** [Makefile:17: build-debug-image] Error 100
make[1]: Leaving directory '/srv/mwbuilder/release/make-container-image'
make: *** [Makefile:35: build-and-push-all-images] Error 2
make: Leaving directory '/srv/mwbuilder/release/make-container-image'
09:01:05 Finished build-and-push-container-images (duration: 00m 30s)
09:01:05 Build of K8s images failed (non-K8s deployment will continue normally)

According to CodeSearch, affected images are:

  • shellbox (lilypond and lilypond-data)
  • integration/config/operations-puppet (shellcheck)
  • integration/config/tox-buster (shellcheck)
  • integration/config/tox-pywikibot (shellcheck)
  • mediawiki/vagrant (php7.4-wikidiff2)

The base buster image also needs to have it removed, as it blocks all deployment of MW-on-K8s on apt-update failure

  • create component/lilypond and import the last version from buster-backports
  • Rebuild base-images
  • Rebuild buster-based production-images
  • Do a full rebuild deployment of mw-on-k8s scap sync-world --k8s-only -D full_image_build:true
  • Restart docker-reporter-base-images and docker-reporter-k8s-images
  • Start docker-system-prune-all.service when the above two are done to clean up build2001

Details

Show related patches Customize query in gerrit

Related Objects

Event Timeline

Clement_Goubert created this task.Mon, Apr 15, 10:09 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Apr 15, 10:09 AM
Clement_Goubert changed the task status from Open to In Progress.Mon, Apr 15, 10:09 AM
Clement_Goubert triaged this task as Unbreak Now! priority.
gerritbot added a comment.Mon, Apr 15, 10:10 AM

Change #1019706 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] docker: Remove buster-backports from sources.list

https://gerrit.wikimedia.org/r/1019706

gerritbot added a project: Patch-For-Review.Mon, Apr 15, 10:10 AM
taavi subscribed.Mon, Apr 15, 10:13 AM
gerritbot added a comment.Mon, Apr 15, 10:19 AM

Change #1019706 merged by Clément Goubert:

[operations/puppet@production] docker: Remove buster-backports from sources.list

https://gerrit.wikimedia.org/r/1019706

gerritbot added a comment.Mon, Apr 15, 10:20 AM

Change #1019710 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add a component/lilypond

https://gerrit.wikimedia.org/r/1019710

Ladsgroup subscribed.Mon, Apr 15, 10:21 AM
gerritbot added a comment.Mon, Apr 15, 10:22 AM

Change #1019710 merged by Muehlenhoff:

[operations/puppet@production] Add a component/lilypond

https://gerrit.wikimedia.org/r/1019710

Stashbot added a comment.Mon, Apr 15, 10:22 AM

Mentioned in SAL (#wikimedia-operations) [2024-04-15T10:22:45Z] <claime> Launching build-base-images on build2001 - T362518

MoritzMuehlenhoff updated the task description. (Show Details)Mon, Apr 15, 10:23 AM
hnowlan subscribed.Mon, Apr 15, 10:23 AM
Clement_Goubert updated the task description. (Show Details)Mon, Apr 15, 10:25 AM
Stashbot added a comment.Mon, Apr 15, 10:31 AM

Mentioned in SAL (#wikimedia-operations) [2024-04-15T10:31:02Z] <moritzm> imported lilypond/lilypond-data 2.22.0-10~bpo10+1 to component/lilypond T362518

hashar subscribed.Mon, Apr 15, 10:32 AM

integration/config/operations-puppet (shellcheck)

The image is based on Buster since the Puppet master in production use / used Debian Buster. That is to ensure we use / used the same version of ruby as in production. I guess that image can be recreated targeting Bullseye / Bookworm.

It also uses tox v4 which is broken enough that I am considering phasing it out. At least for operations/puppet that seems to work for now.

integration/config/tox-buster (shellcheck)
integration/config/tox-pywikibot (shellcheck)

Those images are in a limbo and should really be rethought. I have made some attempt over the winter but have hit a few walls , the biggest one being the upgrade of tox from v3 to v4 but I guess we can stick to v3 until we replace tox by something else.

The new image to use would eventually be releng/tox-v3 which is based on Bookworm and use pyenv to ship various python versions.

Those two can wait though.

mediawiki/vagrant (php7.4-wikidiff2)

MediaWiki-Vagrant is based on Buster, then it is no more officially supported. There is a task to update it to Bullseye T319167

jijiki updated the task description. (Show Details)Mon, Apr 15, 10:32 AM
hnowlan added a comment.Mon, Apr 15, 10:40 AM
In T362518#9712748, @hashar wrote:

mediawiki/vagrant (php7.4-wikidiff2)

MediaWiki-Vagrant is based on Buster, then it is no more officially supported. There is a task to update it to Bullseye T319167

There is a version of php7.4-wikidiff2 in buster-wikimedia that can be used in place of this, assuming the underlying stuff is compatible.

MoritzMuehlenhoff subscribed.Mon, Apr 15, 10:44 AM
In T362518#9712748, @hashar wrote:

integration/config/operations-puppet (shellcheck)

The image is based on Buster since the Puppet master in production use / used Debian Buster. That is to ensure we use / used the same version of ruby as in production. I guess that image can be recreated targeting Bullseye / Bookworm.

The Puppet 5 servers are running on Buster, but as of now only 441 servers are using them. The other 1739 systems in production get served by the Puppet 7 servers running on Bookworm, but the check can't easily know which Puppet version gets targeted, after all we have fleet-wide changes in Puppet which get applied to the 5 and 7 hosts. One option might be a separate operations-puppet7 check and then run them in parallel until we phase out the legacy check in 1-2 months.

gerritbot added a comment.Mon, Apr 15, 10:52 AM

Change #1019716 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[mediawiki/libs/Shellbox@master] blubber: Use lilypond component

https://gerrit.wikimedia.org/r/1019716

gerritbot added a comment.Mon, Apr 15, 10:52 AM

Change #1019717 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add a component/shellcheck

https://gerrit.wikimedia.org/r/1019717

MoritzMuehlenhoff updated the task description. (Show Details)Mon, Apr 15, 10:55 AM
gerritbot added a comment.Mon, Apr 15, 10:56 AM

Change #1019717 merged by Muehlenhoff:

[operations/puppet@production] Add a component/shellcheck

https://gerrit.wikimedia.org/r/1019717

JMeybohm updated the task description. (Show Details)Mon, Apr 15, 10:56 AM
Stashbot added a comment.Mon, Apr 15, 11:07 AM

Mentioned in SAL (#wikimedia-operations) [2024-04-15T11:07:44Z] <moritzm> imported shellcheck 0.7.1-1~bpo10+1 to component/shellcheck T362518

gerritbot added a comment.Mon, Apr 15, 11:11 AM

Change #1019721 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove obsolete apt::pin for buster-backports

https://gerrit.wikimedia.org/r/1019721

Clement_Goubert updated the task description. (Show Details)Mon, Apr 15, 11:17 AM
Clement_Goubert updated the task description. (Show Details)Mon, Apr 15, 11:20 AM
gerritbot added a comment.Mon, Apr 15, 11:31 AM

Change #1019726 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Only install Go from backports on bullseye-based stat hosts

https://gerrit.wikimedia.org/r/1019726

gerritbot added a comment.Mon, Apr 15, 11:37 AM

Change #1019730 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] beta::mediawiki_packages: Install lilypond from component

https://gerrit.wikimedia.org/r/1019730

gerritbot added a comment.Mon, Apr 15, 11:51 AM

Change #1019734 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Only add backports on Bullseye/Bookworm

https://gerrit.wikimedia.org/r/1019734

gerritbot added a comment.Mon, Apr 15, 11:55 AM

Change #1019734 merged by Muehlenhoff:

[operations/puppet@production] Only add backports on Bullseye/Bookworm

https://gerrit.wikimedia.org/r/1019734

akosiaris subscribed.Mon, Apr 15, 12:14 PM
gerritbot added a comment.Mon, Apr 15, 12:21 PM

Change #1019748 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] aptrepo: remove buster-backports haproxy components

https://gerrit.wikimedia.org/r/1019748

gerritbot added a comment.Mon, Apr 15, 12:26 PM

Change #1019748 abandoned by Hnowlan:

[operations/puppet@production] aptrepo: remove buster-backports haproxy components

Reason:

Not needed at this time.

https://gerrit.wikimedia.org/r/1019748

TheresNoTime subscribed.Mon, Apr 15, 1:45 PM
JMeybohm subscribed.Mon, Apr 15, 1:50 PM

Production images rebuild is done:

Apr 15 13:46:55 build2001 update-production-images[4127624]: Successfully published image docker-registry.discovery.wmnet/php7.4-fpm:7.4.33-4-20240415
Apr 15 13:46:55 build2001 update-production-images[4127624]: Successfully published image docker-registry.discovery.wmnet/php7.4-fpm-multiversion-base:7.4.33-9-20240415
Apr 15 13:46:55 build2001 update-production-images[4127624]: Successfully published image docker-registry.discovery.wmnet/php7.4-cli:7.4.33-1-s2-20240415
JMeybohm updated the task description. (Show Details)Mon, Apr 15, 1:50 PM
JMeybohm updated the task description. (Show Details)Mon, Apr 15, 2:29 PM
JMeybohm updated the task description. (Show Details)Mon, Apr 15, 2:32 PM
akosiaris lowered the priority of this task from Unbreak Now! to High.Mon, Apr 15, 2:34 PM

The immediate issue blocking the train has been resolved and new images have been pushed. Hence, lowering to High. There's a tail of images being rebuilt still and it's going to take a while longer, but this is no longer a UBN

JMeybohm updated the task description. (Show Details)Mon, Apr 15, 4:49 PM
akosiaris mentioned this in T362567: Add docker production images repo to codesearch.Mon, Apr 15, 5:40 PM
gerritbot added a comment.Tue, Apr 16, 6:45 AM

Change #1019730 merged by Muehlenhoff:

[operations/puppet@production] beta::mediawiki_packages: Install lilypond from component

https://gerrit.wikimedia.org/r/1019730

gerritbot added a comment.Tue, Apr 16, 6:53 AM

Change #1019721 merged by Muehlenhoff:

[operations/puppet@production] Remove obsolete apt::pin for buster-backports

https://gerrit.wikimedia.org/r/1019721

gerritbot added a comment.Tue, Apr 16, 11:24 AM

Change #1019716 merged by jenkins-bot:

[mediawiki/libs/Shellbox@master] blubber: Use lilypond component

https://gerrit.wikimedia.org/r/1019716

Stevemunene mentioned this in T362648: Rebuild conda-analytics container on Bullseye.Tue, Apr 16, 12:23 PM
Stevemunene mentioned this in T356231: Package versions in Conda-Analytics are not pinned.
Clement_Goubert added a comment.Tue, Apr 16, 2:00 PM

The following images fail docker-reporter checks because they haven't been rebuilt on top of the new buster base image:

base images
docker-registry.wikimedia.org/docker-gc:1.0.0-20230402              [FAIL]
docker-registry.wikimedia.org/golang:1.14-1-20240407                [FAIL]
docker-registry.wikimedia.org/httpd-fcgi:2.4.38-10-u5-20240407      [FAIL]
docker-registry.wikimedia.org/kubeflow-kfserving-agent:0.6.0-1-20211017[FAIL]
docker-registry.wikimedia.org/kubeflow-kfserving-controller:0.6.0-1-20211017[FAIL]
docker-registry.wikimedia.org/kubeflow-kfserving-storage-initializer:0.6.0-5-20211010[FAIL]
docker-registry.wikimedia.org/loki:1.5.0-2-20230604                 [FAIL]
docker-registry.wikimedia.org/mediawiki-httpd:0.1.8-s2-20240407     [FAIL]
docker-registry.wikimedia.org/php7.2-cli:0.2.0-s3-20221204          [FAIL]
docker-registry.wikimedia.org/php7.2-fpm:0.4.0-20221204             [FAIL]
docker-registry.wikimedia.org/php7.2-fpm-multiversion-base:1.0.7-20221204[FAIL]
docker-registry.wikimedia.org/php7.4-cli-icu67:7.4.33-1-s2-20231106-20231106[FAIL]
docker-registry.wikimedia.org/php7.4-fpm-icu67:7.4.33-3-20231106-20231106[FAIL]
docker-registry.wikimedia.org/wikimedia-buster:20210523             [FAIL]
service images
docker-registry.wikimedia.org/wikimedia/blubber:stable              [FAIL]
docker-registry.wikimedia.org/wikimedia/generated-data-platform-datasets-image-suggestions:stable[FAIL]
docker-registry.wikimedia.org/wikimedia/labs-libraryupgrader:web    [FAIL]
docker-registry.wikimedia.org/wikimedia/mediawiki:wmf-1.37.0-wmf.17 [FAIL]
docker-registry.wikimedia.org/wikimedia/mediawiki-libs-shellbox:icu67[FAIL]
docker-registry.wikimedia.org/wikimedia/mediawiki-services-apple-search:production[FAIL]
docker-registry.wikimedia.org/wikimedia/mediawiki-services-example-node-api:2021-11-09-141841-candidate[FAIL]
docker-registry.wikimedia.org/wikimedia/mediawiki-services-image-suggestion-api:2023-01-31-172753-production[FAIL]
docker-registry.wikimedia.org/wikimedia/mediawiki-services-kartotherian:kartotherian[FAIL]
docker-registry.wikimedia.org/wikimedia/mediawiki-services-similar-users:2022-08-17-104210-production[FAIL]
docker-registry.wikimedia.org/wikimedia/operations-container-miscweb:fifteenwp[FAIL]
docker-registry.wikimedia.org/wikimedia/operations-software-thumbor-plugins:2023-10-06-153556-production[FAIL]
docker-registry.wikimedia.org/wikimedia/research-mwaddlink:test     [FAIL]
docker-registry.wikimedia.org/wikimedia/wikidata-query-flink-rdf-streaming-updater:2023-02-16-151433-production[FAIL]
docker-registry.wikimedia.org/wikimedia/wikimedia-portals:2024-04-15-124732-production[FAIL]
Jdforrester-WMF subscribed.Tue, Apr 16, 4:17 PM

This has also broken building CI images. Will have to migrate them to bullseye immediately, I suppose.

Jdforrester-WMF mentioned this in T335766: Migrate Quibble images from buster to bullseye.Tue, Apr 16, 5:30 PM
hashar added a comment.Wed, Apr 17, 6:18 AM
In T362518#9719270, @Jdforrester-WMF wrote:

This has also broken building CI images. Will have to migrate them to bullseye immediately, I suppose.

See my comment above: T362518#9712748

MoritzMuehlenhoff added a comment.Wed, Apr 17, 7:01 AM
In T362518#9719270, @Jdforrester-WMF wrote:

This has also broken building CI images. Will have to migrate them to bullseye immediately, I suppose.

To unblock the current images I created a repository component/shellcheck on apt.wikimedia.org and imported the version of shellcheck which was previously on buster-backports, with that we can easily unblock the current images.

gerritbot added a comment.Wed, Apr 17, 10:03 AM

Change #1019726 merged by Muehlenhoff:

[operations/puppet@production] Only install Go from backports on bullseye-based stat hosts

https://gerrit.wikimedia.org/r/1019726

Jdforrester-WMF added a comment.Wed, Apr 17, 1:13 PM
In T362518#9721307, @hashar wrote:
In T362518#9719270, @Jdforrester-WMF wrote:

This has also broken building CI images. Will have to migrate them to bullseye immediately, I suppose.

See my comment above: T362518#9712748

Yes, sure, those were also broken, but as discussed yesterday in IRC my concern is about quibble images which are unbuildable at the first apt step (apt-get install --yes build-essential unzip python --no-install-recommends fails with The repository 'http://mirrors.wikimedia.org/debian buster-backports Release' does not have a Release file.).

EBernhardson subscribed.Wed, Apr 17, 7:24 PM
dancy merged a task: T362931: mw-on-k8s container builds failing due to The repository 'http://mirrors.wikimedia.org/debian buster-backports Release' does not have a Release file..Thu, Apr 18, 7:28 PM
dancy subscribed.
dancy added a comment.Thu, Apr 18, 7:34 PM

Nothing that I'm blocked on scap development/testing in train-dev due to docker-registry.wikimedia.org/mediawiki-httpd:latest getting the The repository 'http://mirrors.wikimedia.org/debian buster-backports Release' no longer has a Release file error during apt-get update. (T362931 (now merged into this ticket))

JMeybohm added a comment.Fri, Apr 19, 8:25 AM

httpd-fcgi + dependent images seem to not have successfully rebuild on Monday. checking.

JMeybohm added a comment.Fri, Apr 19, 8:29 AM

Successfully published image docker-registry.discovery.wmnet/httpd-fcgi:2.4.38-10-u5-20240415
Successfully published image docker-registry.discovery.wmnet/mediawiki-httpd:0.1.8-s2-20240415

docker run --rm -it --pull=always --user 0 --entrypoint /bin/bash docker-registry.wikimedia.org/mediawiki-httpd:latest -c "apt-get update" LGTM

gerritbot added a comment.Fri, Apr 19, 9:21 AM

Change #1021877 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] build-bare-slim: Stop building wikimedia-buster

https://gerrit.wikimedia.org/r/1021877

gerritbot added a comment.Fri, Apr 19, 9:24 AM

Change #1021878 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/docker-images/production-images@master] Replace wikimedia-buster base images with buster

https://gerrit.wikimedia.org/r/1021878

Jdforrester-WMF mentioned this in T362981: Migrate mw-on-k8s base image from buster to bullseye.Fri, Apr 19, 1:06 PM
gerritbot added a comment.Fri, Apr 19, 10:57 PM

Change #1022215 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] codsearch: use thirdparty-ci repo to get docker-ce on buster

https://gerrit.wikimedia.org/r/1022215

Dzahn subscribed.Fri, Apr 19, 10:58 PM

According to CodeSearch, affected images are:

Codesearch didn't find itself ?:)

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1022215

gerritbot added a comment.Tue, Apr 23, 9:32 AM

Change #1021878 merged by JMeybohm:

[operations/docker-images/production-images@master] Replace wikimedia-buster base images with buster

https://gerrit.wikimedia.org/r/1021878

gerritbot added a comment.Tue, Apr 23, 10:01 AM

Change #1021877 merged by Clément Goubert:

[operations/puppet@production] build-bare-slim: Date tag wikimedia-buster images

https://gerrit.wikimedia.org/r/1021877

gerritbot added a comment.Tue, Apr 23, 6:50 PM

Change #1022215 abandoned by Dzahn:

[operations/puppet@production] codsearch: use thirdparty-ci repo to get docker-ce on buster

Reason:

per comments above, real fix is to stop using buster

https://gerrit.wikimedia.org/r/1022215

gerritbot added a comment.Wed, Apr 24, 1:17 PM

Change #1023853 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: rm buster-backports and use component/shellcheck

https://gerrit.wikimedia.org/r/1023853

gerritbot added a comment.Wed, Apr 24, 1:40 PM

Change #1023853 merged by jenkins-bot:

[integration/config@master] dockerfiles: rm buster-backports and use component/shellcheck

https://gerrit.wikimedia.org/r/1023853

Stashbot added a comment.Wed, Apr 24, 1:45 PM

Mentioned in SAL (#wikimedia-releng) [2024-04-24T13:45:14Z] <hashar> Building Buster CI images to get rid of buster-backports | T362518 | https://gerrit.wikimedia.org/r/1023853

gerritbot added a comment.Wed, Apr 24, 2:36 PM

Change #1023870 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: update jobs for buster-backports removal

https://gerrit.wikimedia.org/r/1023870

Stashbot added a comment.Wed, Apr 24, 2:54 PM

Mentioned in SAL (#wikimedia-releng) [2024-04-24T14:54:43Z] <hashar> Updating Jenkins jobs for buster-backports removal | T362518 | https://gerrit.wikimedia.org/r/c/integration/config/+/1023870

gerritbot added a comment.Wed, Apr 24, 2:57 PM

Change #1023870 merged by jenkins-bot:

[integration/config@master] jjb: update jobs for buster-backports removal

https://gerrit.wikimedia.org/r/1023870

Maintenance_bot removed a project: Patch-For-Review.Fri, Apr 26, 6:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL