Page MenuHomePhabricator
Create Task
Maniphest T368366

Upgrade K8s docker images to running in production on Buster with either Bullseye or Bookworm
Open, Needs TriagePublic
Actions

Description

Hi folks!

Our dear Buster is not going to be supported soon by Debian, so we should upgrade to either Bullseye or Bookworm. Debmonitor is now able to report the Debian version-id for a lot of Docker images in our registry:
https://debmonitor.wikimedia.org/images/

Core images:

  • envoy
  • envoy-future
  • cfssl-issuer
  • coredns
  • helm-state-metrics
  • echoserver (not really but used for testing by folks etc..)

Mediawiki-related:

  • mcrouter
  • prometheus-mcrouter-exporter

Will add more as I review the Debmonitor's report in more details.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT368366 Upgrade K8s docker images to running in production on Buster with either Bullseye or Bookworm
 OpenNoneT368359 Upgrade Knative control plane Docker images to Bullseye/Bookworm
 OpenelukeyT367427 Cleanup old Docker images running Debian Stretch/Jessie
 OpenelukeyT368744 Allow debmonitor to store the Debian version-id in the OS field
 StalledNoneT356293 Migrate MW appservers' base images to bullseye
 OpenNoneT290536 Serve production traffic via Kubernetes
 ResolvedjijikiT280497 Benchmark performance of MediaWiki on k8s
 ResolvedaaronT293630 Investigate performance degradation at high concurrencies in php-fpm
OpenjijikiT292707 Migrate Wikitech to Kubernetes
 In ProgressClement_GoubertT321786 Deploy mediawiki kubernetes services
 ResolvedClement_GoubertT321895 Create mw-api-int helmfile deployment
 ResolvedClement_GoubertT321896 Create mw-api-ext helmfile deployment
 ResolvedClement_GoubertT321897 Create mw-jobrunner helmfile deployment
 StalledClement_GoubertT321899 Create mw-videoscaler helmfile deployment
 ResolvedClement_GoubertT321900 Create mw-web helmfile deployment
 ResolvedClement_GoubertT323296 Stop spamming SAL with helmfile on scap deployments
 ResolvedClement_GoubertT323621 Make mw-web and mw-api-ext available behind LVS
 ResolvedJoeT323349 Improve performance of deployment to mw on k8s
 ResolvedcolewhiteT325085 Increase in Logstash warnings
 ResolvedClement_GoubertT331318 Find a sensible way to direct traffic to mw-on-k8s
 ResolvedClement_GoubertT331268 Migrate testwikidata to Kubernetes
 In ProgressClement_GoubertT333120 Migrate internal traffic to k8s
 ResolvedClement_GoubertT334060 Migrate linkrecommendations to mw-api-int
 ResolvedClement_GoubertT334061 Migrate push-notifications to mw-api-int
 ResolvedClement_GoubertT334062 Migrate recommendation-api to mw-api-int
 InvalidClement_GoubertT334063 Migrate tegola-vector-tiles to mw-api-int
 ResolvedJoeT334064 Migrate termbox to mw-api-int
 ResolvedClement_GoubertT334065 Migrate api-gateway to mw-api-int
 ResolvedClement_GoubertT334204 Migrate cxserver to mw-api-int
 ResolvedClement_GoubertT334456 Monitor all mw-on-k8s deployments with httpbb
 ResolvedClement_GoubertT334561 Add mw-on-k8s deployments to mediawiki certificates
 ResolvedJoeT342252 Migrate rdf-streaming-updater to connect to mw-on-k8s
 ResolvedJoeT346447 Migrate wikifeeds to mw-api-int
 ResolvedJoeT346448 Migrate all eventgate installations to mw-api-int
 ResolvedJMeybohmT347397 Migrate functions-orchestrator service to mw-api-int
 ResolvedJoeT350846 Migrate mobileapps to k8s
 ResolvedClement_GoubertT357785 Migrate mw-page-content-change-enrich to mw-api-int
 ResolvedClement_GoubertT358213 Migrate restbase from mwapi-async to mw-api-int
 ResolvedClement_GoubertT298265 Have internal MediaWiki to MediaWiki HTTP requests use an envoyproxy on appservers
 DeclinedClement_GoubertT360625 Alter changeprop chart to use the service mesh
 ResolvedClement_GoubertT360767 Migrate changeprop to mw-api-int
 ResolvedelukeyT362316 Migrate ml-services to mw-api-int
 StalledKrinkleT333269 Benchmark baremetal vs k8s mediawiki perf (2023)
 ResolvedjijikiT336025 ResourceLoader icon rasterization fails via MediaWiki-on-Kubernetes
 ResolvedJoeT336037 Add configuration file support to mw-on-k8s.lua ATS script
 ResolvedJoeT336038 Add traffic sampling support to mw-on-k8s.lua ATS script
 ResolvedClement_GoubertT337490 Migrate group0 to Kubernetes
 ResolvedBUG REPORTClement_GoubertT340483 ExtensionDistributor is broken on k8s
 ResolvedClement_GoubertT337489 Run QTE test suite on testwiki on kubernetes
 ResolvedClement_GoubertT340549 Migrate group1 to Kubernetes
 ResolvedClement_GoubertT341078 Direct 0.5% of all traffic to mw-on-k8s
 ResolvedClement_GoubertT341463 Direct 1% of all traffic to mw-on-k8s
 ResolvedClement_GoubertT341780 Direct 5% of all traffic to mw-on-k8s
 ResolvedJoeT341825 Max upload size on k8s is 2M
 ResolvedClement_GoubertT342748 mw-on-k8s app container CPU throttling at low average load
 ResolvedClement_GoubertT343306 Wikikube CPU capacity issue
 ResolvedJclark-ctrT343708 Physical re-labeling of mw1497 and mw1498 to kubernetes1025 and kubernetes1026
 ResolvedJMeybohmT343978 Allow more flexibility in ResourceQuota and LimitRanger config
 ResolvedJoeT283056 Create a mwdebug deployment for mediawiki on kubernetes
 ResolvedJoeT284417 Add the puppet CA to the MediaWiki deployment
 ResolvedNoneT284418 Add conditional to mediawiki-config for stuff running on kubernetes
 ResolvedjijikiT284420 Add all redis and memcached backends to mw on k8s automatically
 ResolvedjijikiT284421 Enable TLS termination on the mwdebug deployment. fix the service definition in the chart
 ResolveddancyT284581 Ownership of the /tmp/mw-cache directories should be www-data in the mediawiki-multiversion image
 ResolveddancyT284582 The restricted/mediawiki-multiversion image should include the production version of private/PrivateSettings.php
 ResolvedJoeT285298 Make all httpbb tests pass on the mwdebug deployment.
 ResolvedJoeT285309 Install wiki-specific php extensions in the mediawiki production image
 ResolvedjeenaT285325 Check out www-portals repo in the mediawiki-webserver and in the mediawiki-multiversion images
 ResolvedJoeT285232 The restricted/mediawiki-webserver image should include skins and resources
 ResolvedJoeT286491 Access mwdebug kubernetes deployment via the 'X-Wikimedia-Debug' header
 ResolveddancyT287512 GitInfo is missing from mwdebug-kubernetes deployment
 ResolvedJoeT287570 Ensure the code is deployed to mediawiki on k8s when it is deployed to production
 Resolved• dpifkeT288164 Ensure WikimediaDebug "log" and "profile" features work with k8s-mwdebug
 ResolveddancyT287495 Create a variant of mediawiki-multiversion which installs php-tideways-xhprof
 ResolvedJoeT288851 Make logging work for mediawiki in k8s
 ResolvedClement_GoubertT326794 Ingest php-slowlog in logstash
 ResolvedcolewhiteT328318 Index orchestrator object fields from ECS 1.11.0 in OpenSearch
 ResolvedjijikiT290485 mediawiki-debug image does not produce profiling info
Resolved• dpifkeT288165 Create separate ArcLamp pipeline for k8s-mwdebug
 ResolvedLegoktmT288848 Make HTTP calls work within mediawiki on kubernetes
ResolvedClement_GoubertT343390 MW-on-k8s traffic logs fewer errors than expected (increase in jsonTruncated)
ResolvedJMeybohmT277876 Reserve resources for system daemons on kubernetes nodes
 ResolvedClement_GoubertT346422 Move 10% of mediawiki external requests to mw on k8s
 ResolvedClement_GoubertT344814 mw-on-k8s tls-proxy container CPU throttling at low average load
 OpenNoneT354532 Limit the concurrency of envoy in service mesh
 OpenClement_GoubertT345243 Remove tls-proxy cpu limits on eventstreams
 In ProgressClement_GoubertT345244 Remove tls-proxy cpu limits on eventgate
 ResolvedClement_GoubertT347493 Serve Wikidata traffic via Kubernetes
 ResolvedClement_GoubertT348122 Move 25% of mediawiki external requests to mw on k8s
 ResolvedhnowlanT349796 Move MediaWiki jobs to mw-on-k8s
 ResolvedakosiarisT352906 mediawiki k8s jobrunner fails connecting to cloudelastic with a TLS error
 OpenhnowlanT354791 Reclaim jobrunner hardware for k8s
 ResolvedClement_GoubertT355622 scap not installed on mw1486.eqiad.wmnet which breaks deployment: /usr/bin/scap: No such file or directory
 ResolvedNoneT355243 MediaModeration maintenance script scanFilesInScanTable.php indirectly calls $wgImageMagickConvertCommand
 ResolvedDreamy_JazzT355309 Don't use RENDER_NOW when generating thumbnails for PhotoDNA scans
 ResolvedPRODUCTION ERRORDreamy_JazzT355177 Error creating thumbnail: mkdir: cannot create directory '/sys/fs/cgroup/memory/mediawiki/job/XXXXX':
 ResolvedkostajhT356047 Attempt to generate thumbnail by requesting URL
 ResolvedjijikiT359154 Update DC switchover cookbooks to handle mw-jobrunners
 OpenNoneT351074 Move servers from the appserver/api cluster to kubernetes
 ResolvedkamilaT354413 Reboot issues for mw13[77-83].eqiad.wmnet
 ResolvedJhancock.wmT367736 Relabel codfw kubernetes nodes
 ResolvedJclark-ctrT367789 Relabel eqiad kubernetes nodes
 ResolvedJhancock.wmT368079 Relabel codfw kubernetes nodes
 OpenNoneT368639 Relabel eqiad kubernetes nodes
 OpenNoneT368743 Relabel codfw kubernetes nodes
 ResolvedClement_GoubertT355532 Move 40% of mediawiki external requests to mw on k8s
 ResolvedClement_GoubertT355534 Move testwiki over to mw-on-k8s
 ResolvedakosiarisT357392 Create parsoid mediawiki deployment and migrate parsoid-php.discovery.wmnet traffic to it
 OpenNoneT358588 A lot of `[info] Wikitext for this page has duplicate ids:` in logstash for mw-parsoid. Possibly related to PageBundle
 ResolvedakosiarisT358752 Reimage parse* hosts as kubernetes nodes
 ResolveddancyT357402 Scap should check errors coming from mw-on-k8s canaries during deployments
 ResolvedClement_GoubertT357508 Move 60% of mediawiki external requests to mw on k8s
 ResolvedClement_GoubertT358117 Adapt scap's testing strategy to mw-on-k8s
 ResolvedClement_GoubertT357507 Move 50% of mediawiki external requests to mw on k8s
 ResolvedClement_GoubertT360763 Move 70% of mediawiki external requests to mw on k8s
 OpenClement_GoubertT362323 Move 100% of external traffic to Kubernetes
 ResolvedCDanisT367894 update status page latency for mw-on-k8s
 OpenjijikiT277711 Memcached, mcrouter in MediaWiki on Kubernetes
 ResolvedJoeT278220 Define the size of a pod for mediawiki in terms of resource usage
 In ProgressjijikiT346690 mcrouter daemonset on mw-on-k8s
 ResolvedjijikiT362766 2024-04-17 mw-on-k8s eqiad outage
 OpenNoneT363186 Cache mw-mcrouter service ClusterIP in apcu cache
 ResolvedJdforrester-WMFT362662 Rename X-Wikimedia-Debug k8s-experimental option
 ResolvedjijikiT365478 XWD: Allow choosing datacentre in k8s-mwdebug
 OpenClement_GoubertT367949 Turn down api_appserver and appserver clusters
 ResolvedClement_GoubertT368058 Set all appservers to pooled=inactive in scap
 OpenJdforrester-WMFT335766 Migrate Quibble images from buster to bullseye
 OpenJdforrester-WMFT366799 Quibble jobs fail on bullseye images in QUnit job: "Multiple targets are not supported"

Event Timeline

elukey created this task.Tue, Jun 25, 8:52 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Jun 25, 8:52 AM
elukey added a subtask: T368359: Upgrade Knative control plane Docker images to Bullseye/Bookworm.Tue, Jun 25, 8:53 AM
elukey added a subtask: T367427: Cleanup old Docker images running Debian Stretch/Jessie.Tue, Jun 25, 8:58 AM
MoritzMuehlenhoff subscribed.Tue, Jun 25, 11:10 AM
Jdforrester-WMF subscribed.Tue, Jun 25, 11:31 AM

Dupe of T362981, I thik.

Jdforrester-WMF added a subtask: T362981: Migrate mw-on-k8s base image from buster to bullseye.Tue, Jun 25, 11:33 AM
elukey added a comment.Tue, Jun 25, 12:29 PM
In T368366#9921269, @Jdforrester-WMF wrote:

Dupe of T362981, I thik.

We can keep T362981 as subtask, the other images do need to be migrate as well.

Jdforrester-WMF removed a subtask: T362981: Migrate mw-on-k8s base image from buster to bullseye.Tue, Jun 25, 3:01 PM
Jdforrester-WMF added a subtask: T356293: Migrate MW appservers' base images to bullseye.
gerritbot added a comment.Tue, Jun 25, 3:04 PM

Change #1049577 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/production-images@master] coredns: upgrade to Bookworm

https://gerrit.wikimedia.org/r/1049577

gerritbot added a project: Patch-For-Review.Tue, Jun 25, 3:04 PM

Change #1049578 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/production-images@master] envoy: upgrade to Bookworm

https://gerrit.wikimedia.org/r/1049578

gerritbot added a comment.Tue, Jun 25, 3:18 PM

Change #1049586 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/production-images@master] helm-state-metrics: upgrade to Bookworm

https://gerrit.wikimedia.org/r/1049586

gerritbot added a comment.Tue, Jun 25, 3:18 PM

Change #1049587 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/production-images@master] mcrouter: upgrade to Bookworm

https://gerrit.wikimedia.org/r/1049587

gerritbot added a comment.Tue, Jun 25, 3:18 PM

Change #1049588 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/production-images@master] prometheus-exporters: upgrade mcrouter and statsd to Bookworm

https://gerrit.wikimedia.org/r/1049588

gerritbot added a comment.Tue, Jun 25, 3:39 PM

Change #1049590 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/production-images@master] service-checker: upgrade to Bookworm

https://gerrit.wikimedia.org/r/1049590

gerritbot added a comment.Tue, Jun 25, 3:39 PM

Change #1049591 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/production-images@master] nutcracker: upgrade to Bookworm

https://gerrit.wikimedia.org/r/1049591

gerritbot added a comment.Wed, Jun 26, 7:58 AM

Change #1049825 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/software/cfssl-issuer@main] Makefile: use 'go install' instead of 'go get'

https://gerrit.wikimedia.org/r/1049825

gerritbot added a comment.Wed, Jun 26, 8:08 AM

Change #1049828 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/production-images@master] echoserver: upgrade to Bookworm

https://gerrit.wikimedia.org/r/1049828

gerritbot added a comment.Wed, Jun 26, 8:26 AM

Change #1049825 merged by Elukey:

[operations/software/cfssl-issuer@main] Makefile: use 'go install' instead of 'go get'

https://gerrit.wikimedia.org/r/1049825

gerritbot added a comment.Wed, Jun 26, 8:31 AM

Change #1049838 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/production-images@master] cfssl-issuer: upgrade to Bookworm

https://gerrit.wikimedia.org/r/1049838

gerritbot added a comment.Wed, Jun 26, 3:44 PM

Change #1049577 merged by Elukey:

[operations/docker-images/production-images@master] coredns: upgrade to Bookworm

https://gerrit.wikimedia.org/r/1049577

gerritbot added a comment.Wed, Jun 26, 3:45 PM

Change #1049578 merged by Elukey:

[operations/docker-images/production-images@master] envoy: upgrade to Bookworm

https://gerrit.wikimedia.org/r/1049578

gerritbot added a comment.Wed, Jun 26, 3:45 PM

Change #1049586 merged by Elukey:

[operations/docker-images/production-images@master] helm-state-metrics: upgrade to Bookworm

https://gerrit.wikimedia.org/r/1049586

gerritbot added a comment.Thu, Jun 27, 2:46 PM

Change #1049588 merged by Elukey:

[operations/docker-images/production-images@master] prometheus-exporters: upgrade mcrouter and statsd to Bookworm

https://gerrit.wikimedia.org/r/1049588

gerritbot added a comment.Thu, Jun 27, 2:47 PM

Change #1049590 merged by Elukey:

[operations/docker-images/production-images@master] service-checker: upgrade to Bookworm

https://gerrit.wikimedia.org/r/1049590

gerritbot added a comment.Thu, Jun 27, 2:47 PM

Change #1049591 merged by Elukey:

[operations/docker-images/production-images@master] nutcracker: upgrade to Bookworm

https://gerrit.wikimedia.org/r/1049591

gerritbot added a comment.Thu, Jun 27, 2:47 PM

Change #1049828 merged by Elukey:

[operations/docker-images/production-images@master] echoserver: upgrade to Bookworm

https://gerrit.wikimedia.org/r/1049828

gerritbot added a comment.Thu, Jun 27, 2:47 PM

Change #1049838 merged by Elukey:

[operations/docker-images/production-images@master] cfssl-issuer: upgrade to Bookworm

https://gerrit.wikimedia.org/r/1049838

gerritbot added a comment.Fri, Jun 28, 9:32 AM

Change #1050568 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] admin_ng: upgrade coredns to 1.8.7-2

https://gerrit.wikimedia.org/r/1050568

gerritbot added a comment.Fri, Jun 28, 9:32 AM

Change #1050569 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] admin_ng: upgrade cfssl-issuer's Docker image

https://gerrit.wikimedia.org/r/1050569

gerritbot added a comment.Fri, Jun 28, 9:32 AM

Change #1050570 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] api,rest-gateway: upgrade Envoy version

https://gerrit.wikimedia.org/r/1050570

gerritbot added a comment.Fri, Jun 28, 9:32 AM

Change #1050571 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] admin_ng: update helm-state-metrics' Docker image version

https://gerrit.wikimedia.org/r/1050571

gerritbot added a comment.Fri, Jun 28, 12:35 PM

Change #1049587 merged by Elukey:

[operations/docker-images/production-images@master] mcrouter: upgrade to Bookworm

https://gerrit.wikimedia.org/r/1049587

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits