This task is to keep a reference of old Docker images deleted because running Debian Stretch/Jessie/Buster.
Description
Description
Details
Details
Related Changes in Gerrit:
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | elukey | T368366 Upgrade K8s docker images running in Wikimedia production on Buster to either Bullseye or Bookworm | |||
| Resolved | elukey | T367427 Cleanup old Docker images running Debian Stretch/Jessie/Buster | |||
| Resolved | elukey | T368744 Allow debmonitor to store the Debian version-id in the OS field |
Event Timeline
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-06-13T15:26:25Z] <elukey> drop mediawiki-services-parsoid docker images from the Docker Registry - T367427
Comment Actions
Dropped eventgate-ci as well (Andrew Otto confirmed that it is not used anymore since ages).
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2024-06-13T15:52:23Z] <elukey> drop mediawiki-services-restbase docker images from the Docker Registry - T367427
Comment Actions
Change #1043131 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] profile::docker::reporter: update k8s_rules.ini exclude list
Comment Actions
Change #1043131 merged by Elukey:
[operations/puppet@production] profile::docker::reporter: update k8s_rules.ini exclude list
Comment Actions
Change #1043780 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/software/debmonitor-client@master] cli: modify get_distro_name to return the version id
Comment Actions
Change #1043780 merged by jenkins-bot:
[operations/software/debmonitor-client@master] cli: modify get_distro_name to return the version id
Comment Actions
Next steps:
- package a new version of debmonitor-client with https://gerrit.wikimedia.org/r/1043780
- install the package on build2001, so that the Docker images report will be updated with Debian version numbers
- If everything goes fine, test the package on a few other nodes
- Fix the backend debmonitor service to accept Debian+version names
- Rollout the package everywhere.
After the above, we should be able to better identify and cleanup old Docker images.
Comment Actions
MariaDB [debmonitor]> insert into src_packages_os(id, name) values (3, 'Debian 10'), (4, 'Debian 11'), (5, 'Debian 12'); Query OK, 3 rows affected (0.001 sec) Records: 3 Duplicates: 0 Warnings: 0 MariaDB [debmonitor]> select * from src_packages_os; +----+-----------+ | id | name | +----+-----------+ | 1 | Debian | | 3 | Debian 10 | | 4 | Debian 11 | | 5 | Debian 12 | | 2 | Ubuntu | +----+-----------+ 5 rows in set (0.000 sec)
I've also installed debmonitor-client 0.4.0-1 (new version with https://gerrit.wikimedia.org/r/1043780) to build2001, so the next report should contain the improvement.
Comment Actions
Change #1049474 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] docker::reporter: remove Stretch/Jessie restrictions
Comment Actions
Change #1049474 abandoned by Elukey:
[operations/puppet@production] docker::reporter: remove Stretch/Jessie restrictions
Reason:
Will do manual runs instead.
Comment Actions
Found some images in the registry with "stretch" or "jessie" stated explicitly:
dev/stretch dev/stretch-apache2 dev/stretch-elasticsearch dev/stretch-php-sury dev/stretch-php72 dev/stretch-php72-apache2 dev/stretch-php72-fpm dev/stretch-php72-fpm-apache2 dev/stretch-php72-fpm-apache2-blubber dev/stretch-php72-fpm-apache2-xdebug dev/stretch-php72-jobrunner dev/stretch-php72-webserver dev/stretch-php72-webserver-xdebug dev/stretch-php73 dev/stretch-php73-fpm dev/stretch-php73-jobrunner dev/stretch-php74 dev/stretch-php74-fpm dev/stretch-php74-jobrunner dev/stretch-scap-deps golang-stretch python3-build-stretch releng/ci-stretch releng/npm-stretch releng/npm-test-stretch releng/scap-deps-stretch stretch wikimedia-stretch
python3-build-jessie releng/hhvm-jessie releng/hhvm-jessie-compile releng/quibble-jessie-php55
These are probably very old images that can be dropped, will ask confirmation before proceeding.
Comment Actions
The images based on Stretch under dev/ have been removed via T290532
For releng/ namespaces, that is the image for Zuul/CI and we have phased out Stretch via T278203 or Jessie T224908.
You can thus remove the Stretch and Jessie images under dev/ and releng/. Do not they might have child images that do not carry the Debian name in their name, which might be the reason the images did not end up being deleted.
For the other images, they are maintained by SRE service ops in https://gerrit.wikimedia.org/g/operations/docker-images/production-images
Comment Actions
+1 for dev/stretch*. The only one I don't know anything about is dev/stretch-scap-deps. Doesn't turn up in codesearch or GitLab search, at any rate, so probably safe.
Comment Actions
Thanks a lot for the feedback, all images in T367427#9921815 removed from the registry.
Comment Actions
Change #1049576 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/docker-images/production-images@master] config.yaml: remove wikimedia-stretch
Comment Actions
Next steps:
- Get a list of Docker images running Jessie/Stretch from the registry (somehow, not sure how to do it right now)
- Get the sign-off to drop old images if possible and drop them.
Comment Actions
Change #1049576 merged by Elukey:
[operations/docker-images/production-images@master] config.yaml: remove wikimedia-stretch
Comment Actions
Change #1049966 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/software/debmonitor@master] Allow to save new OS names without them being present on the DB
Comment Actions
Change #1051359 had a related patch set uploaded (by Elukey; author: Elukey):
[blubber-doc/example/calculator-service@master] blubber: upgrade to Bookworm
Comment Actions
Change #1051360 had a related patch set uploaded (by Elukey; author: Elukey):
[blubber-doc/example/helloworldoid@master] Upgrade to Nodejs-18
Comment Actions
Change #1051379 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] docker::reporter: update exclude rules
Comment Actions
Change #1051360 abandoned by Elukey:
[blubber-doc/example/helloworldoid@master] Upgrade to Nodejs-18
Reason:
dropping the image instead
Comment Actions
Change #1051359 abandoned by Elukey:
[blubber-doc/example/calculator-service@master] blubber: upgrade to Bookworm
Reason:
dropping the image instead
Comment Actions
Change #1051379 merged by Elukey:
[operations/puppet@production] docker::reporter: update exclude rules
Comment Actions
Next steps:
- Merge and package https://gerrit.wikimedia.org/r/c/operations/docker-images/docker-report/+/966200
- Run docker-report in a separate tmux without any exclude restriction, using as min supported os Debian 10.
- Collect the logs related to not supported images, those ones will be older than Debian 10 so to be deleted.
Comment Actions
Change #1054845 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/docker-images/docker-report@master] reporter.py: fix warning log
Comment Actions
Change #1054845 merged by jenkins-bot:
[operations/docker-images/docker-report@master] reporter.py: fix warning log
Comment Actions
Change #1055150 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] profile::docker::reporter: remove unnecessary filters
Comment Actions
Change #1055150 merged by Elukey:
[operations/puppet@production] profile::docker::reporter: remove unnecessary filters
Comment Actions
$ docker run -it --rm --entrypoint /bin/bash docker-registry.wikimedia.org/python3:0.0.2-20230423 Unable to find image 'docker-registry.wikimedia.org/python3:0.0.2-20230423' locally 0.0.2-20230423: Pulling from python3 4d0a7a49faa0: Already exists ac6ae5c76e41: Pull complete Digest: sha256:72f715d32fabdd90e7c43224fe2dfb2c4d38377b40d8826b104cedd1fb415225 Status: Downloaded newer image for docker-registry.wikimedia.org/python3:0.0.2-20230423 root@3450d8279668:/# cat /etc/debian_version 9.13
docker run -it --rm --entrypoint /bin/bash docker-registry.wikimedia.org/ruby:0.0.2-s1-20230423 Unable to find image 'docker-registry.wikimedia.org/ruby:0.0.2-s1-20230423' locally 0.0.2-s1-20230423: Pulling from ruby 4d0a7a49faa0: Already exists 9da14cd1e084: Pull complete Digest: sha256:29255228530234c0166c81b54b9a0dcbe5edfa1480b9eab9ff3ef4704187d731 Status: Downloaded newer image for docker-registry.wikimedia.org/ruby:0.0.2-s1-20230423 root@735b7253b4c4:/# cat /etc/debian_version 9.13
Comment Actions
Change #1056888 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/puppet@production] profile::docker::reporter::report: add min_debian_version arg
Comment Actions
Change #1056888 merged by Elukey:
[operations/puppet@production] profile::docker::reporter::report: add min_debian_version arg
Comment Actions
A lot of cleanup has been done, so far it seems that the task can be closed. We have a better way to figure out what images are not supported by the docker reporter now, so it is also good.