Page MenuHomePhabricator

docker-report-releng failing on multiple image tags because of certificate validation error
Closed, ResolvedPublic

Description

We have multiple images failing since March 7 because the certificate of https://packages.sury.org can no longer be verified (with a stretch base image I guess):

Exemplarily error log:

Mar 28 14:23:22 deneb docker-report-releng[10680]: ERROR[docker-report] Report generation exited with exit code 100. Output:
Mar 28 14:23:22 deneb docker-report-releng[10680]: ERROR[docker-report] Get:1 http://security.debian.org stretch/updates InRelease [53.0 kB]
Mar 28 14:23:22 deneb docker-report-releng[10680]: Ign:2 http://mirrors.wikimedia.org/debian stretch InRelease
Mar 28 14:23:22 deneb docker-report-releng[10680]: Get:3 http://apt.wikimedia.org/wikimedia stretch-wikimedia InRelease [138 kB]
Mar 28 14:23:22 deneb docker-report-releng[10680]: Get:4 http://mirrors.wikimedia.org/debian stretch-updates InRelease [93.6 kB]
Mar 28 14:23:22 deneb docker-report-releng[10680]: Ign:5 https://packages.sury.org/php stretch InRelease
Mar 28 14:23:22 deneb docker-report-releng[10680]: Get:6 http://mirrors.wikimedia.org/debian stretch-backports InRelease [91.8 kB]
Mar 28 14:23:22 deneb docker-report-releng[10680]: Ign:7 https://packages.sury.org/php stretch Release
Mar 28 14:23:22 deneb docker-report-releng[10680]: Get:8 http://mirrors.wikimedia.org/debian stretch Release [118 kB]
Mar 28 14:23:22 deneb docker-report-releng[10680]: Ign:9 https://packages.sury.org/php stretch/main all Packages
Mar 28 14:23:22 deneb docker-report-releng[10680]: Get:10 http://mirrors.wikimedia.org/debian stretch Release.gpg [3,177 B]
Mar 28 14:23:22 deneb docker-report-releng[10680]: Ign:11 https://packages.sury.org/php stretch/main amd64 Packages
Mar 28 14:23:22 deneb docker-report-releng[10680]: Get:12 http://security.debian.org stretch/updates/main amd64 Packages [964 kB]
Mar 28 14:23:22 deneb docker-report-releng[10680]: Ign:9 https://packages.sury.org/php stretch/main all Packages
Mar 28 14:23:22 deneb docker-report-releng[10680]: Ign:11 https://packages.sury.org/php stretch/main amd64 Packages
Mar 28 14:23:22 deneb docker-report-releng[10680]: Ign:9 https://packages.sury.org/php stretch/main all Packages
Mar 28 14:23:22 deneb docker-report-releng[10680]: Ign:11 https://packages.sury.org/php stretch/main amd64 Packages
Mar 28 14:23:22 deneb docker-report-releng[10680]: Get:13 http://apt.wikimedia.org/wikimedia stretch-wikimedia/main amd64 Packages [137 kB]
Mar 28 14:23:22 deneb docker-report-releng[10680]: Ign:9 https://packages.sury.org/php stretch/main all Packages
Mar 28 14:23:22 deneb docker-report-releng[10680]: Ign:11 https://packages.sury.org/php stretch/main amd64 Packages
Mar 28 14:23:22 deneb docker-report-releng[10680]: Ign:9 https://packages.sury.org/php stretch/main all Packages
Mar 28 14:23:22 deneb docker-report-releng[10680]: Ign:11 https://packages.sury.org/php stretch/main amd64 Packages
Mar 28 14:23:22 deneb docker-report-releng[10680]: Ign:9 https://packages.sury.org/php stretch/main all Packages
Mar 28 14:23:22 deneb docker-report-releng[10680]: Err:11 https://packages.sury.org/php stretch/main amd64 Packages
Mar 28 14:23:22 deneb docker-report-releng[10680]:   server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
Mar 28 14:23:22 deneb docker-report-releng[10680]: Get:14 http://mirrors.wikimedia.org/debian stretch-backports/contrib amd64 Packages [12.0 kB]
Mar 28 14:23:22 deneb docker-report-releng[10680]: Get:15 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages [790 kB]
Mar 28 14:23:22 deneb docker-report-releng[10680]: Get:16 http://mirrors.wikimedia.org/debian stretch/main amd64 Packages [9,610 kB]
Mar 28 14:23:22 deneb docker-report-releng[10680]: Fetched 12.0 MB in 2s (4,471 kB/s)
Mar 28 14:23:22 deneb docker-report-releng[10680]: Reading package lists...
Mar 28 14:23:22 deneb docker-report-releng[10680]: W: The repository 'https://packages.sury.org/php stretch Release' does not have a Release file.
Mar 28 14:23:22 deneb docker-report-releng[10680]: E: Failed to fetch https://packages.sury.org/php/dists/stretch/main/binary-amd64/Packages  server certi
ficate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
Mar 28 14:23:22 deneb docker-report-releng[10680]: E: Some index files failed to download. They have been ignored, or old ones used instead.
Mar 28 14:23:22 deneb docker-report-releng[10680]: ERROR[docker-report] Debmonitor report for image docker-registry.wikimedia.org/releng/composer-package-
php71:0.2.0 failed

Affected images:

Mar 28 16:17:29 deneb docker-report-releng[10680]: docker-registry.wikimedia.org/releng/composer-package-php71:0.2.0   [FAIL]
Mar 28 16:17:29 deneb docker-report-releng[10680]: docker-registry.wikimedia.org/releng/composer-php71:0.2.0           [FAIL]
Mar 28 16:17:29 deneb docker-report-releng[10680]: docker-registry.wikimedia.org/releng/php71:0.1.8-s2                 [FAIL]
Mar 28 16:17:29 deneb docker-report-releng[10680]: docker-registry.wikimedia.org/releng/php71-compile:0.1.3-s7         [FAIL]

Can those tags just be removed from docker-registry (as they are potentially pretty old)?

Event Timeline

Those php71 images are obsolete, we have removed them from integration/config in October 2020 when we have dropped support for PHP 7.1. They can be deleted from the Docker registry.

JMeybohm closed this task as Resolved.EditedApr 14 2022, 8:12 AM
JMeybohm claimed this task.

I've removed the tags from the registry (https://wikitech.wikimedia.org/wiki/Docker-registry#Deleting_images) and triggered docker-reporter-releng-images.service