Maniphest T251918

docker-reporter-releng-images failed on deneb
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	colewhite
	May 5 2020, 4:29 PM

Description

May 04 16:23:52 deneb docker-report-releng[32224]: INFO[docker-report] Running: image pruning
May 04 16:23:53 deneb docker-report-releng[32224]: Detailer results:
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/castor:0.2.4                     [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/ci-buster:0.1.0                  [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/ci-jessie:0.4.1                  [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/ci-src-setup:0.3.1-s3            [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/ci-src-setup-simple:0.2.1-s1     [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/ci-stretch:0.1.5                 [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/civicrm:0.1.1-s4                 [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/commit-message-validator:0.6.1-s1[OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer:0.1.10-s3               [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-hhvm:0.2.6-s2           [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-package:0.1.6-s4        [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-package-hhvm:0.2.5-s3   [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-package-php70:0.2.0-s1  [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-package-php71:0.2.0     [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-package-php72:0.3.0-s1  [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-package-php73:0.2.0-s1  [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-package-php74:0.1.0-s1  [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-php56:0.2.0-s1          [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-php70:0.2.0-s1          [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-php71:0.2.0             [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-php72:0.3.0-s1          [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-php73:0.2.0-s1          [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-php74:0.1.0-s1          [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-test:0.1.7-s3           [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-test-hhvm:0.2.6-s2      [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-test-php56:0.2.0-s1     [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-test-php70:0.2.0-s1     [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/composer-test-php72:0.3.0-s1     [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/curl:0.1.0-s1                    [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/doxygen:0.6.1-s1                 [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/gerrit:0.1.0-s7                  [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/gradle:0.3.0-s1                  [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/helm-linter:0.2.0                [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/hhvm:0.1.3-s1                    [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/hhvm-compile:0.1.0-s1            [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/jar-updater:0.0.7                [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/java8:0.6.3                      [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/java8-mjolnir:0.3.0              [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/java8-sonar-scanner:0.6.0        [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/java8-wikidata-query-rdf:0.4.0   [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/java8-xgboost:0.3.0              [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/jsduck:0.1.0-s1                  [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/lintr:0.2.1                      [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/mediawiki-phan:0.5.3-s10         [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/mediawiki-phan-seccheck:0.4.0-s12[OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/mediawiki-phan-testrun:0.1.5-s3  [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/mediawiki-phpcs:0.1.0            [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/mediawiki-phpcs-dryrun:0.1.1     [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/mediawiki-tarball:0.2.4-s1       [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/node10:0.5.0-s1                  [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/node10-kartotherian:0.1.2        [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/node10-portals:0.1.2             [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/node10-test:0.6.0                [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/node10-test-browser:0.6.1        [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/node10-test-browser-php72-composer:0.1.3-s5[OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/npm:0.4.0                        [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/npm-browser-test:0.5.0           [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/npm-php:0.3.0                    [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/npm-stretch:0.2.0                [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/npm-test:0.7.0                   [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/npm-test-3d2png:0.3.0            [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/npm-test-graphoid:0.3.0          [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/npm-test-librdkafka:0.2.1-s1     [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/npm-test-maps-service:0.3.0      [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/npm-test-oojsui:0.2.2-s3         [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/npm-test-stretch:0.3.0           [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/npm6:0.1.1                       [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/npm6-browser-test:0.1.2          [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/npm6-test:0.2.0                  [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/operations-dnslint:0.0.6         [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/operations-puppet:0.7.1          [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/perl:5.24.1-s1                   [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/php:0.1.4-s1                     [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/php-ast:1.0.1-2-s2               [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/php-compile:0.1.2-s2             [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/php56:0.1.2                      [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/php70:0.1.6                      [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/php70-compile:0.1.4-s1           [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/php71:0.1.8                      [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/php71-compile:0.1.3-s5           [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/php72:0.2.7                      [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/php72-compile:0.2.1-s8           [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/php73:0.2.6                      [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/php73-compile:0.2.3-s5           [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/php74:0.1.1                      [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/php74-compile:0.0.4-s2           [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/phpmetrics:2.4.3-s8              [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/quibble-coverage:0.0.42          [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/quibble-fresnel:0.0.42           [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/quibble-jessie:0.0.31-1          [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/quibble-jessie-hhvm:0.0.31-1     [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/quibble-jessie-php56:0.0.31-1    [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/quibble-stretch:0.0.42           [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/quibble-stretch-bundle:0.0.42    [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/quibble-stretch-hhvm:0.0.35      [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/quibble-stretch-php70:0.0.42     [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/quibble-stretch-php71:0.0.42     [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/quibble-stretch-php72:0.0.42     [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/quibble-stretch-php73:0.0.42     [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/quibble-stretch-php74:0.0.42     [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/rake:0.2.1-s1                    [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/rake-poolcounter:0.1.0           [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/rake-vagrant:0.2.1-s1            [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/release-notes:0.0.6              [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/sury-php:0.3.0-s2                [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/tabs:0.0.2                       [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/tox:0.4.3                        [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/tox-acme-chief:0.4.0-s1          [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/tox-censorshipmonitoring:0.1.0   [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/tox-cergen:0.4.0-s1              [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/tox-certcentral:0.3.2            [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/tox-conftool:0.4.0-s1            [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/tox-eventlogging:0.4.0-s1        [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/tox-homer:0.1.0                  [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/tox-labs-striker:0.4.0-s1        [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/tox-mysqld:0.4.2                 [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/tox-poolcounter:0.4.0-s1         [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/tox-pyspark:0.5.1                [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/tox-pywikibot:0.4.0-s1           [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/typos:0.0.3-s1                   [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/wikimedia-audit-resources:0.1.2-s1[OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/zuul-cloner:0.2.0-s1             [OK]
May 04 16:23:53 deneb docker-report-releng[32224]: docker-registry.wikimedia.org/releng/bazel:0.4.0                    [FAIL]
May 04 16:23:53 deneb systemd[1]: docker-reporter-releng-images.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
May 04 16:23:53 deneb systemd[1]: docker-reporter-releng-images.service: Failed with result 'exit-code'.

Details

Subject	Repo	Branch	Lines +/-
docker-reporter: filter out old removed images	operations/puppet	production	+6 -0
P:docker: update filter file	operations/puppet	production	+1 -1
R:systemd::timer::job: use the syslog_identier for the syslog programname	operations/puppet	production	+5 -1
P:docker::reporter: exclude all jessie images	operations/puppet	production	+5 -1
postints: update postinstall to use /etc/os-release	operations/software/debmonitor	debian	+15 -14
Manually pull the image before creating a container	operations/docker-images/docker-report	master	+34 -5
Check if images are debian based before generating report	operations/docker-images/docker-report	master	+83 -13

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		JMeybohm	T251918 docker-reporter-releng-images failed on deneb
		Open		None	T284539 Update docker-reporter to only check images available in the respective repos

Event Timeline

colewhite created this task.May 5 2020, 4:29 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 5 2020, 4:29 PM

colewhite triaged this task as Medium priority.May 5 2020, 4:29 PM

@MoritzMuehlenhoff filed a similar task previously.

Do you have details as to why the releng/bazel:0.4.0 fails?

hashar added a project: Continuous-Integration-Config.May 5 2020, 5:51 PM

Jdforrester-WMF subscribed.May 5 2020, 7:16 PM

In T251918#6109976, @hashar wrote:

Do you have details as to why the releng/bazel:0.4.0 fails?

No, what's I opened a task so that we can figure it out :-)

ACKing the alert https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=deneb&service=Check+systemd+state with this task.

ayounsi@deneb:~$ sudo systemctl list-units --state=failed
  UNIT                                  LOAD   ACTIVE SUB    DESCRIPTION                         
● docker-reporter-releng-images.service loaded failed failed Report on upgrades to releng images.

hashar removed a project: Continuous-Integration-Config.May 26 2020, 9:48 AM

06:36 mutante: deneb - systemctl start docker-reporter-releng-images

06:37 <+icinga-wm> RECOVERY - Check systemd state on deneb is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state

ayounsi unsubscribed.May 29 2020, 7:49 AM

jbond subscribed.Jul 1 2020, 10:38 AM

The service failed 3 days ago due to another image this time:

root@deneb:~# journalctl -u docker-reporter-releng-images.service | grep FAIL
Jul 06 16:54:38 deneb docker-report-releng[27623]: docker-registry.wikimedia.org/releng/ci-common:0.2                  [FAIL]

That releng/ci-common image is a scratch image containing scripts shared by our base images ci-jessie, ci-stretch, ci-buster. It does not have any Debian OS layer, thus if the reporter attempts to run a command in it, that will fail since .. there is nothing in the image.

Maybe it is not a good pattern to use :/

In T251918#6296048, @hashar wrote:

That releng/ci-common image is a scratch image containing scripts shared by our base images ci-jessie, ci-stretch, ci-buster. It does not have any Debian OS layer, thus if the reporter attempts to run a command in it, that will fail since .. there is nothing in the image.

Maybe it is not a good pattern to use :/

Yeah. The tool actually tries to install and run debmonitor-client in the container which fails in that case.

Maybe we can just skip images that are not debian based?

In T251918#6296059, @JMeybohm wrote:

Maybe we can just skip images that are not debian based?

Sounds good, we could simply test for the presence of /etc/debian_version which is owned by the base-files package which has "Essential" priority and should be universally present on any image using Debian.

In T251918#6296062, @MoritzMuehlenhoff wrote:

Sounds good, we could simply test for the presence of /etc/debian_version which is owned by the base-files package which has "Essential" priority and should be universally present on any image using Debian.

Okay. Will prepare a patch.

Change 611251 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/docker-images/docker-report@master] Check if images are debian based before generating report

https://gerrit.wikimedia.org/r/611251

gerritbot added a project: Patch-For-Review.Jul 10 2020, 10:34 AM

Change 611251 merged by jenkins-bot:
[operations/docker-images/docker-report@master] Check if images are debian based before generating report

https://gerrit.wikimedia.org/r/611251

Maintenance_bot removed a project: Patch-For-Review.Jul 22 2020, 8:10 AM

Change 615411 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/docker-images/docker-report@master] Manually pull the image before creating a container

https://gerrit.wikimedia.org/r/615411

gerritbot added a project: Patch-For-Review.Jul 22 2020, 8:21 AM

Change 615411 merged by jenkins-bot:
[operations/docker-images/docker-report@master] Manually pull the image before creating a container

https://gerrit.wikimedia.org/r/615411

Jul 22 09:15:57 deneb docker-report-releng[3273]: INFO[docker-report] Building debmonitor report for docker-registry.wikimedia.org/releng/ci-common:0.2
Jul 22 09:15:58 deneb docker-report-releng[3273]: WARNING[docker-report] Unable to create a report for non debian images

Looking good now.

New failure. Here's the output:

ep 28 16:38:10 deneb docker-report-releng[23588]: INFO[docker-report] Building debmonitor report for docker-registry.wikimedia.org/releng/zuul-cloner:0.2.1-s3
Sep 28 16:38:17 deneb docker-report-releng[23588]: INFO[docker-report] Running: report generation
Sep 28 16:38:37 deneb docker-report-releng[23588]: INFO[docker-report] Running: submit report
Sep 28 16:38:38 deneb docker-report-releng[23588]: INFO[docker-report] Running: image pruning
Sep 28 16:38:39 deneb docker-report-releng[23588]: Detailer results:
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/bazel:0.4.0-s2                   [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/cassandra311:0.0.1               [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/castor:0.2.4                     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/ci-buster:0.2.2-s1               [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/ci-jessie:0.5.1-s1               [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/ci-src-setup:0.3.1-s3            [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/ci-src-setup-simple:0.4.2-s1     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/ci-stretch:0.3.0-s2              [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/civicrm:0.2.0-s2                 [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/commit-message-validator:0.6.1-s4[OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer:0.1.10-s3               [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-hhvm:0.2.6-s2           [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-package:0.1.6-s4        [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-package-hhvm:0.2.5-s3   [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-package-php70:0.2.0-s5  [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-package-php71:0.2.0     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-package-php72:0.3.1-s2  [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-package-php73:0.2.1-s2  [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-package-php74:0.1.1-s2  [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-php56:0.2.0-s2          [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-php70:0.3.0             [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-php71:0.2.0             [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-php72:0.4.0             [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-php73:0.2.0-s4          [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-php74:0.1.0-s4          [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-security-check:1.1      [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-test:0.1.7-s3           [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-test-hhvm:0.2.6-s2      [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-test-php56:0.2.0-s2     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-test-php70:0.2.0-s4     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-test-php72:0.3.1-s2     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/composer-test-php73:0.1.0-s2     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/curl:0.1.0-s3                    [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/doxygen:0.8.1-s1                 [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/gradle:0.3.0-s3                  [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/helm-linter:0.2.6-s1             [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/hhvm:0.1.3-s1                    [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/hhvm-compile:0.1.0-s1            [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/jar-updater:0.0.7-s1             [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/java8:0.6.4-s3                   [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/java8-mjolnir:0.3.0-s3           [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/java8-scala-spark:0.1.0-s3       [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/java8-sonar-scanner:0.6.2-s4     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/java8-wikidata-query-rdf:0.4.1-s3[OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/java8-xgboost:0.3.0-s3           [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/jsduck:0.1.0-s1                  [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/lintr:0.2.1-s3                   [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/logstash-filter-verifier:0.0.1-s1[OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/mediawiki-phan:0.8.1             [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/mediawiki-phan-php73:0.1.2       [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/mediawiki-phan-php74:0.1.1       [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/mediawiki-phan-seccheck:0.4.0-s13[OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/mediawiki-phan-testrun:0.2.0     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/mediawiki-phpcs:0.1.0            [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/mediawiki-phpcs-dryrun:0.1.1     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/mediawiki-tarball:0.2.4-s3       [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/node10:0.5.2-s2                  [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/node10-kartotherian:0.1.2-s4     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/node10-portals:0.1.3-s2          [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/node10-test:0.6.1-s2             [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/node10-test-browser:0.6.3-s2     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/node10-test-browser-php72-composer:0.1.4-s3[OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/npm:0.4.0-s1                     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/npm-browser-test:0.5.1-s2        [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/npm-php:0.3.0-s2                 [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/npm-stretch:0.2.0-s2             [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/npm-test:0.7.1-s1                [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/npm-test-3d2png:0.3.0-s2         [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/npm-test-graphoid:0.3.0-s2       [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/npm-test-librdkafka:0.2.1-s1     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/npm-test-maps-service:0.3.0-s2   [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/npm-test-oojsui:0.2.2-s3         [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/npm-test-stretch:0.3.0-s3        [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/npm6:0.1.1                       [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/npm6-browser-test:0.1.2          [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/npm6-test:0.2.0                  [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/operations-dnslint:0.0.8         [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/operations-puppet:0.7.6-s2       [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/perl:5.24.1-s3                   [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/php:0.1.4-s1                     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/php-ast:1.0.6-s2                 [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/php-compile:0.1.2-s2             [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/php56:0.1.2                      [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/php70:0.1.6-s2                   [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/php70-compile:0.1.4-s3           [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/php71:0.1.8-s2                   [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/php71-compile:0.1.3-s7           [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/php72:0.2.7-s2                   [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/php72-compile:0.2.1-s10          [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/php73:0.2.6-s2                   [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/php73-compile:0.2.3-s7           [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/php74:0.1.1-s2                   [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/php74-compile:0.0.4-s4           [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/phpmetrics:2.4.3-s11             [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/quibble-buster:0.0.45            [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/quibble-buster-php72:0.0.45      [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/quibble-coverage:0.0.45          [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/quibble-fresnel:0.0.45           [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/quibble-jessie:0.0.31-1          [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/quibble-jessie-hhvm:0.0.31-1     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/quibble-jessie-php56:0.0.31-1    [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/quibble-stretch:0.0.45           [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/quibble-stretch-bundle:0.0.45    [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/quibble-stretch-hhvm:0.0.35      [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/quibble-stretch-php70:0.0.45     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/quibble-stretch-php71:0.0.45     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/quibble-stretch-php72:0.0.45     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/quibble-stretch-php73:0.0.45     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/quibble-stretch-php74:0.0.45     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/rake:0.2.2-s1                    [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/rake-poolcounter:0.1.0           [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/rake-vagrant:0.2.1-s5            [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/release-notes:0.0.6-s2           [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/rust:1.46.0-1                    [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/rust-coverage:0.1.3-s2           [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/sury-php:0.3.0-s4                [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/tabs:0.0.2                       [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/tox:0.4.4-s3                     [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/tox-acme-chief:0.5.1-s3          [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/tox-censorshipmonitoring:0.1.0-s3[OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/tox-cergen:0.4.0-s4              [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/tox-certcentral:0.3.2            [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/tox-conftool:0.4.0-s4            [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/tox-eventlogging:0.4.0-s4        [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/tox-homer:0.1.0-s3               [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/tox-labs-striker:0.4.0-s4        [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/tox-mysqld:0.5.0-s2              [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/tox-poolcounter:0.4.0-s4         [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/tox-pyspark:0.6.0-s3             [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/tox-pywikibot:0.4.0-s4           [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/typos:0.0.3-s3                   [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/wikimedia-audit-resources:0.1.2-s3[OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/zuul-cloner:0.2.1-s3             [OK]
Sep 28 16:38:39 deneb docker-report-releng[23588]: docker-registry.wikimedia.org/releng/gerrit:0.1.0-s10               [FAIL]

This was some kind of connection issue to a apt repo while installing debmonitor-cli:

Sep 28 14:51:47 deneb docker-report-releng[23588]: INFO[docker-report] Building debmonitor report for docker-registry.wikimedia.org/releng/gerrit:0.1.0-s10
Sep 28 14:52:29 deneb docker-report-releng[23588]: INFO[docker-report] Running: report generation
Sep 28 14:58:34 deneb docker-report-releng[23588]: ERROR[docker-report] Report generation exited with exit code 100. Output:
Sep 28 14:58:34 deneb docker-report-releng[23588]: ERROR[docker-report] Get:1 http://security.debian.org stretch/updates InRelease [53.0 kB]
Sep 28 14:58:34 deneb docker-report-releng[23588]: Ign:2 http://mirrors.wikimedia.org/debian stretch InRelease
Sep 28 14:58:34 deneb docker-report-releng[23588]: Get:3 http://apt.wikimedia.org/wikimedia stretch-wikimedia InRelease [105 kB]
Sep 28 14:58:34 deneb docker-report-releng[23588]: Get:4 http://mirrors.wikimedia.org/debian stretch-updates InRelease [93.6 kB]
Sep 28 14:58:34 deneb docker-report-releng[23588]: Get:5 http://security.debian.org stretch/updates/main amd64 Packages [759 kB]
Sep 28 14:58:34 deneb docker-report-releng[23588]: Get:6 http://mirrors.wikimedia.org/debian stretch Release [118 kB]
Sep 28 14:58:34 deneb docker-report-releng[23588]: Get:7 http://mirrors.wikimedia.org/debian stretch Release.gpg [2,410 B]
Sep 28 14:58:34 deneb docker-report-releng[23588]: Get:8 http://apt.wikimedia.org/wikimedia stretch-wikimedia/component/git amd64 Packages [5,010 B]
Sep 28 14:58:34 deneb docker-report-releng[23588]: Get:9 http://apt.wikimedia.org/wikimedia stretch-wikimedia/main amd64 Packages [129 kB]
Sep 28 14:58:34 deneb docker-report-releng[23588]: Get:10 http://storage.googleapis.com/bazel-apt stable InRelease
Sep 28 14:58:34 deneb docker-report-releng[23588]: Get:11 http://mirrors.wikimedia.org/debian stretch-updates/main amd64 Packages [2,711 B]
Sep 28 14:58:34 deneb docker-report-releng[23588]: Get:12 http://apt.wikimedia.org/wikimedia stretch-wikimedia/component/node10 amd64 Packages [2,293 B]
Sep 28 14:58:34 deneb docker-report-releng[23588]: Get:13 http://mirrors.wikimedia.org/debian stretch/main amd64 Packages [9,610 kB]
Sep 28 14:58:34 deneb docker-report-releng[23588]: Ign:10 http://storage.googleapis.com/bazel-apt stable InRelease
Sep 28 14:58:34 deneb docker-report-releng[23588]: Get:14 http://storage.googleapis.com/bazel-apt stable Release [1,374 B]
Sep 28 14:58:34 deneb docker-report-releng[23588]: Get:15 http://storage.googleapis.com/bazel-apt stable Release.gpg [566 B]
Sep 28 14:58:34 deneb docker-report-releng[23588]: Get:16 http://storage.googleapis.com/bazel-apt stable/jdk1.8 amd64 Packages [4,567 B]
Sep 28 14:58:34 deneb docker-report-releng[23588]: Ign:16 http://storage.googleapis.com/bazel-apt stable/jdk1.8 amd64 Packages
Sep 28 14:58:34 deneb docker-report-releng[23588]: Get:16 http://storage.googleapis.com/bazel-apt stable/jdk1.8 amd64 Packages [33.3 kB]
Sep 28 14:58:34 deneb docker-report-releng[23588]: Err:16 http://storage.googleapis.com/bazel-apt stable/jdk1.8 amd64 Packages
Sep 28 14:58:34 deneb docker-report-releng[23588]:   Connection timed out
Sep 28 14:58:34 deneb docker-report-releng[23588]: Fetched 10.9 MB in 6min 2s (30.0 kB/s)
Sep 28 14:58:34 deneb docker-report-releng[23588]: Reading package lists...
Sep 28 14:58:34 deneb docker-report-releng[23588]: E: Failed to fetch http://storage.googleapis.com/bazel-apt/dists/stable/jdk1.8/binary-amd64/Packages  Connection timed out
Sep 28 14:58:34 deneb docker-report-releng[23588]: E: Some index files failed to download. They have been ignored, or old ones used instead.
Sep 28 14:58:34 deneb docker-report-releng[23588]: ERROR[docker-report] Debmonitor report for image docker-registry.wikimedia.org/releng/gerrit:0.1.0-s10 failed

Tentatively closing this again as the previous and next run went okay.

2021-06-03 19:19:30 3d 2h 47m 35s 3/3 CRITICAL - degraded: The following units failed: docker-reporter-releng-images.service

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=deneb&service=Check+systemd+state

Mentioned in SAL (#wikimedia-operations) [2021-06-03T19:33:28Z] <mutante> [deneb:~] $ sudo systemctl start docker-reporter-releng-images - T251918 - icinga-wm> RECOVERY - Check systemd state on deneb is OK

May 31 16:02:36 deneb docker-report-releng[31493]: ERROR[docker-report] Debmonitor report for image docker-registry.wikimedia.org/releng/quibble-jessie-php56:0.0.31-1 failed

Change 698164 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] P:docker::reporter: exclude all debian images

https://gerrit.wikimedia.org/r/698164

jbond reopened this task as Open.Jun 4 2021, 10:02 AM

jbond added a project: User-jbond.

jbond moved this task from Unsorted 💣 to Active 🚁 on the User-jbond board.

now seeing lsb_release: command not found on the docker buster instances

var/lib/dpkg/info/debmonitor-client.postinst: line 16: lsb_release: command not found                                                                                      
dpkg: error processing package debmonitor-client (--configure):                                        
 installed debmonitor-client package post-installation script subprocess returned error exit status 127                                                                    
Processing triggers for libc-bin (2.28-10) ...                                                            
Processing triggers for mime-support (3.62) ...                                                     
Errors were encountered while processing:                                                      
 debmonitor-client

In T251918#7133852, @jbond wrote:

now seeing lsb_release: command not found on the docker buster instances

var/lib/dpkg/info/debmonitor-client.postinst: line 16: lsb_release: command not found                                                                                      
dpkg: error processing package debmonitor-client (--configure):                                        
 installed debmonitor-client package post-installation script subprocess returned error exit status 127                                                                    
Processing triggers for libc-bin (2.28-10) ...                                                            
Processing triggers for mime-support (3.62) ...                                                     
Errors were encountered while processing:                                                      
 debmonitor-client

This is a packaging error I guess. The script tries to install debmonitor-client inside of the container.

Change 698168 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/software/debmonitor@debian] postints: update postinstall to check for lsb_release before using it

https://gerrit.wikimedia.org/r/698168

This is a packaging error I guess. The script tries to install debmonitor-client inside of the container.

Exactly, see https://gerrit.wikimedia.org/r/698168

jbond moved this task from Active 🚁 to Patch for Review on the User-jbond board.Jun 4 2021, 10:25 AM

should we ensure lsb_release is installed in the base images?

Change 698168 merged by Jbond:

[operations/software/debmonitor@debian] postints: update postinstall to use /etc/os-release

https://gerrit.wikimedia.org/r/698168

In T251918#7133990, @Joe wrote:

should we ensure lsb_release is installed in the base images?

I don't think we should. If it is really needed for something is should be installed as dependency. For this particular issue the way @jbond factored out the use of lsb_release seems pretty fine.

I now see an error with docker-registry.wikimedia.org/kubernetes-fluentd-daemonset:0.0.1-20190122

E: Failed to fetch http://mirrors.wikimedia.org/debian/dists/stretch/main/binary-amd64/Packages  Unsupported proxy configured: \http://webproxy.eqiad.wmnet:8080            
E: Failed to fetch http://mirrors.wikimedia.org/debian/dists/stretch-updates/main/binary-amd64/Packages  Unsupported proxy configured: \http://webproxy.eqiad.wmnet:8080    
E: Failed to fetch http://security.debian.org/dists/stretch/updates/main/binary-amd64/Packages  Unsupported proxy configured: \http://webproxy.eqiad.wmnet:8080             
E: Failed to fetch http://mirrors.wikimedia.org/debian/dists/stretch-backports/main/binary-amd64/Packages  Unsupported proxy configured: \http://webproxy.eqiad.wmnet:8080  
E: Failed to fetch http://apt.wikimedia.org/wikimedia/dists/stretch-wikimedia/main/binary-amd64/Packages  Unsupported proxy configured: \http://webproxy.eqiad.wmnet:8080

The issue is cased baceus the image has the following

/etc/apt/apt.conf.d/80_proxy

Acquire::http::Proxy \"http://webproxy.eqiad.wmnet:8080\";

which should instead be

/etc/apt/apt.conf.d/80proxy

Acquire::http::Proxy "http://webproxy.eqiad.wmnet:8080";

Looking at https://docker-registry.wikimedia.org/kubernetes-fluentd-daemonset/tags/ i see "Last updated at: 2021-06-04 12:32". however i cant find a definition in any of the repos i know of (integrations-config , production-images, toollabs-images) anyone able to provide some pointers to update this?

The "last updated" on that page means nothing - it barely tells you when the script ran the last time.

That image is old and should really be retired from the registry. I'm going for now to filter it out.

The "last updated" on that page means nothing - it barely tells you when the script ran the last time.

I was beginning to wonder :)

That image is old and should really be retired from the registry. I'm going for now to filter it out.

Thanks

Change 698164 merged by Jbond:

[operations/puppet@production] P:docker::reporter: exclude all jessie images

https://gerrit.wikimedia.org/r/698164

Change 698462 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] R:systemd::timer::job: use the syslog_identier for the syslog programname

https://gerrit.wikimedia.org/r/698462

Change 698462 merged by Jbond:

[operations/puppet@production] R:systemd::timer::job: use the syslog_identier for the syslog programname

https://gerrit.wikimedia.org/r/698462

We have another bunch of Release-Engineering-Team images failing, probably all due to being jessie based:

Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/ci-jessie:0.5.1-s1             [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/ci-src-setup:0.3.1-s3          [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/composer-php56:0.2.0-s2        [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/composer-test-php56:0.2.0-s2   [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/npm:0.4.0-s1                   [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/npm-test:0.7.1-s1              [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/npm-test-3d2png:0.3.0-s2       [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/npm-test-graphoid:0.3.0-s2     [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/npm-test-librdkafka:0.2.1-s1   [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/npm-test-maps-service:0.3.0-s2 [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/php56:0.1.2                    [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/quibble-jessie:0.0.31-1        [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/quibble-jessie-hhvm:0.0.31-1   [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/quibble-jessie-php56:0.0.31-1  [FAIL]

Can we remove all jessie-related containers from the registry?

In T251918#7138799, @JMeybohm wrote:

We have another bunch of Release-Engineering-Team images failing, probably all due to being jessie based:

Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/ci-jessie:0.5.1-s1             [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/ci-src-setup:0.3.1-s3          [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/composer-php56:0.2.0-s2        [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/composer-test-php56:0.2.0-s2   [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/npm:0.4.0-s1                   [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/npm-test:0.7.1-s1              [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/npm-test-3d2png:0.3.0-s2       [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/npm-test-graphoid:0.3.0-s2     [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/npm-test-librdkafka:0.2.1-s1   [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/npm-test-maps-service:0.3.0-s2 [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/php56:0.1.2                    [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/quibble-jessie:0.0.31-1        [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/quibble-jessie-hhvm:0.0.31-1   [FAIL]
Jun  7 11:28:11 deneb docker-report-releng[9839]: docker-registry.wikimedia.org/releng/quibble-jessie-php56:0.0.31-1  [FAIL]

Yeah, all those images have been dropped a while ago as part of our migration off jessie (final patch was 589eaacd in Feburary), but we have no means to remove things from the registry so they're still there. Please do remove them if that's possible.

In T251918#7138881, @MoritzMuehlenhoff wrote:

Can we remove all jessie-related containers from the registry?

Yeah, we "kind of" can: https://wikitech.wikimedia.org/wiki/Docker-registry#Deleting_images

Change 698583 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] docker-reporter: filter out old removed images

https://gerrit.wikimedia.org/r/698583

Change 698583 merged by Jbond:

[operations/puppet@production] docker-reporter: filter out old removed images

https://gerrit.wikimedia.org/r/698583

jbond added a subtask: T284539: Update docker-reporter to only check images available in the respective repos.Jun 8 2021, 8:22 AM

Mentioned in SAL (#wikimedia-operations) [2021-06-08T09:04:16Z] <jayme> removing docker-images from registry: releng/ci-jessie, releng/ci-src-setup, releng/composer-php56, releng/composer-test-php56, releng/npm, releng/npm-test, releng/npm-test-3d2png, releng/npm-test-graphoid, releng/npm-test-librdkafka, releng/npm-test-maps-service, releng/php56, releng/quibble-jessie, releng/quibble-jessie-hhvm, releng/quibble-jessie-php56 - T251918

+1 on the image that got deleted.

Note that we might have some images that got moved from Jessie, Stretch, Buster but having kept their name. I don't have an handy example, but there must be such a case.

Change 698763 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] P:docker: update filter file

https://gerrit.wikimedia.org/r/698763

Change 698763 merged by Jbond:

[operations/puppet@production] P:docker: update filter file

https://gerrit.wikimedia.org/r/698763

Thanks @jbond for looking after this. I'll bluntly close this task again now.

In T251918#7142275, @JMeybohm wrote:

Thanks @jbond for looking after this. I'll bluntly close this task again now.

thanks and can confirm the latest run completed successfully

Jun 08 14:06:20 deneb systemd[1]: docker-reporter-releng-images.service: Succeeded.

@JMeybohm the patch to filter still required considering https://phabricator.wikimedia.org/T251918#7141433 ?

In T251918#7144893, @jbond wrote:

@JMeybohm the patch to filter still required considering https://phabricator.wikimedia.org/T251918#7141433 ?

Oh, no. That should no longer be needed.

linked the wrong CR earlier i meant https://gerrit.wikimedia.org/r/698763, however assuming you saw past my error ill revert that now, thanks :)

hashar mentioned this in T292485: docker-reporter-releng-images => docker registry: status=3/NOTIMPLEMENTED.Oct 5 2021, 7:56 PM

docker-reporter-releng-images failed on denebClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

docker-reporter-releng-images failed on deneb
Closed, ResolvedPublic
Actions

Related Objects
Search...