Page MenuHomePhabricator

docker-pkg is unhappy on contint1001
Closed, ResolvedPublic

Description

on contint1001:

/srv/deployment/docker-pkg/venv/bin/docker-pkg --debug -c /etc/docker-pkg/integration.yaml /etc/zuul/wikimedia/dockerfiles

2019-04-01 11:32:52,684 [docker-pkg-build] INFO - Step 1/3 : FROM docker-registry.discovery.wmnet/releng/ci-stretch:0.1.4 (image.py:176)
2019-04-01 11:32:52,684 [docker-pkg-build] INFO -  (image.py:176)
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.4/logging/__init__.py", line 978, in emit
    msg = self.format(record)
  File "/usr/lib/python3.4/logging/__init__.py", line 828, in format
    return fmt.format(record)
  File "/usr/lib/python3.4/logging/__init__.py", line 565, in format
    record.message = record.getMessage()
  File "/usr/lib/python3.4/logging/__init__.py", line 328, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "/srv/deployment/docker-pkg/venv/bin/docker-pkg", line 11, in <module>
    sys.exit(main())
  File "/srv/deployment/docker-pkg/venv/lib/python3.4/site-packages/docker_pkg/cli.py", line 89, in main
    build(application, log_to_stdout)
  File "/srv/deployment/docker-pkg/venv/lib/python3.4/site-packages/docker_pkg/cli.py", line 106, in build
    img.build()
  File "/srv/deployment/docker-pkg/venv/lib/python3.4/site-packages/docker_pkg/builder.py", line 99, in build
    if self.image.build():
  File "/srv/deployment/docker-pkg/venv/lib/python3.4/site-packages/docker_pkg/image.py", line 287, in build
    super().build(self.build_path)
  File "/srv/deployment/docker-pkg/venv/lib/python3.4/site-packages/docker_pkg/image.py", line 200, in build
    stream_to_log(image_logger, line)
  File "/srv/deployment/docker-pkg/venv/lib/python3.4/site-packages/docker_pkg/image.py", line 173, in stream_to_log
    logger.error('Build failed: $s', error_msg)
Message: 'Build failed: $s'
Arguments: ('Get https://docker-registry.discovery.wmnet/v2/releng/ci-stretch/manifests/0.1.4: no basic auth credentials',)
2019-04-01 11:32:52,694 [docker-pkg-build] ERROR - Building image docker-registry.discovery.wmnet/releng/sury-php:0.3.0 failed - check your Dockerfile: Building image docker-registry.discovery.wmnet/releng/sury-php:0.3.0 failed (image.py:290)
Traceback (most recent call last):
  File "/srv/deployment/docker-pkg/venv/lib/python3.4/site-packages/docker_pkg/image.py", line 287, in build
    super().build(self.build_path)
  File "/srv/deployment/docker-pkg/venv/lib/python3.4/site-packages/docker_pkg/image.py", line 200, in build
    stream_to_log(image_logger, line)
  File "/srv/deployment/docker-pkg/venv/lib/python3.4/site-packages/docker_pkg/image.py", line 174, in stream_to_log
    raise docker.errors.BuildError('Building image {} failed'.format(self.image))
docker.errors.BuildError: Building image docker-registry.discovery.wmnet/releng/sury-php:0.3.0 failed
2019-04-01 11:32:52,694 [docker-pkg-build] INFO - Removing build context /tmp/docker-pkg-sury-phppkkkhagy (image.py:359)
 ERROR: image docker-registry.discovery.wmnet/releng/sury-php failed to build, see logs for details

Event Timeline

hashar created this task.Apr 1 2019, 11:40 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 1 2019, 11:40 AM

So the root cause is:

Get https://docker-registry.discovery.wmnet/v2/releng/ci-stretch/manifests/0.1.4: no basic auth credentials'

There is also a concern with logger.error complaining with not all arguments converted during string formatting

    logger.error('Build failed: $s', error_msg)
Message: 'Build failed: $s'
Arguments: ('Get https://docker-registry.discovery.wmnet/v2/releng/ci-stretch/manifests/0.1.4: no basic auth credentials',)

I guess because error_msg is not just a string but a tuple?:!?

Change 500429 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/docker-images/docker-pkg@master] Fix build error logging $s -> %s

https://gerrit.wikimedia.org/r/500429

Message: 'Build failed: $s'
                       ^^^

Must be a percentage sign, not a dollar!

Mentioned in SAL (#wikimedia-operations) [2019-04-01T12:02:10Z] <oblivian@deploy1001> Started deploy [docker-pkg/deploy@UNKNOWN]: Rollback to 1.0.0, T219778

Mentioned in SAL (#wikimedia-operations) [2019-04-01T12:02:44Z] <oblivian@deploy1001> Finished deploy [docker-pkg/deploy@UNKNOWN]: Rollback to 1.0.0, T219778 (duration: 00m 34s)

Mentioned in SAL (#wikimedia-operations) [2019-04-01T12:08:30Z] <oblivian@deploy1001> Started deploy [docker-pkg/deploy@0c32dc1]: Rollback to 1.0.0, T219778

Mentioned in SAL (#wikimedia-operations) [2019-04-01T12:08:48Z] <oblivian@deploy1001> Finished deploy [docker-pkg/deploy@0c32dc1]: Rollback to 1.0.0, T219778 (duration: 00m 18s)

Joe added a subscriber: Joe.Apr 1 2019, 12:26 PM

The underlying issue is that we're trying to pull from a private registry (since we're contacting the internal registry there) and we don't provide credentials IIRC.

It should be relatively easy to fix.

Change 500429 merged by jenkins-bot:
[operations/docker-images/docker-pkg@master] Fix build error logging $s -> %s

https://gerrit.wikimedia.org/r/500429

Change 500445 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/docker-images/docker-pkg@master] Log in to the registry if credentials are provided

https://gerrit.wikimedia.org/r/500445

Change 500445 merged by jenkins-bot:
[operations/docker-images/docker-pkg@master] Log in to the registry if credentials are provided

https://gerrit.wikimedia.org/r/500445

Mentioned in SAL (#wikimedia-operations) [2019-04-02T05:58:19Z] <oblivian@deploy1001> Started deploy [docker-pkg/deploy@2a090ef]: New version for T219778

Mentioned in SAL (#wikimedia-operations) [2019-04-02T05:58:38Z] <oblivian@deploy1001> Finished deploy [docker-pkg/deploy@2a090ef]: New version for T219778 (duration: 00m 19s)

hashar closed this task as Resolved.Apr 2 2019, 11:31 AM
hashar assigned this task to Joe.

Tentatively resolved. Thank you very much @Joe