Page MenuHomePhabricator

docker-pkg: "certificate verify failed: unable to get local issuer certificate" for docker-registry.discovery.wmnet when publishing dev-images from contint2001
Closed, ResolvedPublic

Description

Tried to publish some dev-images changes for T262976 with the usual tox -e fabric -- deploy_docker. Got:

[contint.wikimedia.org] run: /srv/deployment/docker-pkg/venv/bin/docker-pkg --info -c /etc/docker-pkg/dev-images.yaml build /srv/dev-images/dockerfiles | tee dev-images-build.log
[contint.wikimedia.org] out: 2021-02-09 21:08:18,966 [docker-pkg-build] INFO - Processing the dockerfile template in /srv/dev-images/dockerfiles/stretch-php74-jobrunner (builder.py:294)
[contint.wikimedia.org] out: 2021-02-09 21:08:18,966 [docker-pkg-build] INFO - Processing the dockerfile template in /srv/dev-images/dockerfiles/stretch (builder.py:294)
[contint.wikimedia.org] out: 2021-02-09 21:08:18,969 [docker-pkg-build] INFO - Processing the dockerfile template in /srv/dev-images/dockerfiles/stretch-php72-webserver (builder.py:294)
[contint.wikimedia.org] out: 2021-02-09 21:08:18,971 [docker-pkg-build] INFO - Processing the dockerfile template in /srv/dev-images/dockerfiles/buster (builder.py:294)
[contint.wikimedia.org] out: 2021-02-09 21:08:18,972 [docker-pkg-build] INFO - Processing the dockerfile template in /srv/dev-images/dockerfiles/stretch-php72-fpm-apache2-xdebug (builder.py:294)
[contint.wikimedia.org] out: 2021-02-09 21:08:18,973 [docker-pkg-build] INFO - Processing the dockerfile template in /srv/dev-images/dockerfiles/stretch-php72-fpm-apache2 (builder.py:294)
[contint.wikimedia.org] out: 2021-02-09 21:08:18,975 [docker-pkg-build] INFO - Processing the dockerfile template in /srv/dev-images/dockerfiles/elasticsearch (builder.py:294)
[contint.wikimedia.org] out: 2021-02-09 21:08:18,977 [docker-pkg-build] INFO - Processing the dockerfile template in /srv/dev-images/dockerfiles/stretch-php74-fpm (builder.py:294)
[contint.wikimedia.org] out: 2021-02-09 21:08:19,060 [docker-pkg-build] ERROR - Could not load image in /srv/dev-images/dockerfiles/elasticsearch: HTTPSConnectionPool(host='docker-registry.discovery.wmnet', port=443): Max retries exceeded with url: /v2/dev/stretch-elasticsearch/manifests/0.0.1 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)'))) (builder.py:298)
[contint.wikimedia.org] out: Traceback (most recent call last):
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 594, in urlopen
[contint.wikimedia.org] out:     self._prepare_proxy(conn)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 805, in _prepare_proxy
[contint.wikimedia.org] out:     conn.connect()
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/connection.py", line 344, in connect
[contint.wikimedia.org] out:     ssl_context=context)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 344, in ssl_wrap_socket
[contint.wikimedia.org] out:     return context.wrap_socket(sock, server_hostname=server_hostname)
[contint.wikimedia.org] out:   File "/usr/lib/python3.7/ssl.py", line 412, in wrap_socket
[contint.wikimedia.org] out:     session=session
[contint.wikimedia.org] out:   File "/usr/lib/python3.7/ssl.py", line 853, in _create
[contint.wikimedia.org] out:     self.do_handshake()
[contint.wikimedia.org] out:   File "/usr/lib/python3.7/ssl.py", line 1117, in do_handshake
[contint.wikimedia.org] out:     self._sslobj.do_handshake()
[contint.wikimedia.org] out: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
[contint.wikimedia.org] out: 
[contint.wikimedia.org] out: During handling of the above exception, another exception occurred:
[contint.wikimedia.org] out: 
[contint.wikimedia.org] out: Traceback (most recent call last):
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
[contint.wikimedia.org] out:     timeout=timeout
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
[contint.wikimedia.org] out:     _stacktrace=sys.exc_info()[2])
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 398, in increment
[contint.wikimedia.org] out:     raise MaxRetryError(_pool, url, error or ResponseError(cause))
[contint.wikimedia.org] out: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='docker-registry.discovery.wmnet', port=443): Max retries exceeded with url: /v2/dev/stretch-elasticsearch/manifests/0.0.1 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)')))
[contint.wikimedia.org] out: 
[contint.wikimedia.org] out: During handling of the above exception, another exception occurred:
[contint.wikimedia.org] out: 
[contint.wikimedia.org] out: Traceback (most recent call last):
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/docker_pkg/builder.py", line 296, in _process_dockerfile_template
[contint.wikimedia.org] out:     return ImageFSM(root, self.client, self.config, self.nocache, self.pull)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/docker_pkg/builder.py", line 60, in __init__
[contint.wikimedia.org] out:     if self._is_published():
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/docker_pkg/builder.py", line 121, in _is_published
[contint.wikimedia.org] out:     resp = requests.get(manifest_url, proxies=proxies)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/requests/api.py", line 75, in get
[contint.wikimedia.org] out:     return request('get', url, params=params, **kwargs)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/requests/api.py", line 60, in request
[contint.wikimedia.org] out:     return session.request(method=method, url=url, **kwargs)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
[contint.wikimedia.org] out:     resp = self.send(prep, **send_kwargs)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
[contint.wikimedia.org] out:     r = adapter.send(request, **kwargs)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/requests/adapters.py", line 514, in send
[contint.wikimedia.org] out:     raise SSLError(e, request=request)
[contint.wikimedia.org] out: requests.exceptions.SSLError: HTTPSConnectionPool(host='docker-registry.discovery.wmnet', port=443): Max retries exceeded with url: /v2/dev/stretch-elasticsearch/manifests/0.0.1 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)')))
[contint.wikimedia.org] out: 2021-02-09 21:08:19,072 [docker-pkg-build] ERROR - Could not load image in /srv/dev-images/dockerfiles/buster: HTTPSConnectionPool(host='docker-registry.discovery.wmnet', port=443): Max retries exceeded with url: /v2/dev/buster/manifests/0.0.1-1 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)'))) (builder.py:298)
[contint.wikimedia.org] out: Traceback (most recent call last):
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 594, in urlopen
[contint.wikimedia.org] out:     self._prepare_proxy(conn)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 805, in _prepare_proxy
[contint.wikimedia.org] out:     conn.connect()
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/connection.py", line 344, in connect
[contint.wikimedia.org] out:     ssl_context=context)
[contint.wikimedia.org] out:   File "/srv/deployment/docker-pkg/venv/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 344, in ssl_wrap_socket
[contint.wikimedia.org] out:     return context.wrap_socket(sock, server_hostname=server_hostname)
[contint.wikimedia.org] out:   File "/usr/lib/python3.7/ssl.py", line 412, in wrap_socket
[contint.wikimedia.org] out:     session=session
[contint.wikimedia.org] out:   File "/usr/lib/python3.7/ssl.py", line 853, in _create
[contint.wikimedia.org] out:     self.do_handshake()
[contint.wikimedia.org] out:   File "/usr/lib/python3.7/ssl.py", line 1117, in do_handshake
[contint.wikimedia.org] out:     self._sslobj.do_handshake()
[contint.wikimedia.org] out: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
...

...it goes on in that vein for quite a while. Unclear if this is failing for other uses of docker-pkg than dev-images.

Event Timeline

brennen triaged this task as High priority.Feb 9 2021, 9:39 PM
brennen added a project: User-brennen.
brennen moved this task from Backlog to Watching on the User-brennen board.

Same result with:

brennen@contint2001:/tmp$ /srv/deployment/docker-pkg/venv/bin/docker-pkg --info -c /etc/docker-pkg/dev-images.yaml build /srv/dev-images/dockerfiles | tee -a dev-images-build.log

curl for one of the URLs in question seems fine:

brennen@contint2001:/tmp$ curl -I https://docker-registry.discovery.wmnet/v2/dev/stretch-php74-jobrunner/manifests/2.0.0
HTTP/1.1 200 OK
Server: nginx/1.13.6
Date: Tue, 09 Feb 2021 22:14:35 GMT
Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
Content-Length: 15653
Connection: keep-alive
Docker-Content-Digest: sha256:77231172b886294941a985fc85a7833cf38cc4702cb33b26348a5cfa7743c6e4
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:77231172b886294941a985fc85a7833cf38cc4702cb33b26348a5cfa7743c6e4"
X-Content-Type-Options: nosniff
Docker-Distribution-Api-Version: registry/2.0
Vary: Accept

Seems docker-pkg specific?

Mentioned in SAL (#wikimedia-releng) [2021-02-10T20:59:28Z] <brennen> Attempting one more update from dev-images docker-pkg on contint2001 for T274306

Change 663588 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/puppet@production] docker-pkg: add ca_bundle configuration

https://gerrit.wikimedia.org/r/663588

For production-images we specify the ca_bundle for python requests to use via 'ca_bundle' in the config.yaml

A ha - thanks! Should have occurred to me to diff the config here with one used elsewhere.

Change 663588 merged by JMeybohm:
[operations/puppet@production] docker-pkg: add ca_bundle configuration

https://gerrit.wikimedia.org/r/663588

@brennen let me know (maybe by closing this :-)) if the patch worked out for you.

brennen claimed this task.

All good - thanks again!