Page MenuHomePhabricator

Can't delete images from docker registry (from build2001 using docker-registryctl)
Closed, InvalidPublic

Description

We want to clean out some old unsued images, e.g. all of the machinelearning-liftwing-inference-services-bloom images. On build2001, I get 401s, however:

root@build2001:~# docker-registryctl  delete-tags docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-bloom
We're about to delete the following tags for image docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-bloom:
2023-05-19-081111-publish
2023-05-19-142929-publish
2023-05-19-145022-publish
2023-05-29-074323-publish
2023-06-06-101745-publish
2023-06-07-080828-publish
2023-06-07-092024-publish
2023-06-07-093841-publish
2023-06-08-132250-publish
2023-06-08-144706-publish
stable
Ok to proceed? (y/n)y
ERROR[docker-registryctl] Error deleting the image wikimedia/machinelearning-liftwing-inference-services-bloom:2023-05-19-081111-publish from the registry
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker_report/registry/operations.py", line 48, in delete_image
    self._request(delete_url, method="DELETE", use_v2=True)
  File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 92, in _request
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://docker-registry.wikimedia.org/v2/wikimedia/machinelearning-liftwing-inference-services-bloom/manifests/sha256:18400a9cdb314f8fc2e6d1bfa2da08bd65531dc95549bab4d1e9c5759e0cc9c7
ERROR[docker-registryctl] Error deleting the image wikimedia/machinelearning-liftwing-inference-services-bloom:2023-05-19-142929-publish from the registry
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker_report/registry/operations.py", line 48, in delete_image
    self._request(delete_url, method="DELETE", use_v2=True)
  File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 92, in _request
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://docker-registry.wikimedia.org/v2/wikimedia/machinelearning-liftwing-inference-services-bloom/manifests/sha256:5832c5fc67012a04baf9a81dadf28c60ff40af199b84306bf619ca8772fb8020
ERROR[docker-registryctl] Error deleting the image wikimedia/machinelearning-liftwing-inference-services-bloom:2023-05-19-145022-publish from the registry
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker_report/registry/operations.py", line 48, in delete_image
    self._request(delete_url, method="DELETE", use_v2=True)
  File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 92, in _request
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://docker-registry.wikimedia.org/v2/wikimedia/machinelearning-liftwing-inference-services-bloom/manifests/sha256:08531c0b40d5610beadfcd21c3b4ef2d3cba1502170107b6fa19e6f79d808303
ERROR[docker-registryctl] Error deleting the image wikimedia/machinelearning-liftwing-inference-services-bloom:2023-05-29-074323-publish from the registry
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker_report/registry/operations.py", line 48, in delete_image
    self._request(delete_url, method="DELETE", use_v2=True)
  File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 92, in _request
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://docker-registry.wikimedia.org/v2/wikimedia/machinelearning-liftwing-inference-services-bloom/manifests/sha256:0cb50e536636ca3b6f9e5f88ee9b40423509ee00e553b9badaef911bfc9e2f77
ERROR[docker-registryctl] Error deleting the image wikimedia/machinelearning-liftwing-inference-services-bloom:2023-06-06-101745-publish from the registry
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker_report/registry/operations.py", line 48, in delete_image
    self._request(delete_url, method="DELETE", use_v2=True)
  File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 92, in _request
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://docker-registry.wikimedia.org/v2/wikimedia/machinelearning-liftwing-inference-services-bloom/manifests/sha256:cc0da885a77f2dd0451b9b477f38e3cb6c591f797b276ff708671b29051fa953
ERROR[docker-registryctl] Error deleting the image wikimedia/machinelearning-liftwing-inference-services-bloom:2023-06-07-080828-publish from the registry
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker_report/registry/operations.py", line 48, in delete_image
    self._request(delete_url, method="DELETE", use_v2=True)
  File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 92, in _request
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://docker-registry.wikimedia.org/v2/wikimedia/machinelearning-liftwing-inference-services-bloom/manifests/sha256:7c72f70c23b041d6585e8ddb6f4eb84adbe8c5a93b3c39d795bf700fff21542c
ERROR[docker-registryctl] Error deleting the image wikimedia/machinelearning-liftwing-inference-services-bloom:2023-06-07-092024-publish from the registry
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker_report/registry/operations.py", line 48, in delete_image
    self._request(delete_url, method="DELETE", use_v2=True)
  File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 92, in _request
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://docker-registry.wikimedia.org/v2/wikimedia/machinelearning-liftwing-inference-services-bloom/manifests/sha256:22dc126b1c46629276980109ef9aa8c32f186404cedaa0d1118d49f6671e9306
ERROR[docker-registryctl] Error deleting the image wikimedia/machinelearning-liftwing-inference-services-bloom:2023-06-07-093841-publish from the registry
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker_report/registry/operations.py", line 48, in delete_image
    self._request(delete_url, method="DELETE", use_v2=True)
  File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 92, in _request
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://docker-registry.wikimedia.org/v2/wikimedia/machinelearning-liftwing-inference-services-bloom/manifests/sha256:81411b0834a310b28e252479627bacf7ffaf153f370f377682b20c20384d6eb2
ERROR[docker-registryctl] Error deleting the image wikimedia/machinelearning-liftwing-inference-services-bloom:2023-06-08-132250-publish from the registry
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker_report/registry/operations.py", line 48, in delete_image
    self._request(delete_url, method="DELETE", use_v2=True)
  File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 92, in _request
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://docker-registry.wikimedia.org/v2/wikimedia/machinelearning-liftwing-inference-services-bloom/manifests/sha256:297fcafa21063ed222349e7298433cb5029396b96908a79429e9039e2dacc45d
ERROR[docker-registryctl] Error deleting the image wikimedia/machinelearning-liftwing-inference-services-bloom:2023-06-08-144706-publish from the registry
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker_report/registry/operations.py", line 48, in delete_image
    self._request(delete_url, method="DELETE", use_v2=True)
  File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 92, in _request
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://docker-registry.wikimedia.org/v2/wikimedia/machinelearning-liftwing-inference-services-bloom/manifests/sha256:324a6313e21e6c5f188980f561d7a80cd5c318d62f985a6aa350fcbcfc6a6458
ERROR[docker-registryctl] Error deleting the image wikimedia/machinelearning-liftwing-inference-services-bloom:stable from the registry
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker_report/registry/operations.py", line 48, in delete_image
    self._request(delete_url, method="DELETE", use_v2=True)
  File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 92, in _request
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://docker-registry.wikimedia.org/v2/wikimedia/machinelearning-liftwing-inference-services-bloom/manifests/sha256:324a6313e21e6c5f188980f561d7a80cd5c318d62f985a6aa350fcbcfc6a6458
docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-bloom:2023-05-19-081111-publish[FAIL]
docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-bloom:2023-05-19-142929-publish[FAIL]
docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-bloom:2023-05-19-145022-publish[FAIL]
docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-bloom:2023-05-29-074323-publish[FAIL]
docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-bloom:2023-06-06-101745-publish[FAIL]
docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-bloom:2023-06-07-080828-publish[FAIL]
docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-bloom:2023-06-07-092024-publish[FAIL]
docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-bloom:2023-06-07-093841-publish[FAIL]
docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-bloom:2023-06-08-132250-publish[FAIL]
docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-bloom:2023-06-08-144706-publish[FAIL]
docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-bloom:stable[FAIL]
ERROR[docker-registryctl] Error interacting with the registry
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/docker_report/registryctl.py", line 153, in main
    delete_tags(options.registry, options.image_name, options.select, options.force)
  File "/usr/lib/python3/dist-packages/docker_report/registryctl.py", line 140, in delete_tags
    raise RegistryError("Could not remove the following tags for image '{}': {}".format(name, ",".join(failed)))
docker_report.registry.RegistryError: Could not remove the following tags for image 'wikimedia/machinelearning-liftwing-inference-services-bloom': 2023-05-19-081111-publish,2023-05-19-142929-publish,2023-05-19-145022-publish,2023-05-29-074323-publish,2023-06-06-101745-publish,2023-06-07-080828-publish,2023-06-07-092024-publish,2023-06-07-093841-publish,2023-06-08-132250-publish,2023-06-08-144706-publish,stable
root@build2001:~#

I'll see if I can find anything in the registry host logs, but I am not very familiar with how it all fits together.

Event Timeline

This was caused by me using the wrong host.

What I _should_ have used:

docker-registryctl delete-tags docker-registry.discovery.wmnet/...

what I did use:

docker-registryctl delete-tags docker-registry.wikimedia.org/...

Note the different domains used. It was because I copied the base URL out of my browser.