Page MenuHomePhabricator
Create Task
Maniphest T375645

Clean up the Docker Registry catalog and Swift storage from old images
Closed, DeclinedPublic
Actions

Description

https://www.raftt.io/post/keeping-your-private-image-registry-clean.html

The link above explains very well the situation in which we are in: every time we delete images/tags via the remove API, nothing is really done behind the scenes. The catalog and Swift storage are not touched, until a proper garbage collection happens.

How can we garbage collect?

We have two options (at least, that we know of):

  • Use the built-in garbage collection feature of Docker Distribution. It works in two steps, one that "marks" all the images/tags to delete, and the other one that "sweeps" and delete them. The mark step is very slow, and in our case it could probably take 2/3 days to complete. When you run GC, the Docker Registry needs to be in read-only mode, something that could affect daily deployments and builds.
  • Use the Docker Distribution Pruner tool, that on paper should be way faster. The main issue is that it is marked as experimental, not production ready and its last commits are only related to supporting new golang versions (so active development of features and general reliability seems to have stopped). The tool can run in dry-run mode as well, so we could test it to see what it would do.
Proposed plan
  • Run both tools in dry-run mode, and report back timings and what they would act on.
  • Establish if any of the results is good/viable, and proceed with a real garbage collection.
  • Think about how to streamline this, even if we'll start from "we do this action manually every 3 months".

Related Objects
Search...

Event Timeline

elukey created this task.Sep 25 2024, 4:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 25 2024, 4:24 PM
elukey added a project: User-Elukey.Sep 25 2024, 4:24 PM
elukey added a subscriber: CDanis.
elukey added a comment.Sep 25 2024, 4:30 PM
elukey@registry1003:~$ /usr/bin/docker-registry help garbage-collect
`garbage-collect` deletes layers not referenced by any manifests

Usage:
  registry garbage-collect <config> [flags]

Flags:
  -m, --delete-untagged   delete manifests that are not currently referenced via tag
  -d, --dry-run           do everything except remove the blobs
  -h, --help              help for garbage-collect

I'd just run the following from registry1005 (eqiad node, codfw is the only DC currently pooled):

sudo -u docker-registry /usr/bin/docker-registry garbage-collect --dry-run /etc/docker/registry/config.yml | tee /home/elukey/docker_registry_dryrun_gc.log

It should run for 2/3 days and emit some useful data.

elukey added a comment.Sep 25 2024, 4:54 PM
elukey:~/go/pkg/mod/gitlab.com/gitlab-org/docker-distribution-pruner@v0.2.0$ EXPERIMENTAL=true ~/go/bin/docker-distribution-pruner
Usage of /home/elukey/go/bin/docker-distribution-pruner:
  -config string
    	Path to registry config file
  -debug
    	Print debug messages
  -delete
    	Delete data, instead of dry run
  -delete-old-tag-versions
    	Delete old tag versions (default true)
  -ignore-blobs
    	Ignore blobs processing and recycling
  -jobs int
    	Number of concurrent jobs to execute (default 10)
  -parallel-blob-walk
    	Allow to use parallel blob walker
  -parallel-repository-walk
    	Allow to use parallel repository walker
  -parallel-walk-jobs int
    	Number of concurrent parallel walk jobs to execute (default 10)
  -repository-csv-output string
    	File to which CSV will be written with all metrics (default "repositories.csv")
  -s3-storage-cache string
    	s3 cache (default "tmp-cache")
  -soft-delete
    	When deleting, do not remove, but move to backup/ folder (default true)
  -soft-errors
    	Print errors, but do not fail
  -verbose
    	Print verbose messages (default true)

So probably something like:

EXPERIMENTAL=true ~/go/bin/docker-distribution-pruner -verbose -config  /etc/docker/registry/config.yml | tee /home/elukey/docker_pruner_dryrun_gc.log

By default it runs in dry-run mode, afaict. To be extra sure, we could create another account that is read-only for this use case.

bking subscribed.Sep 25 2024, 8:13 PM
Stashbot added a comment.Sep 26 2024, 10:21 AM

Mentioned in SAL (#wikimedia-operations) [2024-09-26T10:21:08Z] <elukey> start dry run of docker distribution GC on registry1004 (info in https://phabricator.wikimedia.org/T375645#10176397, you can find a root tmux session named as the task on the host to stop)

elukey moved this task from Backlog to In Progress on the User-Elukey board.Sep 26 2024, 12:36 PM
elukey added a comment.Sep 30 2024, 7:31 AM

The dry run took ~33 hrs and its log can be found on registry1004:/home/elukey/docker_registry_dryrun_gc.log. The blobs/layers up for deletion are quite a few:

elukey@registry1004:~$ grep deletion docker_registry_dryrun_gc.log | wc -l
72484

All the deletion entries are in the following form:

blob eligible for deletion: sha256:793df3a604ee0b446fa491f2b92e1c1d6c93be4a7f5e5223ef6a4926f3ce7a0b

Some random notes:

  • The total time for mark/sweep is 33h, that is long but not impossible. We could probably start early EU morning on a Friday and let it finish on the next day. The only downside is that something could go south right before/during the weekend, but we'd avoid stopping deployments etc.. (so less disrupt actions for other teams).
  • From the 33hrs we'd need to add the time to actually drop the 75k blobs from swift, so I'd add 2/3 hours more to the count.
  • It would be nice to verify that the blobs eligible for deletion are indeed the ones that we'd expect, not sure if we can double check using the Docker Distribution's API or not. Alternatively we could review the "mark" logs since all images listed shouldn't be up for deletion.
elukey added a comment.Sep 30 2024, 1:26 PM

Some stats:

swift_account_stats_objects_total{account="AUTH_docker", cluster="swift", instance="ms-fe1009:9112", job="statsd_exporter", prometheus="ops", site="eqiad"}
1235468
swift_account_stats_objects_total{account="AUTH_docker", cluster="swift", instance="ms-fe2009:9112", job="statsd_exporter", prometheus="ops", site="codfw"}
1235468

swift_account_stats_bytes_total{account="AUTH_docker", cluster="swift", instance="ms-fe1009:9112", job="statsd_exporter", prometheus="ops", site="eqiad"}
5804462330952
swift_account_stats_bytes_total{account="AUTH_docker", cluster="swift", instance="ms-fe2009:9112", job="statsd_exporter", prometheus="ops", site="codfw"}

So atm it seems that we have ~1M objects and ~6TB of storage used for the Docker registry :D

akosiaris added a comment.Sep 30 2024, 1:37 PM

Nice job!

So, 75k is a not insignificant but also a not particularly large percentage. ~6%. I have my doubts it will substantially help (although it might). Definitely go ahead and run it next week, we need to gauge the effects.

However, what probably makes some sense is to GC a ton of old mediawiki images (probably keeping like 30 days at most?) and then run it again.

elukey added a comment.Sep 30 2024, 2:05 PM

Hi @akosiaris! I had a chat with Janis and there are some details that needs to be highlighted:

  • I didn't run it with --delete-untagged (of course not mentioned clearly in the upstream docs) so probably the scope of the sweep step is broader. I can re-run the dryrun with the option and measure how long it will take. In theory it should take the same amount of time, but we'll have a better idea about the final target.
  • We are are a little scared to drop unwanted images when running GC, since it doesn't seem that trustworthy. We are exploring options about how to verify what it is going to be deployed, the tool outlines only the blobs' sha1 and not their related image name.
elukey added a comment.EditedSep 30 2024, 2:06 PM

Started:

elukey@registry1004:~$ time sudo -u docker-registry /usr/bin/docker-registry garbage-collect --dry-run --delete-untagged /etc/docker/registry/config.yml | tee /home/elukey/docker_registry_dryrun_deleteuntagged_gc.log
elukey added a comment.Oct 2 2024, 7:57 AM

It failed with:

failed to garbage collect: failed to mark: swift: swift: failed to retrieve tags unknown repository name=dev/stretch-php73-fpm

This is an image that we deleted with requestctl, so IIUC it is present in the catalog but not on Swift. After a chat with Janis we agreed that the garbage collector feature of Docker Distribution is probably a little less experimental than the docker-distribution-pruner, so we may just test the latter in dry-run mode and see how it goes. The caveat is that it doesn't support Swift natively, but only the s3 API, so we'll see if we can adapt to it.

elukey added a comment.Oct 2 2024, 2:34 PM

Backtracking a little before proceeding further, there are some things that I don't fully grasp.

In most of the docs that I am reading, people seems to be interested in garbage collection only to reclaim storage space, they don't mention anything about the catalog being slow or messy. So now I am wondering if there are two issues:

  1. We use registryctl to drop tags via API, it is a custom-built Python script part of Docker Report. The catalog is still full of things that are not present in the UI, and that should be gone. For example, I tried to curl /v2/_catalog to get all the repositories (paginated) and I see releng/npm-stretch. Registryctl doesn't return any available tag to delete (elukey@build2001:~$ sudo -i docker-registryctl delete-tags docker-registry.discovery.wmnet/releng/npm-stretch) but I see some binaries stored in swift (used swift list etc.. on ms-fe1009). So maybe we are not properly cleaning up the catalog via registryctl?
  1. Space on Swift is not actively reclaimed, but it is not related to the catalog.

The first issue is now causing docker-report to timeout after some calls, so I'd concentrate on it for the time being. I think that the timeouts need another task, to avoid too much confusion. In this one I'll try to tackle the above issues.

elukey added a comment.EditedOct 3 2024, 9:43 AM

I may have found some clue related to why the catalog contains so many stale things: https://github.com/distribution/distribution/issues/2747

Currently the only way to delete repository from registry is by deleting it's directory manually from registry-v2/docker/registry/v2/repositories directory in storage. Until then the repo will be returned by catalog API and cause confusion when people delete manifests and tags from the repo.

A repository is basically an image name, so when we delete tags via registryctl we leave "empty" repositories in the catalog. IIUC docker-registry just returns the list of repositories stored in Swift as catalog, so we could create a script that cleans swift from empty repositories. Since we also use Redis for blob caching, it may be used also for the catalog, so I'd say that a flush to its cache will probably be needed?

Found also https://forums.docker.com/t/delete-repository-from-v2-private-registry/16767 that seems to confirm what we are seeing (repositories not dropped via delete API).

elukey added a comment.Oct 8 2024, 3:40 PM

I dumped all the files stored in swift in a text file on ms-fe1009, and ran the following:

from pprint import pprint

print("Retrieving list of repositories")
repositories = set()
repositories_with_tags = set()
with open('T375645_container_dump.txt', 'r') as dump:
    for line in dump.readlines():
        if "_manifest" in line:
            repository = line.split("_")[0]
            if repository not in repositories:
                repositories.add(repository)
            if "/tags/" in line:
                if repository not in repositories_with_tags:
                    repositories_with_tags.add(repository)

print(f"Repositories: {len(repositories)}, Repositories with tags: {len(repositories_with_tags)}")
print("\nRepositories without tags:")
pprint(repositories - repositories_with_tags)

print("\nRepositories that may be obsolete:")
for repository in repositories_with_tags:
    if ("buster" or "stretch" or "jessie") in repository:
        print(repository)

Results:

Retrieving list of repositories
Repositories: 687, Repositories with tags: 630

Repositories without tags:
{'files/docker/registry/v2/repositories/dev/stretch-php73-fpm/',
 'files/docker/registry/v2/repositories/fluent-bit/',
 'files/docker/registry/v2/repositories/fluentd/',
 'files/docker/registry/v2/repositories/golang/',
 'files/docker/registry/v2/repositories/kubernetes-fluentd-daemonset/',
 'files/docker/registry/v2/repositories/python3-build-jessie/',
 'files/docker/registry/v2/repositories/python3-build-stretch/',
 'files/docker/registry/v2/repositories/python3/',
 'files/docker/registry/v2/repositories/releng/ci-jessie/',
 'files/docker/registry/v2/repositories/releng/ci-src-setup/',
 'files/docker/registry/v2/repositories/releng/ci-stretch/',
 'files/docker/registry/v2/repositories/releng/composer-package-php71/',
 'files/docker/registry/v2/repositories/releng/composer-php56/',
 'files/docker/registry/v2/repositories/releng/composer-php71/',
 'files/docker/registry/v2/repositories/releng/composer-test-php56/',
 'files/docker/registry/v2/repositories/releng/hhvm-jessie-compile/',
 'files/docker/registry/v2/repositories/releng/hhvm-jessie/',
 'files/docker/registry/v2/repositories/releng/npm-stretch/',
 'files/docker/registry/v2/repositories/releng/npm-test-3d2png/',
 'files/docker/registry/v2/repositories/releng/npm-test-graphoid/',
 'files/docker/registry/v2/repositories/releng/npm-test-librdkafka/',
 'files/docker/registry/v2/repositories/releng/npm-test-maps-service/',
 'files/docker/registry/v2/repositories/releng/npm-test-stretch/',
 'files/docker/registry/v2/repositories/releng/npm-test/',
 'files/docker/registry/v2/repositories/releng/npm/',
 'files/docker/registry/v2/repositories/releng/php56/',
 'files/docker/registry/v2/repositories/releng/php71-compile/',
 'files/docker/registry/v2/repositories/releng/php71/',
 'files/docker/registry/v2/repositories/releng/quibble-jessie-hhvm/',
 'files/docker/registry/v2/repositories/releng/quibble-jessie-php55/',
 'files/docker/registry/v2/repositories/releng/quibble-jessie-php56/',
 'files/docker/registry/v2/repositories/releng/quibble-jessie/',
 'files/docker/registry/v2/repositories/releng/quibble-stretch-bundle/',
 'files/docker/registry/v2/repositories/releng/quibble-stretch-hhvm/',
 'files/docker/registry/v2/repositories/releng/quibble-stretch-php70/',
 'files/docker/registry/v2/repositories/releng/quibble-stretch-php71/',
 'files/docker/registry/v2/repositories/releng/quibble-stretch-php72/',
 'files/docker/registry/v2/repositories/releng/quibble-stretch-php73/',
 'files/docker/registry/v2/repositories/releng/quibble-stretch-php74/',
 'files/docker/registry/v2/repositories/releng/quibble-stretch-php80/',
 'files/docker/registry/v2/repositories/releng/quibble-stretch/',
 'files/docker/registry/v2/repositories/releng/wikimedia-audit-resources/',
 'files/docker/registry/v2/repositories/ruby/',
 'files/docker/registry/v2/repositories/statsd-proxy/',
 'files/docker/registry/v2/repositories/stretch/',
 'files/docker/registry/v2/repositories/wikimedia-jessie/',
 'files/docker/registry/v2/repositories/wikimedia-stretch/',
 'files/docker/registry/v2/repositories/wikimedia/analytics-datahub/',
 'files/docker/registry/v2/repositories/wikimedia/blubber/',
 'files/docker/registry/v2/repositories/wikimedia/eventgate-ci/',
 'files/docker/registry/v2/repositories/wikimedia/labs-libraryupgrader/',
 'files/docker/registry/v2/repositories/wikimedia/mediawiki-core/',
 'files/docker/registry/v2/repositories/wikimedia/mediawiki-services-function-evaluator/',
 'files/docker/registry/v2/repositories/wikimedia/mediawiki-services-function-orchestrator/',
 'files/docker/registry/v2/repositories/wikimedia/mediawiki-services-graphoid/',
 'files/docker/registry/v2/repositories/wikimedia/mediawiki-services-parsoid/',
 'files/docker/registry/v2/repositories/wikimedia/operations-container-miscweb/'}

Repositories that may be obsolete:
files/docker/registry/v2/repositories/dev/buster/
files/docker/registry/v2/repositories/releng/buster-rsyslog/
files/docker/registry/v2/repositories/releng/buster-php73-jobrunner/
files/docker/registry/v2/repositories/releng/buster-php80/
files/docker/registry/v2/repositories/releng/quibble-buster-php83/
files/docker/registry/v2/repositories/repos/releng/scap/deploy-buster/
files/docker/registry/v2/repositories/dev/buster-php80-fpm/
files/docker/registry/v2/repositories/releng/fundraising-civiproxy-buster-php73-apache2/
files/docker/registry/v2/repositories/dev/buster-rsyslog/
files/docker/registry/v2/repositories/dev/buster-php80-jobrunner/
files/docker/registry/v2/repositories/dev/fundraising-civiproxy-buster-php73-apache2/
files/docker/registry/v2/repositories/dev/buster-php-sury/
files/docker/registry/v2/repositories/buster/
files/docker/registry/v2/repositories/releng/quibble-buster-php72-bundle/
files/docker/registry/v2/repositories/releng/fundraising-civicrm-buster-php73/
files/docker/registry/v2/repositories/dev/fundraising-civicrm-buster-php73-apache2/
files/docker/registry/v2/repositories/releng/ci-buster/
files/docker/registry/v2/repositories/buster-nodejs10-slim/
files/docker/registry/v2/repositories/releng/fundraising-privatebin-buster-php73-apache2/
files/docker/registry/v2/repositories/dev/buster-php74-jobrunner/
files/docker/registry/v2/repositories/dev/buster-swift53/
files/docker/registry/v2/repositories/releng/fundraising-smashpig-buster-php73-apache2/
files/docker/registry/v2/repositories/dev/fundraising-privatebin-buster-php73-apache2/
files/docker/registry/v2/repositories/releng/fundraising-mediawiki-buster-php73-apache2/
files/docker/registry/v2/repositories/releng/quibble-buster-php72-apache/
files/docker/registry/v2/repositories/dev/buster-php73-fpm/
files/docker/registry/v2/repositories/releng/tox-buster/
files/docker/registry/v2/repositories/dev/buster-apache2/
files/docker/registry/v2/repositories/releng/buster-php73/
files/docker/registry/v2/repositories/releng/buster-php74/
files/docker/registry/v2/repositories/dev/buster-php73/
files/docker/registry/v2/repositories/releng/quibble-buster-php74-bundle/
files/docker/registry/v2/repositories/python3-build-buster/
files/docker/registry/v2/repositories/releng/buster/
files/docker/registry/v2/repositories/dev/fundraising-smashpig-buster-php73-apache2/
files/docker/registry/v2/repositories/releng/buster-apache2/
files/docker/registry/v2/repositories/dev/fundraising-buster-php73-apache2-xdebug/
files/docker/registry/v2/repositories/releng/quibble-buster-php80/
files/docker/registry/v2/repositories/releng/buster-php72-jobrunner/
files/docker/registry/v2/repositories/repos/releng/scap/target-buster/
files/docker/registry/v2/repositories/dev/fundraising-mediawiki-buster-php73-apache2/
files/docker/registry/v2/repositories/releng/buster-php72/
files/docker/registry/v2/repositories/releng/quibble-buster-php74-coverage/
files/docker/registry/v2/repositories/releng/quibble-buster-php73/
files/docker/registry/v2/repositories/releng/quibble-buster-php81/
files/docker/registry/v2/repositories/python3-buster/
files/docker/registry/v2/repositories/releng/fundraising-buster/
files/docker/registry/v2/repositories/dev/buster-php72-jobrunner/
files/docker/registry/v2/repositories/buster-nodejs10-devel/
files/docker/registry/v2/repositories/dev/buster-php72/
files/docker/registry/v2/repositories/releng/buster-php80-jobrunner/
files/docker/registry/v2/repositories/dev/buster-php81/
files/docker/registry/v2/repositories/releng/quibble-buster-php73-coverage/
files/docker/registry/v2/repositories/releng/buster-php-sury/
files/docker/registry/v2/repositories/dev/buster-php73-jobrunner/
files/docker/registry/v2/repositories/releng/quibble-buster-php74/
files/docker/registry/v2/repositories/releng/buster-php72-fpm/
files/docker/registry/v2/repositories/releng/buster-php74-fpm/
files/docker/registry/v2/repositories/wikimedia-buster/
files/docker/registry/v2/repositories/python2-build-buster/
files/docker/registry/v2/repositories/releng/fundraising-civicrm-buster-php73-apache2/
files/docker/registry/v2/repositories/dev/buster-php81-fpm/
files/docker/registry/v2/repositories/repos/releng/scap/buster/
files/docker/registry/v2/repositories/dev/fundraising-buster/
files/docker/registry/v2/repositories/dev/fundraising-buster-php73/
files/docker/registry/v2/repositories/releng/tox-poetry-buster/
files/docker/registry/v2/repositories/releng/buster-php73-fpm/
files/docker/registry/v2/repositories/dev/buster-php80/
files/docker/registry/v2/repositories/releng/quibble-buster-php72/
files/docker/registry/v2/repositories/dev/buster-php74-fpm/
files/docker/registry/v2/repositories/dev/buster-php72-fpm/
files/docker/registry/v2/repositories/releng/buster-php74-jobrunner/
files/docker/registry/v2/repositories/releng/quibble-buster-php82/
files/docker/registry/v2/repositories/releng/buster-php80-fpm/
files/docker/registry/v2/repositories/dev/buster-php81-jobrunner/
files/docker/registry/v2/repositories/dev/buster-php74/
files/docker/registry/v2/repositories/releng/fundraising-buster-php73/
files/docker/registry/v2/repositories/releng/quibble-buster/
files/docker/registry/v2/repositories/wmf-debci-buster/
files/docker/registry/v2/repositories/dev/fundraising-civicrm-buster-php73/

To double check, I tried to delete docker-registry.wikimedia.org/python2-build-buster and indeed it showed to me a lot of tags.

Worth to notice: we have 687 total repositories (i.e. docker images that at some point in time were present on the registry), and 630 of them have tags (so we didn't delete them explicitly). Only 57 are "empty" repositories, so even if we cleaned them up we wouldn't really gain much (worth to do it anyway of course).

Complete list of repositories in https://phabricator.wikimedia.org/T376724

Next step:

  • Figure out if garbage collection may work if we drop the empty repositories from swift (and possibly from Redis).
elukey closed subtask T376285: Timeout while retrieving the catalog from the Docker Registry as Resolved.Oct 10 2024, 9:56 AM
elukey added a comment.EditedOct 10 2024, 1:26 PM

I put some thoughts on the current situation, and even if there are a lot of unknowns, I realized that garbage collection may not be what we need if we accept a compromise: we drop data only when we delete all tags to a Docker image. Basically we give up on reclaiming space related to Docker images with tags in the registry, and we concentrate on deep cleaning after we dockerctl delete-tags. In this way, after we drop all tags via dockerctl, we would just need to run a Python script that does the following:

  • Retrieve all the files correspondent to the repository/docker-image name from Redis. In case of dev/stretch-php73-fpm:
127.0.0.1:6382> KEYS *dev/stretch-php73-fpm*
 1) "repository::dev/stretch-php73-fpm::blobs::sha256:4b7e3e185519e0eac2496d54e7344fbdafe4c2bd817265cb63583ffc57e106af"
 2) "repository::dev/stretch-php73-fpm::blobs::sha256:129cd9268f2542b9b61be98363ce86c6a3cc7387674729af8fcbf899eeb31ec4"
 3) "repository::dev/stretch-php73-fpm::blobs::sha256:010f8a907a124799a9b068133ccbaf3e5ad07f9d4b98719891b9d2c43ae600d1"
 4) "repository::dev/stretch-php73-fpm::blobs::sha256:f5713261151c029cf62b93e3920c19f39e1d8f89a1a8ea8001faed2aca41ec1a"
 5) "repository::dev/stretch-php73-fpm::blobs::sha256:ad6b363efe757163f82bff09e86bdb238d6796a2357cf837caf06bcf6d339ac6"
 6) "repository::dev/stretch-php73-fpm::blobs::sha256:0b392d63fa40ac78eb1ce66d8ef290a7185a2793c141f233a6cb7da56fbfe5b4"
 7) "repository::dev/stretch-php73-fpm::blobs::sha256:7e5928aa10e10eef408e1e657d02799ea16f1b83ba1e3c167e250af7c4362cdf"
 8) "repository::dev/stretch-php73-fpm::blobs::sha256:43488a5cac640b9941fbc35bc83e7a76c4818e23aa0eb4260de2fd0fd48fa59a"
 9) "repository::dev/stretch-php73-fpm::blobs::sha256:7b04e34b88d37cf1d8957cdaf46841c2904bd749e60c2b57dcf0b7c9fda51360"
10) "repository::dev/stretch-php73-fpm::blobs::sha256:d2f98a7cbf93dea125a9d45449cca7cb09c14c59829e5c9e7745029f49969728"
11) "repository::dev/stretch-php73-fpm::blobs::sha256:2db4ca3b312c6f4c4fad476523d224f3443330f34061a12494baccd4b3b735b5"
12) "repository::dev/stretch-php73-fpm::blobs::sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
13) "repository::dev/stretch-php73-fpm::blobs::sha256:84f6be64c2fb77e0ae4b7806063e9caea7e9b62d541e5402211c93d4b8c1d135"
14) "repository::dev/stretch-php73-fpm::blobs::sha256:eb0ebb16774921a2b983b1bf2df163f9b0b8d7e7d45dc23af7a95def7afa361b"
15) "repository::dev/stretch-php73-fpm::blobs::sha256:cfa32658670a998af0e47635ee194695aa54138b833b9d7964f8ca6c32411396"
16) "repository::dev/stretch-php73-fpm::blobs::sha256:bea4c98c135acba444cb9c73dc59154256d1971f5974f6f0e635a464592f0f50"
17) "repository::dev/stretch-php73-fpm::blobs::sha256:022244a42edb560a222c438bc0e3a7a51ee2ee512381af02fbb95dd3a03089f2"
18) "repository::dev/stretch-php73-fpm::blobs::sha256:f46ed641a06570d42ee992ec443e9eafb9d4b878658f1c9a9c176bccb5f9d7c2"
19) "repository::dev/stretch-php73-fpm::blobs::sha256:dad406b0352dd7987a9cfa78c7c470d52aa92afc57f81f2b2759228dd3d44d02"
20) "repository::dev/stretch-php73-fpm::blobs::sha256:8ac7a1b21ef06d253a5ffeea6fd52d386025c4466e1867be12865d66b4e8e3cc"
21) "repository::dev/stretch-php73-fpm::blobs::sha256:e4fe83949190781c79d71bd38de18997d80e638abf2a51d8ae97fe0f12625ca9"
22) "repository::dev/stretch-php73-fpm::blobs::sha256:4c3c3001bee901ef7d8a0e8204694a19e93ea078f30c251c8c5eb2d96f0254f3"
23) "repository::dev/stretch-php73-fpm::blobs::sha256:ab32434d3734fb454f6b345eebacc2db0db1adc990ef481953c02e1c747ff4c4"
24) "repository::dev/stretch-php73-fpm::blobs::sha256:a2bd9a3442c94785aaf09fa2a2ea060648379bfb43127e08f1b687a6af233b86"
25) "repository::dev/stretch-php73-fpm::blobs::sha256:d8ba795098528bea1e1ed68b86caf7da7280d6af8a1e86d9e0433dc403303640"
26) "repository::dev/stretch-php73-fpm::blobs::sha256:98351905c68907ee6fb47a375be94c2670e01d66089f1243e7fdcd20ec82e08f"
27) "repository::dev/stretch-php73-fpm::blobs::sha256:a529ad64e3b7b8e4b6d1ff4938f10ac69fb14e217928d77648b8d095fae59832"
28) "repository::dev/stretch-php73-fpm::blobs::sha256:500676f2d3b0d8903107b2ef8b2a74e1b64aa917ec399b53c1d4c829795b4a4f"
29) "repository::dev/stretch-php73-fpm::blobs::sha256:1f49037844d7b2c3fc630c4d22a3e99b8cac722755d65c229d4553be2e95fba9"
30) "repository::dev/stretch-php73-fpm::blobs::sha256:e01d5733b39bc21eff470c105819bb181c49e14b9ef42a83cf6b0dbf162f459e"
31) "repository::dev/stretch-php73-fpm::blobs::sha256:d08b7d87bf42969dfe810de01d775eec56e32b9d170b7c96a7fa836262e01ac4"
32) "repository::dev/stretch-php73-fpm::blobs::sha256:e356aede486dccb8b5129b498324e2f4e25ee3758600349495be83d1aa1ebd91"
33) "repository::dev/stretch-php73-fpm::blobs::sha256:b309166981080a7e6a29101f0c5cfcf276b6132adbea21957739233ea653883b"
34) "repository::dev/stretch-php73-fpm::blobs::sha256:ae927f96aa8deadbd0979370f8d99499e693fb52f860bf55f5f447120f815ee8"
35) "repository::dev/stretch-php73-fpm::blobs::sha256:97f6a247bff7e4bf4204639265888595ad5786794af41d7c5ba480c69de03bda"
36) "repository::dev/stretch-php73-fpm::blobs::sha256:a9cc5f2c166b5b9af935b8d2d9eb9d24ddd645f36dad5031b011c52ae0903a92"
37) "repository::dev/stretch-php73-fpm::blobs::sha256:b04f3a3f39306b637d63bbc50394ba6fd2b4f62b3335f91d92e10cdfba74f81f"
38) "repository::dev/stretch-php73-fpm::blobs::sha256:d8155d9e06681e2d82a17e38c9128deb486186190029c164195e37b2bb42ef30"
39) "repository::dev/stretch-php73-fpm::blobs::sha256:da2cecce4e3b88838a5d612a3ebbce440bb5a3dffb993f5f9ee3e31270e7cd88"
40) "repository::dev/stretch-php73-fpm::blobs::sha256:73496f94b4099b89cc44892584a49ce49a5f4f9e445f548ed7a9ae8f99dadc3d"
41) "repository::dev/stretch-php73-fpm::blobs::sha256:9639d59a686e96521275800289762c777cacfbdf14dc0434ee0ac48f33d87f58"
42) "repository::dev/stretch-php73-fpm::blobs::sha256:fe377e8b6466ca14e2c948ccec5a7e8d8f21389dd537cfb34e67096f38ddb7fa"
43) "repository::dev/stretch-php73-fpm::blobs::sha256:901cdc7ffc2cf0d70ca91a5fffa22d27c2b1273ed43348d2211734289b655d54"
44) "repository::dev/stretch-php73-fpm::blobs::sha256:7c7b95d4bd7ca9c2c7f61533e128fbd3d2d6d628370c2c78ad45cadc59b88617"
45) "repository::dev/stretch-php73-fpm::blobs::sha256:c9c87b33a9fdb0677422e0587bf94ec2908c6a4b8433ab704042b2d7b2ea2811"
46) "repository::dev/stretch-php73-fpm::blobs::sha256:0c51f8b393644f1594e7c4481fe0dbc2ccb7a1209ba3d7db014bca0237be82d4"
47) "repository::dev/stretch-php73-fpm::blobs::sha256:9b6d9807117f105a4c8dc96853f53967fe42054b0929de4f7e90951cef9e4822"
48) "repository::dev/stretch-php73-fpm::blobs::sha256:5fbf0593ce10536be59d6d81e9f0894468953a47b243b10b52a09c474a3eaba3"
49) "repository::dev/stretch-php73-fpm::blobs::sha256:a58d2a0d47e1746c9e15feed2abc3b108c17aa0bcbc3994fe6d849760dac7d7c"
50) "repository::dev/stretch-php73-fpm::blobs::sha256:e786f51ed20a2fd111befaebb2195478c32f8b37db48b1d9c6a443535df3552d"
51) "repository::dev/stretch-php73-fpm::blobs::sha256:e12dcc58d8bee09ece4395cfd9325ba45e646a19899ff0bfb80cec2c34d9320b"
52) "repository::dev/stretch-php73-fpm::blobs::sha256:4d62fa773ee07eeefe09afe6f054b57cb729d2bfc3578c1d4830c0a271ac08aa"
53) "repository::dev/stretch-php73-fpm::blobs::sha256:d38adb396606a721096018554077eff4fa1f2ed8b05343edf43d735dbc6323d2"
54) "repository::dev/stretch-php73-fpm::blobs::sha256:d0c4633d271740903e10b64f37019843a29c6c6ac5c817e61b04360a92f1dd56"
55) "repository::dev/stretch-php73-fpm::blobs::sha256:fb61622d721a05a25d3703532cd03b1a64793955bc23b0b6a062408ad884e83e"
56) "repository::dev/stretch-php73-fpm::blobs::sha256:e834627d93cf269bae28acb64b75471b557923ed4c293414960180811ba35a55"
57) "repository::dev/stretch-php73-fpm::blobs::sha256:fa7164297a6da553f95fab2e3c51f32115664afdb153219add5d879674d8fc43"
58) "repository::dev/stretch-php73-fpm::blobs::sha256:f9202505c57535ef83349e992b042b6ea1bbffc83b12072c22328b648e0e8071"
59) "repository::dev/stretch-php73-fpm::blobs::sha256:9b2091813f13af8893c9f9f1389349c35356bcdfea86f14cbf0cedcddf564de2"
60) "repository::dev/stretch-php73-fpm::blobs::sha256:b858c1cf7b12eaf269f099e844f1492352d1c7fc05cea94f06d27056c8696af4"
61) "repository::dev/stretch-php73-fpm::blobs::sha256:5c9ef1291f9b0477dc8cc21b043a36cd61c1e5d90c965e2319a80608a28aeab5"
62) "repository::dev/stretch-php73-fpm::blobs::sha256:d70878c68b55b729320e69d172d4db8a54aecdd319f2c93d3688626fc470ad71"
63) "repository::dev/stretch-php73-fpm::blobs::sha256:b8dac1515ef6fbece43eae740ec9b7ac1e217d8cdbd3f8af4cd468a8df9041c1"
64) "repository::dev/stretch-php73-fpm::blobs::sha256:7da1640ae982194ad5c32b92d9024f19cc1fa3b16507016328b0fb253e6adf6f"
65) "repository::dev/stretch-php73-fpm::blobs::sha256:ebe0c0daef21f30f0244e0244bf0bd9ffd8734ba1d6df3addc78f1fdd049dda9"
66) "repository::dev/stretch-php73-fpm::blobs::sha256:2c93bba5818d8e23be37f243159576fd8e5068cd28ad9e66bea3330de955301e"
67) "repository::dev/stretch-php73-fpm::blobs::sha256:4d4b7bd35dd80f2395e11506993b5109680410636d5897a38125ba980cb23466"
68) "repository::dev/stretch-php73-fpm::blobs"
69) "repository::dev/stretch-php73-fpm::blobs::sha256:9c816581d1b1d789f64cd29a52ec1a32f9c32d3837288addd6211914ac5277af"
70) "repository::dev/stretch-php73-fpm::blobs::sha256:1106614dbecabadb10017d8b220f0be81cdfedbf55eb69c959d99c5811b8eba2"
71) "repository::dev/stretch-php73-fpm::blobs::sha256:73df6afcb848b74ed3a1eb8ad2e450d4c9de2b4a7604514e2754bd080a382b29"
72) "repository::dev/stretch-php73-fpm::blobs::sha256:c4398f768745d8ea0d7bcd9120104136131add82e462ae747e1d4ae1103049a5"
73) "repository::dev/stretch-php73-fpm::blobs::sha256:a65fb5b70d1e78cf38f1f51ace27f02e2171fb05618a2147cf70fbc70d134c24"
74) "repository::dev/stretch-php73-fpm::blobs::sha256:ab66cf5625aea9e4536a0712747367371195e7f4801f488ff3ca1f4abd06cb25"
75) "repository::dev/stretch-php73-fpm::blobs::sha256:60fd9e531cfc3aaf4596d56001b7ce17fa5dcfca09dc608ed6d1fedd74ff362b"
  • For every blob's sha256, clean up what's on Redis. If we pick 4b7e3e185519e0eac2496d54e7344fbdafe4c2bd817265cb63583ffc57e106af:
127.0.0.1:6382> type "repository::dev/stretch-php73-fpm::blobs::sha256:4b7e3e185519e0eac2496d54e7344fbdafe4c2bd817265cb63583ffc57e106af"
hash
127.0.0.1:6382> HGETALL "repository::dev/stretch-php73-fpm::blobs::sha256:4b7e3e185519e0eac2496d54e7344fbdafe4c2bd817265cb63583ffc57e106af"
1) "mediatype"
2) "application/octet-stream"
127.0.0.1:6382> type "blobs::sha256:4b7e3e185519e0eac2496d54e7344fbdafe4c2bd817265cb63583ffc57e106af"
hash
127.0.0.1:6382> HGETALL "blobs::sha256:4b7e3e185519e0eac2496d54e7344fbdafe4c2bd817265cb63583ffc57e106af"
1) "digest"
2) "sha256:4b7e3e185519e0eac2496d54e7344fbdafe4c2bd817265cb63583ffc57e106af"
3) "mediatype"
4) "application/octet-stream"
5) "size"
6) "261"
  • Delete the repository's set stored on Redis:
127.0.0.1:6382> type "repository::dev/stretch-php73-fpm::blobs"
set
127.0.0.1:6382> smembers "repository::dev/stretch-php73-fpm::blobs"
 1) "sha256:ad6b363efe757163f82bff09e86bdb238d6796a2357cf837caf06bcf6d339ac6"
 2) "sha256:9b6d9807117f105a4c8dc96853f53967fe42054b0929de4f7e90951cef9e4822"
 3) "sha256:e4fe83949190781c79d71bd38de18997d80e638abf2a51d8ae97fe0f12625ca9"
 4) "sha256:f5713261151c029cf62b93e3920c19f39e1d8f89a1a8ea8001faed2aca41ec1a"
 5) "sha256:ae927f96aa8deadbd0979370f8d99499e693fb52f860bf55f5f447120f815ee8"
 6) "sha256:1106614dbecabadb10017d8b220f0be81cdfedbf55eb69c959d99c5811b8eba2"
 7) "sha256:c9c87b33a9fdb0677422e0587bf94ec2908c6a4b8433ab704042b2d7b2ea2811"
 8) "sha256:73496f94b4099b89cc44892584a49ce49a5f4f9e445f548ed7a9ae8f99dadc3d"
 9) "sha256:d8155d9e06681e2d82a17e38c9128deb486186190029c164195e37b2bb42ef30"
10) "sha256:eb0ebb16774921a2b983b1bf2df163f9b0b8d7e7d45dc23af7a95def7afa361b"
11) "sha256:e834627d93cf269bae28acb64b75471b557923ed4c293414960180811ba35a55"
12) "sha256:4c3c3001bee901ef7d8a0e8204694a19e93ea078f30c251c8c5eb2d96f0254f3"
13) "sha256:73df6afcb848b74ed3a1eb8ad2e450d4c9de2b4a7604514e2754bd080a382b29"
14) "sha256:010f8a907a124799a9b068133ccbaf3e5ad07f9d4b98719891b9d2c43ae600d1"
15) "sha256:97f6a247bff7e4bf4204639265888595ad5786794af41d7c5ba480c69de03bda"
16) "sha256:d08b7d87bf42969dfe810de01d775eec56e32b9d170b7c96a7fa836262e01ac4"
17) "sha256:a58d2a0d47e1746c9e15feed2abc3b108c17aa0bcbc3994fe6d849760dac7d7c"
18) "sha256:8ac7a1b21ef06d253a5ffeea6fd52d386025c4466e1867be12865d66b4e8e3cc"
19) "sha256:2c93bba5818d8e23be37f243159576fd8e5068cd28ad9e66bea3330de955301e"
20) "sha256:129cd9268f2542b9b61be98363ce86c6a3cc7387674729af8fcbf899eeb31ec4"
21) "sha256:a529ad64e3b7b8e4b6d1ff4938f10ac69fb14e217928d77648b8d095fae59832"
22) "sha256:ebe0c0daef21f30f0244e0244bf0bd9ffd8734ba1d6df3addc78f1fdd049dda9"
23) "sha256:4b7e3e185519e0eac2496d54e7344fbdafe4c2bd817265cb63583ffc57e106af"
24) "sha256:0c51f8b393644f1594e7c4481fe0dbc2ccb7a1209ba3d7db014bca0237be82d4"
25) "sha256:4d4b7bd35dd80f2395e11506993b5109680410636d5897a38125ba980cb23466"
26) "sha256:a9cc5f2c166b5b9af935b8d2d9eb9d24ddd645f36dad5031b011c52ae0903a92"
27) "sha256:7da1640ae982194ad5c32b92d9024f19cc1fa3b16507016328b0fb253e6adf6f"
28) "sha256:ab66cf5625aea9e4536a0712747367371195e7f4801f488ff3ca1f4abd06cb25"
29) "sha256:fb61622d721a05a25d3703532cd03b1a64793955bc23b0b6a062408ad884e83e"
30) "sha256:e356aede486dccb8b5129b498324e2f4e25ee3758600349495be83d1aa1ebd91"
31) "sha256:f46ed641a06570d42ee992ec443e9eafb9d4b878658f1c9a9c176bccb5f9d7c2"
32) "sha256:4d62fa773ee07eeefe09afe6f054b57cb729d2bfc3578c1d4830c0a271ac08aa"
33) "sha256:022244a42edb560a222c438bc0e3a7a51ee2ee512381af02fbb95dd3a03089f2"
34) "sha256:9639d59a686e96521275800289762c777cacfbdf14dc0434ee0ac48f33d87f58"
35) "sha256:2db4ca3b312c6f4c4fad476523d224f3443330f34061a12494baccd4b3b735b5"
36) "sha256:ab32434d3734fb454f6b345eebacc2db0db1adc990ef481953c02e1c747ff4c4"
37) "sha256:0b392d63fa40ac78eb1ce66d8ef290a7185a2793c141f233a6cb7da56fbfe5b4"
38) "sha256:98351905c68907ee6fb47a375be94c2670e01d66089f1243e7fdcd20ec82e08f"
39) "sha256:f9202505c57535ef83349e992b042b6ea1bbffc83b12072c22328b648e0e8071"
40) "sha256:d70878c68b55b729320e69d172d4db8a54aecdd319f2c93d3688626fc470ad71"
41) "sha256:b04f3a3f39306b637d63bbc50394ba6fd2b4f62b3335f91d92e10cdfba74f81f"
42) "sha256:b858c1cf7b12eaf269f099e844f1492352d1c7fc05cea94f06d27056c8696af4"
43) "sha256:dad406b0352dd7987a9cfa78c7c470d52aa92afc57f81f2b2759228dd3d44d02"
44) "sha256:b8dac1515ef6fbece43eae740ec9b7ac1e217d8cdbd3f8af4cd468a8df9041c1"
45) "sha256:43488a5cac640b9941fbc35bc83e7a76c4818e23aa0eb4260de2fd0fd48fa59a"
46) "sha256:84f6be64c2fb77e0ae4b7806063e9caea7e9b62d541e5402211c93d4b8c1d135"
47) "sha256:d8ba795098528bea1e1ed68b86caf7da7280d6af8a1e86d9e0433dc403303640"
48) "sha256:7e5928aa10e10eef408e1e657d02799ea16f1b83ba1e3c167e250af7c4362cdf"
49) "sha256:a65fb5b70d1e78cf38f1f51ace27f02e2171fb05618a2147cf70fbc70d134c24"
50) "sha256:d38adb396606a721096018554077eff4fa1f2ed8b05343edf43d735dbc6323d2"
51) "sha256:60fd9e531cfc3aaf4596d56001b7ce17fa5dcfca09dc608ed6d1fedd74ff362b"
52) "sha256:fe377e8b6466ca14e2c948ccec5a7e8d8f21389dd537cfb34e67096f38ddb7fa"
53) "sha256:901cdc7ffc2cf0d70ca91a5fffa22d27c2b1273ed43348d2211734289b655d54"
54) "sha256:7c7b95d4bd7ca9c2c7f61533e128fbd3d2d6d628370c2c78ad45cadc59b88617"
55) "sha256:e01d5733b39bc21eff470c105819bb181c49e14b9ef42a83cf6b0dbf162f459e"
56) "sha256:d0c4633d271740903e10b64f37019843a29c6c6ac5c817e61b04360a92f1dd56"
57) "sha256:cfa32658670a998af0e47635ee194695aa54138b833b9d7964f8ca6c32411396"
58) "sha256:1f49037844d7b2c3fc630c4d22a3e99b8cac722755d65c229d4553be2e95fba9"
59) "sha256:a2bd9a3442c94785aaf09fa2a2ea060648379bfb43127e08f1b687a6af233b86"
60) "sha256:c4398f768745d8ea0d7bcd9120104136131add82e462ae747e1d4ae1103049a5"
61) "sha256:fa7164297a6da553f95fab2e3c51f32115664afdb153219add5d879674d8fc43"
62) "sha256:bea4c98c135acba444cb9c73dc59154256d1971f5974f6f0e635a464592f0f50"
63) "sha256:b309166981080a7e6a29101f0c5cfcf276b6132adbea21957739233ea653883b"
64) "sha256:d2f98a7cbf93dea125a9d45449cca7cb09c14c59829e5c9e7745029f49969728"
65) "sha256:9b2091813f13af8893c9f9f1389349c35356bcdfea86f14cbf0cedcddf564de2"
66) "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
67) "sha256:da2cecce4e3b88838a5d612a3ebbce440bb5a3dffb993f5f9ee3e31270e7cd88"
68) "sha256:7b04e34b88d37cf1d8957cdaf46841c2904bd749e60c2b57dcf0b7c9fda51360"
69) "sha256:500676f2d3b0d8903107b2ef8b2a74e1b64aa917ec399b53c1d4c829795b4a4f"
70) "sha256:e786f51ed20a2fd111befaebb2195478c32f8b37db48b1d9c6a443535df3552d"
71) "sha256:9c816581d1b1d789f64cd29a52ec1a32f9c32d3837288addd6211914ac5277af"
72) "sha256:5fbf0593ce10536be59d6d81e9f0894468953a47b243b10b52a09c474a3eaba3"
73) "sha256:e12dcc58d8bee09ece4395cfd9325ba45e646a19899ff0bfb80cec2c34d9320b"
74) "sha256:5c9ef1291f9b0477dc8cc21b043a36cd61c1e5d90c965e2319a80608a28aeab5"
  • Then finally, clean up all Swift files. In this case:
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/010f8a907a124799a9b068133ccbaf3e5ad07f9d4b98719891b9d2c43ae600d1/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/022244a42edb560a222c438bc0e3a7a51ee2ee512381af02fbb95dd3a03089f2/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/0b392d63fa40ac78eb1ce66d8ef290a7185a2793c141f233a6cb7da56fbfe5b4/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/0c51f8b393644f1594e7c4481fe0dbc2ccb7a1209ba3d7db014bca0237be82d4/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/1106614dbecabadb10017d8b220f0be81cdfedbf55eb69c959d99c5811b8eba2/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/129cd9268f2542b9b61be98363ce86c6a3cc7387674729af8fcbf899eeb31ec4/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/1f49037844d7b2c3fc630c4d22a3e99b8cac722755d65c229d4553be2e95fba9/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/2c93bba5818d8e23be37f243159576fd8e5068cd28ad9e66bea3330de955301e/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/2db4ca3b312c6f4c4fad476523d224f3443330f34061a12494baccd4b3b735b5/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/43488a5cac640b9941fbc35bc83e7a76c4818e23aa0eb4260de2fd0fd48fa59a/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/4b7e3e185519e0eac2496d54e7344fbdafe4c2bd817265cb63583ffc57e106af/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/4c3c3001bee901ef7d8a0e8204694a19e93ea078f30c251c8c5eb2d96f0254f3/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/4d4b7bd35dd80f2395e11506993b5109680410636d5897a38125ba980cb23466/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/4d62fa773ee07eeefe09afe6f054b57cb729d2bfc3578c1d4830c0a271ac08aa/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/500676f2d3b0d8903107b2ef8b2a74e1b64aa917ec399b53c1d4c829795b4a4f/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/5c9ef1291f9b0477dc8cc21b043a36cd61c1e5d90c965e2319a80608a28aeab5/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/5fbf0593ce10536be59d6d81e9f0894468953a47b243b10b52a09c474a3eaba3/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/60fd9e531cfc3aaf4596d56001b7ce17fa5dcfca09dc608ed6d1fedd74ff362b/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/73496f94b4099b89cc44892584a49ce49a5f4f9e445f548ed7a9ae8f99dadc3d/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/73df6afcb848b74ed3a1eb8ad2e450d4c9de2b4a7604514e2754bd080a382b29/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/7b04e34b88d37cf1d8957cdaf46841c2904bd749e60c2b57dcf0b7c9fda51360/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/7c7b95d4bd7ca9c2c7f61533e128fbd3d2d6d628370c2c78ad45cadc59b88617/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/7da1640ae982194ad5c32b92d9024f19cc1fa3b16507016328b0fb253e6adf6f/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/7e5928aa10e10eef408e1e657d02799ea16f1b83ba1e3c167e250af7c4362cdf/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/84f6be64c2fb77e0ae4b7806063e9caea7e9b62d541e5402211c93d4b8c1d135/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/8ac7a1b21ef06d253a5ffeea6fd52d386025c4466e1867be12865d66b4e8e3cc/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/901cdc7ffc2cf0d70ca91a5fffa22d27c2b1273ed43348d2211734289b655d54/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/9639d59a686e96521275800289762c777cacfbdf14dc0434ee0ac48f33d87f58/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/97f6a247bff7e4bf4204639265888595ad5786794af41d7c5ba480c69de03bda/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/98351905c68907ee6fb47a375be94c2670e01d66089f1243e7fdcd20ec82e08f/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/9b2091813f13af8893c9f9f1389349c35356bcdfea86f14cbf0cedcddf564de2/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/9b6d9807117f105a4c8dc96853f53967fe42054b0929de4f7e90951cef9e4822/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/9c816581d1b1d789f64cd29a52ec1a32f9c32d3837288addd6211914ac5277af/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a2bd9a3442c94785aaf09fa2a2ea060648379bfb43127e08f1b687a6af233b86/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a529ad64e3b7b8e4b6d1ff4938f10ac69fb14e217928d77648b8d095fae59832/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a58d2a0d47e1746c9e15feed2abc3b108c17aa0bcbc3994fe6d849760dac7d7c/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a65fb5b70d1e78cf38f1f51ace27f02e2171fb05618a2147cf70fbc70d134c24/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a9cc5f2c166b5b9af935b8d2d9eb9d24ddd645f36dad5031b011c52ae0903a92/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/ab32434d3734fb454f6b345eebacc2db0db1adc990ef481953c02e1c747ff4c4/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/ab66cf5625aea9e4536a0712747367371195e7f4801f488ff3ca1f4abd06cb25/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/ad6b363efe757163f82bff09e86bdb238d6796a2357cf837caf06bcf6d339ac6/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/ae927f96aa8deadbd0979370f8d99499e693fb52f860bf55f5f447120f815ee8/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/b04f3a3f39306b637d63bbc50394ba6fd2b4f62b3335f91d92e10cdfba74f81f/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/b309166981080a7e6a29101f0c5cfcf276b6132adbea21957739233ea653883b/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/b858c1cf7b12eaf269f099e844f1492352d1c7fc05cea94f06d27056c8696af4/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/b8dac1515ef6fbece43eae740ec9b7ac1e217d8cdbd3f8af4cd468a8df9041c1/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/bea4c98c135acba444cb9c73dc59154256d1971f5974f6f0e635a464592f0f50/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/c4398f768745d8ea0d7bcd9120104136131add82e462ae747e1d4ae1103049a5/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/c9c87b33a9fdb0677422e0587bf94ec2908c6a4b8433ab704042b2d7b2ea2811/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/cfa32658670a998af0e47635ee194695aa54138b833b9d7964f8ca6c32411396/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d08b7d87bf42969dfe810de01d775eec56e32b9d170b7c96a7fa836262e01ac4/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d0c4633d271740903e10b64f37019843a29c6c6ac5c817e61b04360a92f1dd56/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d2f98a7cbf93dea125a9d45449cca7cb09c14c59829e5c9e7745029f49969728/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d38adb396606a721096018554077eff4fa1f2ed8b05343edf43d735dbc6323d2/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d70878c68b55b729320e69d172d4db8a54aecdd319f2c93d3688626fc470ad71/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d8155d9e06681e2d82a17e38c9128deb486186190029c164195e37b2bb42ef30/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d8ba795098528bea1e1ed68b86caf7da7280d6af8a1e86d9e0433dc403303640/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/da2cecce4e3b88838a5d612a3ebbce440bb5a3dffb993f5f9ee3e31270e7cd88/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/dad406b0352dd7987a9cfa78c7c470d52aa92afc57f81f2b2759228dd3d44d02/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e01d5733b39bc21eff470c105819bb181c49e14b9ef42a83cf6b0dbf162f459e/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e12dcc58d8bee09ece4395cfd9325ba45e646a19899ff0bfb80cec2c34d9320b/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e356aede486dccb8b5129b498324e2f4e25ee3758600349495be83d1aa1ebd91/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e4fe83949190781c79d71bd38de18997d80e638abf2a51d8ae97fe0f12625ca9/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e786f51ed20a2fd111befaebb2195478c32f8b37db48b1d9c6a443535df3552d/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e834627d93cf269bae28acb64b75471b557923ed4c293414960180811ba35a55/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/eb0ebb16774921a2b983b1bf2df163f9b0b8d7e7d45dc23af7a95def7afa361b/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/ebe0c0daef21f30f0244e0244bf0bd9ffd8734ba1d6df3addc78f1fdd049dda9/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/f46ed641a06570d42ee992ec443e9eafb9d4b878658f1c9a9c176bccb5f9d7c2/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/f5713261151c029cf62b93e3920c19f39e1d8f89a1a8ea8001faed2aca41ec1a/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/f9202505c57535ef83349e992b042b6ea1bbffc83b12072c22328b648e0e8071/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/fa7164297a6da553f95fab2e3c51f32115664afdb153219add5d879674d8fc43/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/fb61622d721a05a25d3703532cd03b1a64793955bc23b0b6a062408ad884e83e/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/fe377e8b6466ca14e2c948ccec5a7e8d8f21389dd537cfb34e67096f38ddb7fa/link
files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_manifests/revisions/sha256/61ad04f1a21015b919479e60f3b1e2243f39ce564cb95660a53404e5a4b48ee4/link
  • And also every layer (example in case of 4b7e3e185519e0eac2496d54e7344fbdafe4c2bd817265cb63583ffc57e106af):
files/docker/registry/v2/blobs/sha256/4b/4b7e3e185519e0eac2496d54e7344fbdafe4c2bd817265cb63583ffc57e106af/data

This is something that we can do without downtime and/or read-only time. The ideal path would be that we had an API doing it for us, so we could also try to send a patch to upstream for the docker-registry's itself, but they dropped support for Swift in 3.x so we couldn't really go much far in convincing them. Maybe when we'll move to APU/S3 that could be an option, but in the meantime I'd try something more tailored for our use case. While upgrading we'll need to verify that the procedure works, but I don't think it is a big deal.

gmodena subscribed.Oct 10 2024, 5:30 PM
elukey added a comment.Oct 11 2024, 9:38 AM

I've created a Python script to dry-run what I highlighted above, this is how it would look like:

No tags, registryctl already used, safe to drop
elukey@build2001:~$ python3 registry_cleaner.py registry_cleaner.yaml "dev/stretch-php73-fpm"


Connecting to Swift..
Checking Swift container docker_registry_codfw..
Safety check: no tags should be listed for prefix files/docker/registry/v2/repositories/dev/stretch-php73-fpm
Proceeding in deleting data for files/docker/registry/v2/repositories/dev/stretch-php73-fpm
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/010f8a907a124799a9b068133ccbaf3e5ad07f9d4b98719891b9d2c43ae600d1/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/022244a42edb560a222c438bc0e3a7a51ee2ee512381af02fbb95dd3a03089f2/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/0b392d63fa40ac78eb1ce66d8ef290a7185a2793c141f233a6cb7da56fbfe5b4/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/0c51f8b393644f1594e7c4481fe0dbc2ccb7a1209ba3d7db014bca0237be82d4/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/1106614dbecabadb10017d8b220f0be81cdfedbf55eb69c959d99c5811b8eba2/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/129cd9268f2542b9b61be98363ce86c6a3cc7387674729af8fcbf899eeb31ec4/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/1f49037844d7b2c3fc630c4d22a3e99b8cac722755d65c229d4553be2e95fba9/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/2c93bba5818d8e23be37f243159576fd8e5068cd28ad9e66bea3330de955301e/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/2db4ca3b312c6f4c4fad476523d224f3443330f34061a12494baccd4b3b735b5/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/43488a5cac640b9941fbc35bc83e7a76c4818e23aa0eb4260de2fd0fd48fa59a/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/4b7e3e185519e0eac2496d54e7344fbdafe4c2bd817265cb63583ffc57e106af/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/4c3c3001bee901ef7d8a0e8204694a19e93ea078f30c251c8c5eb2d96f0254f3/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/4d4b7bd35dd80f2395e11506993b5109680410636d5897a38125ba980cb23466/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/4d62fa773ee07eeefe09afe6f054b57cb729d2bfc3578c1d4830c0a271ac08aa/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/500676f2d3b0d8903107b2ef8b2a74e1b64aa917ec399b53c1d4c829795b4a4f/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/5c9ef1291f9b0477dc8cc21b043a36cd61c1e5d90c965e2319a80608a28aeab5/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/5fbf0593ce10536be59d6d81e9f0894468953a47b243b10b52a09c474a3eaba3/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/60fd9e531cfc3aaf4596d56001b7ce17fa5dcfca09dc608ed6d1fedd74ff362b/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/73496f94b4099b89cc44892584a49ce49a5f4f9e445f548ed7a9ae8f99dadc3d/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/73df6afcb848b74ed3a1eb8ad2e450d4c9de2b4a7604514e2754bd080a382b29/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/7b04e34b88d37cf1d8957cdaf46841c2904bd749e60c2b57dcf0b7c9fda51360/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/7c7b95d4bd7ca9c2c7f61533e128fbd3d2d6d628370c2c78ad45cadc59b88617/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/7da1640ae982194ad5c32b92d9024f19cc1fa3b16507016328b0fb253e6adf6f/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/7e5928aa10e10eef408e1e657d02799ea16f1b83ba1e3c167e250af7c4362cdf/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/84f6be64c2fb77e0ae4b7806063e9caea7e9b62d541e5402211c93d4b8c1d135/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/8ac7a1b21ef06d253a5ffeea6fd52d386025c4466e1867be12865d66b4e8e3cc/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/901cdc7ffc2cf0d70ca91a5fffa22d27c2b1273ed43348d2211734289b655d54/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/9639d59a686e96521275800289762c777cacfbdf14dc0434ee0ac48f33d87f58/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/97f6a247bff7e4bf4204639265888595ad5786794af41d7c5ba480c69de03bda/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/98351905c68907ee6fb47a375be94c2670e01d66089f1243e7fdcd20ec82e08f/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/9b2091813f13af8893c9f9f1389349c35356bcdfea86f14cbf0cedcddf564de2/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/9b6d9807117f105a4c8dc96853f53967fe42054b0929de4f7e90951cef9e4822/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/9c816581d1b1d789f64cd29a52ec1a32f9c32d3837288addd6211914ac5277af/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a2bd9a3442c94785aaf09fa2a2ea060648379bfb43127e08f1b687a6af233b86/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a529ad64e3b7b8e4b6d1ff4938f10ac69fb14e217928d77648b8d095fae59832/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a58d2a0d47e1746c9e15feed2abc3b108c17aa0bcbc3994fe6d849760dac7d7c/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a65fb5b70d1e78cf38f1f51ace27f02e2171fb05618a2147cf70fbc70d134c24/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a9cc5f2c166b5b9af935b8d2d9eb9d24ddd645f36dad5031b011c52ae0903a92/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/ab32434d3734fb454f6b345eebacc2db0db1adc990ef481953c02e1c747ff4c4/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/ab66cf5625aea9e4536a0712747367371195e7f4801f488ff3ca1f4abd06cb25/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/ad6b363efe757163f82bff09e86bdb238d6796a2357cf837caf06bcf6d339ac6/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/ae927f96aa8deadbd0979370f8d99499e693fb52f860bf55f5f447120f815ee8/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/b04f3a3f39306b637d63bbc50394ba6fd2b4f62b3335f91d92e10cdfba74f81f/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/b309166981080a7e6a29101f0c5cfcf276b6132adbea21957739233ea653883b/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/b858c1cf7b12eaf269f099e844f1492352d1c7fc05cea94f06d27056c8696af4/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/b8dac1515ef6fbece43eae740ec9b7ac1e217d8cdbd3f8af4cd468a8df9041c1/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/bea4c98c135acba444cb9c73dc59154256d1971f5974f6f0e635a464592f0f50/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/c4398f768745d8ea0d7bcd9120104136131add82e462ae747e1d4ae1103049a5/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/c9c87b33a9fdb0677422e0587bf94ec2908c6a4b8433ab704042b2d7b2ea2811/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/cfa32658670a998af0e47635ee194695aa54138b833b9d7964f8ca6c32411396/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d08b7d87bf42969dfe810de01d775eec56e32b9d170b7c96a7fa836262e01ac4/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d0c4633d271740903e10b64f37019843a29c6c6ac5c817e61b04360a92f1dd56/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d2f98a7cbf93dea125a9d45449cca7cb09c14c59829e5c9e7745029f49969728/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d38adb396606a721096018554077eff4fa1f2ed8b05343edf43d735dbc6323d2/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d70878c68b55b729320e69d172d4db8a54aecdd319f2c93d3688626fc470ad71/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d8155d9e06681e2d82a17e38c9128deb486186190029c164195e37b2bb42ef30/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d8ba795098528bea1e1ed68b86caf7da7280d6af8a1e86d9e0433dc403303640/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/da2cecce4e3b88838a5d612a3ebbce440bb5a3dffb993f5f9ee3e31270e7cd88/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/dad406b0352dd7987a9cfa78c7c470d52aa92afc57f81f2b2759228dd3d44d02/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e01d5733b39bc21eff470c105819bb181c49e14b9ef42a83cf6b0dbf162f459e/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e12dcc58d8bee09ece4395cfd9325ba45e646a19899ff0bfb80cec2c34d9320b/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e356aede486dccb8b5129b498324e2f4e25ee3758600349495be83d1aa1ebd91/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e4fe83949190781c79d71bd38de18997d80e638abf2a51d8ae97fe0f12625ca9/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e786f51ed20a2fd111befaebb2195478c32f8b37db48b1d9c6a443535df3552d/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e834627d93cf269bae28acb64b75471b557923ed4c293414960180811ba35a55/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/eb0ebb16774921a2b983b1bf2df163f9b0b8d7e7d45dc23af7a95def7afa361b/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/ebe0c0daef21f30f0244e0244bf0bd9ffd8734ba1d6df3addc78f1fdd049dda9/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/f46ed641a06570d42ee992ec443e9eafb9d4b878658f1c9a9c176bccb5f9d7c2/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/f5713261151c029cf62b93e3920c19f39e1d8f89a1a8ea8001faed2aca41ec1a/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/f9202505c57535ef83349e992b042b6ea1bbffc83b12072c22328b648e0e8071/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/fa7164297a6da553f95fab2e3c51f32115664afdb153219add5d879674d8fc43/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/fb61622d721a05a25d3703532cd03b1a64793955bc23b0b6a062408ad884e83e/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/fe377e8b6466ca14e2c948ccec5a7e8d8f21389dd537cfb34e67096f38ddb7fa/link
DELETE: files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_manifests/revisions/sha256/61ad04f1a21015b919479e60f3b1e2243f39ce564cb95660a53404e5a4b48ee4/link
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/010f8a907a124799a9b068133ccbaf3e5ad07f9d4b98719891b9d2c43ae600d1/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/022244a42edb560a222c438bc0e3a7a51ee2ee512381af02fbb95dd3a03089f2/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/0b392d63fa40ac78eb1ce66d8ef290a7185a2793c141f233a6cb7da56fbfe5b4/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/0c51f8b393644f1594e7c4481fe0dbc2ccb7a1209ba3d7db014bca0237be82d4/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/1106614dbecabadb10017d8b220f0be81cdfedbf55eb69c959d99c5811b8eba2/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/129cd9268f2542b9b61be98363ce86c6a3cc7387674729af8fcbf899eeb31ec4/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/1f49037844d7b2c3fc630c4d22a3e99b8cac722755d65c229d4553be2e95fba9/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/2c93bba5818d8e23be37f243159576fd8e5068cd28ad9e66bea3330de955301e/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/2db4ca3b312c6f4c4fad476523d224f3443330f34061a12494baccd4b3b735b5/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/43488a5cac640b9941fbc35bc83e7a76c4818e23aa0eb4260de2fd0fd48fa59a/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/4b7e3e185519e0eac2496d54e7344fbdafe4c2bd817265cb63583ffc57e106af/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/4c3c3001bee901ef7d8a0e8204694a19e93ea078f30c251c8c5eb2d96f0254f3/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/4d4b7bd35dd80f2395e11506993b5109680410636d5897a38125ba980cb23466/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/4d62fa773ee07eeefe09afe6f054b57cb729d2bfc3578c1d4830c0a271ac08aa/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/500676f2d3b0d8903107b2ef8b2a74e1b64aa917ec399b53c1d4c829795b4a4f/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/5c9ef1291f9b0477dc8cc21b043a36cd61c1e5d90c965e2319a80608a28aeab5/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/5fbf0593ce10536be59d6d81e9f0894468953a47b243b10b52a09c474a3eaba3/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/60fd9e531cfc3aaf4596d56001b7ce17fa5dcfca09dc608ed6d1fedd74ff362b/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/73496f94b4099b89cc44892584a49ce49a5f4f9e445f548ed7a9ae8f99dadc3d/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/73df6afcb848b74ed3a1eb8ad2e450d4c9de2b4a7604514e2754bd080a382b29/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/7b04e34b88d37cf1d8957cdaf46841c2904bd749e60c2b57dcf0b7c9fda51360/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/7c7b95d4bd7ca9c2c7f61533e128fbd3d2d6d628370c2c78ad45cadc59b88617/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/7da1640ae982194ad5c32b92d9024f19cc1fa3b16507016328b0fb253e6adf6f/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/7e5928aa10e10eef408e1e657d02799ea16f1b83ba1e3c167e250af7c4362cdf/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/84f6be64c2fb77e0ae4b7806063e9caea7e9b62d541e5402211c93d4b8c1d135/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/8ac7a1b21ef06d253a5ffeea6fd52d386025c4466e1867be12865d66b4e8e3cc/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/901cdc7ffc2cf0d70ca91a5fffa22d27c2b1273ed43348d2211734289b655d54/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/9639d59a686e96521275800289762c777cacfbdf14dc0434ee0ac48f33d87f58/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/97f6a247bff7e4bf4204639265888595ad5786794af41d7c5ba480c69de03bda/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/98351905c68907ee6fb47a375be94c2670e01d66089f1243e7fdcd20ec82e08f/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/9b2091813f13af8893c9f9f1389349c35356bcdfea86f14cbf0cedcddf564de2/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/9b6d9807117f105a4c8dc96853f53967fe42054b0929de4f7e90951cef9e4822/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/9c816581d1b1d789f64cd29a52ec1a32f9c32d3837288addd6211914ac5277af/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a2bd9a3442c94785aaf09fa2a2ea060648379bfb43127e08f1b687a6af233b86/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a529ad64e3b7b8e4b6d1ff4938f10ac69fb14e217928d77648b8d095fae59832/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a58d2a0d47e1746c9e15feed2abc3b108c17aa0bcbc3994fe6d849760dac7d7c/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a65fb5b70d1e78cf38f1f51ace27f02e2171fb05618a2147cf70fbc70d134c24/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/a9cc5f2c166b5b9af935b8d2d9eb9d24ddd645f36dad5031b011c52ae0903a92/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/ab32434d3734fb454f6b345eebacc2db0db1adc990ef481953c02e1c747ff4c4/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/ab66cf5625aea9e4536a0712747367371195e7f4801f488ff3ca1f4abd06cb25/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/ad6b363efe757163f82bff09e86bdb238d6796a2357cf837caf06bcf6d339ac6/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/ae927f96aa8deadbd0979370f8d99499e693fb52f860bf55f5f447120f815ee8/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/b04f3a3f39306b637d63bbc50394ba6fd2b4f62b3335f91d92e10cdfba74f81f/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/b309166981080a7e6a29101f0c5cfcf276b6132adbea21957739233ea653883b/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/b858c1cf7b12eaf269f099e844f1492352d1c7fc05cea94f06d27056c8696af4/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/b8dac1515ef6fbece43eae740ec9b7ac1e217d8cdbd3f8af4cd468a8df9041c1/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/bea4c98c135acba444cb9c73dc59154256d1971f5974f6f0e635a464592f0f50/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/c4398f768745d8ea0d7bcd9120104136131add82e462ae747e1d4ae1103049a5/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/c9c87b33a9fdb0677422e0587bf94ec2908c6a4b8433ab704042b2d7b2ea2811/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/cfa32658670a998af0e47635ee194695aa54138b833b9d7964f8ca6c32411396/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d08b7d87bf42969dfe810de01d775eec56e32b9d170b7c96a7fa836262e01ac4/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d0c4633d271740903e10b64f37019843a29c6c6ac5c817e61b04360a92f1dd56/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d2f98a7cbf93dea125a9d45449cca7cb09c14c59829e5c9e7745029f49969728/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d38adb396606a721096018554077eff4fa1f2ed8b05343edf43d735dbc6323d2/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d70878c68b55b729320e69d172d4db8a54aecdd319f2c93d3688626fc470ad71/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d8155d9e06681e2d82a17e38c9128deb486186190029c164195e37b2bb42ef30/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/d8ba795098528bea1e1ed68b86caf7da7280d6af8a1e86d9e0433dc403303640/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/da2cecce4e3b88838a5d612a3ebbce440bb5a3dffb993f5f9ee3e31270e7cd88/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/dad406b0352dd7987a9cfa78c7c470d52aa92afc57f81f2b2759228dd3d44d02/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e01d5733b39bc21eff470c105819bb181c49e14b9ef42a83cf6b0dbf162f459e/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e12dcc58d8bee09ece4395cfd9325ba45e646a19899ff0bfb80cec2c34d9320b/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e356aede486dccb8b5129b498324e2f4e25ee3758600349495be83d1aa1ebd91/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e4fe83949190781c79d71bd38de18997d80e638abf2a51d8ae97fe0f12625ca9/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e786f51ed20a2fd111befaebb2195478c32f8b37db48b1d9c6a443535df3552d/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/e834627d93cf269bae28acb64b75471b557923ed4c293414960180811ba35a55/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/eb0ebb16774921a2b983b1bf2df163f9b0b8d7e7d45dc23af7a95def7afa361b/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/ebe0c0daef21f30f0244e0244bf0bd9ffd8734ba1d6df3addc78f1fdd049dda9/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/f46ed641a06570d42ee992ec443e9eafb9d4b878658f1c9a9c176bccb5f9d7c2/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/f5713261151c029cf62b93e3920c19f39e1d8f89a1a8ea8001faed2aca41ec1a/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/f9202505c57535ef83349e992b042b6ea1bbffc83b12072c22328b648e0e8071/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/fa7164297a6da553f95fab2e3c51f32115664afdb153219add5d879674d8fc43/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/fb61622d721a05a25d3703532cd03b1a64793955bc23b0b6a062408ad884e83e/data
DELETE files/docker/registry/v2/blobs/sha256/fi/files/docker/registry/v2/repositories/dev/stretch-php73-fpm/_layers/sha256/fe377e8b6466ca14e2c948ccec5a7e8d8f21389dd537cfb34e67096f38ddb7fa/data
DELETE files/docker/registry/v2/blobs/sha256/61/61ad04f1a21015b919479e60f3b1e2243f39ce564cb95660a53404e5a4b48ee4/data


Connecting to Redis..
Deleting repository keys..
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:e01d5733b39bc21eff470c105819bb181c49e14b9ef42a83cf6b0dbf162f459e'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:a65fb5b70d1e78cf38f1f51ace27f02e2171fb05618a2147cf70fbc70d134c24'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:d0c4633d271740903e10b64f37019843a29c6c6ac5c817e61b04360a92f1dd56'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:fb61622d721a05a25d3703532cd03b1a64793955bc23b0b6a062408ad884e83e'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:4d62fa773ee07eeefe09afe6f054b57cb729d2bfc3578c1d4830c0a271ac08aa'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:d08b7d87bf42969dfe810de01d775eec56e32b9d170b7c96a7fa836262e01ac4'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:5fbf0593ce10536be59d6d81e9f0894468953a47b243b10b52a09c474a3eaba3'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:9b6d9807117f105a4c8dc96853f53967fe42054b0929de4f7e90951cef9e4822'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:f46ed641a06570d42ee992ec443e9eafb9d4b878658f1c9a9c176bccb5f9d7c2'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:d8ba795098528bea1e1ed68b86caf7da7280d6af8a1e86d9e0433dc403303640'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:1106614dbecabadb10017d8b220f0be81cdfedbf55eb69c959d99c5811b8eba2'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:a2bd9a3442c94785aaf09fa2a2ea060648379bfb43127e08f1b687a6af233b86'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:fe377e8b6466ca14e2c948ccec5a7e8d8f21389dd537cfb34e67096f38ddb7fa'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:da2cecce4e3b88838a5d612a3ebbce440bb5a3dffb993f5f9ee3e31270e7cd88'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:4c3c3001bee901ef7d8a0e8204694a19e93ea078f30c251c8c5eb2d96f0254f3'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:a529ad64e3b7b8e4b6d1ff4938f10ac69fb14e217928d77648b8d095fae59832'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:500676f2d3b0d8903107b2ef8b2a74e1b64aa917ec399b53c1d4c829795b4a4f'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:ae927f96aa8deadbd0979370f8d99499e693fb52f860bf55f5f447120f815ee8'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:a58d2a0d47e1746c9e15feed2abc3b108c17aa0bcbc3994fe6d849760dac7d7c'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:e834627d93cf269bae28acb64b75471b557923ed4c293414960180811ba35a55'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:98351905c68907ee6fb47a375be94c2670e01d66089f1243e7fdcd20ec82e08f'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:a9cc5f2c166b5b9af935b8d2d9eb9d24ddd645f36dad5031b011c52ae0903a92'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:97f6a247bff7e4bf4204639265888595ad5786794af41d7c5ba480c69de03bda'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:60fd9e531cfc3aaf4596d56001b7ce17fa5dcfca09dc608ed6d1fedd74ff362b'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:b309166981080a7e6a29101f0c5cfcf276b6132adbea21957739233ea653883b'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:9b2091813f13af8893c9f9f1389349c35356bcdfea86f14cbf0cedcddf564de2'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:b858c1cf7b12eaf269f099e844f1492352d1c7fc05cea94f06d27056c8696af4'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:8ac7a1b21ef06d253a5ffeea6fd52d386025c4466e1867be12865d66b4e8e3cc'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:4d4b7bd35dd80f2395e11506993b5109680410636d5897a38125ba980cb23466'
DELETE b'repository::dev/stretch-php73-fpm::blobs'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:b8dac1515ef6fbece43eae740ec9b7ac1e217d8cdbd3f8af4cd468a8df9041c1'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:cfa32658670a998af0e47635ee194695aa54138b833b9d7964f8ca6c32411396'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:4b7e3e185519e0eac2496d54e7344fbdafe4c2bd817265cb63583ffc57e106af'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:d70878c68b55b729320e69d172d4db8a54aecdd319f2c93d3688626fc470ad71'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:ab66cf5625aea9e4536a0712747367371195e7f4801f488ff3ca1f4abd06cb25'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:0c51f8b393644f1594e7c4481fe0dbc2ccb7a1209ba3d7db014bca0237be82d4'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:c9c87b33a9fdb0677422e0587bf94ec2908c6a4b8433ab704042b2d7b2ea2811'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:901cdc7ffc2cf0d70ca91a5fffa22d27c2b1273ed43348d2211734289b655d54'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:dad406b0352dd7987a9cfa78c7c470d52aa92afc57f81f2b2759228dd3d44d02'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:bea4c98c135acba444cb9c73dc59154256d1971f5974f6f0e635a464592f0f50'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:e786f51ed20a2fd111befaebb2195478c32f8b37db48b1d9c6a443535df3552d'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:7c7b95d4bd7ca9c2c7f61533e128fbd3d2d6d628370c2c78ad45cadc59b88617'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:b04f3a3f39306b637d63bbc50394ba6fd2b4f62b3335f91d92e10cdfba74f81f'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:84f6be64c2fb77e0ae4b7806063e9caea7e9b62d541e5402211c93d4b8c1d135'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:e356aede486dccb8b5129b498324e2f4e25ee3758600349495be83d1aa1ebd91'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:d38adb396606a721096018554077eff4fa1f2ed8b05343edf43d735dbc6323d2'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:9639d59a686e96521275800289762c777cacfbdf14dc0434ee0ac48f33d87f58'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:7e5928aa10e10eef408e1e657d02799ea16f1b83ba1e3c167e250af7c4362cdf'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:ab32434d3734fb454f6b345eebacc2db0db1adc990ef481953c02e1c747ff4c4'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:e12dcc58d8bee09ece4395cfd9325ba45e646a19899ff0bfb80cec2c34d9320b'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:5c9ef1291f9b0477dc8cc21b043a36cd61c1e5d90c965e2319a80608a28aeab5'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:129cd9268f2542b9b61be98363ce86c6a3cc7387674729af8fcbf899eeb31ec4'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:7da1640ae982194ad5c32b92d9024f19cc1fa3b16507016328b0fb253e6adf6f'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:fa7164297a6da553f95fab2e3c51f32115664afdb153219add5d879674d8fc43'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:ebe0c0daef21f30f0244e0244bf0bd9ffd8734ba1d6df3addc78f1fdd049dda9'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:73df6afcb848b74ed3a1eb8ad2e450d4c9de2b4a7604514e2754bd080a382b29'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:c4398f768745d8ea0d7bcd9120104136131add82e462ae747e1d4ae1103049a5'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:f5713261151c029cf62b93e3920c19f39e1d8f89a1a8ea8001faed2aca41ec1a'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:d8155d9e06681e2d82a17e38c9128deb486186190029c164195e37b2bb42ef30'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:e4fe83949190781c79d71bd38de18997d80e638abf2a51d8ae97fe0f12625ca9'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:2db4ca3b312c6f4c4fad476523d224f3443330f34061a12494baccd4b3b735b5'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:f9202505c57535ef83349e992b042b6ea1bbffc83b12072c22328b648e0e8071'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:eb0ebb16774921a2b983b1bf2df163f9b0b8d7e7d45dc23af7a95def7afa361b'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:0b392d63fa40ac78eb1ce66d8ef290a7185a2793c141f233a6cb7da56fbfe5b4'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:ad6b363efe757163f82bff09e86bdb238d6796a2357cf837caf06bcf6d339ac6'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:9c816581d1b1d789f64cd29a52ec1a32f9c32d3837288addd6211914ac5277af'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:43488a5cac640b9941fbc35bc83e7a76c4818e23aa0eb4260de2fd0fd48fa59a'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:7b04e34b88d37cf1d8957cdaf46841c2904bd749e60c2b57dcf0b7c9fda51360'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:2c93bba5818d8e23be37f243159576fd8e5068cd28ad9e66bea3330de955301e'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:d2f98a7cbf93dea125a9d45449cca7cb09c14c59829e5c9e7745029f49969728'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:1f49037844d7b2c3fc630c4d22a3e99b8cac722755d65c229d4553be2e95fba9'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:010f8a907a124799a9b068133ccbaf3e5ad07f9d4b98719891b9d2c43ae600d1'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:73496f94b4099b89cc44892584a49ce49a5f4f9e445f548ed7a9ae8f99dadc3d'
DELETE b'repository::dev/stretch-php73-fpm::blobs::sha256:022244a42edb560a222c438bc0e3a7a51ee2ee512381af02fbb95dd3a03089f2'
Deleting sha256 blobs' metadata..
DELETE sha256:e01d5733b39bc21eff470c105819bb181c49e14b9ef42a83cf6b0dbf162f459e
DELETE sha256:a65fb5b70d1e78cf38f1f51ace27f02e2171fb05618a2147cf70fbc70d134c24
DELETE sha256:d0c4633d271740903e10b64f37019843a29c6c6ac5c817e61b04360a92f1dd56
DELETE sha256:fb61622d721a05a25d3703532cd03b1a64793955bc23b0b6a062408ad884e83e
DELETE sha256:4d62fa773ee07eeefe09afe6f054b57cb729d2bfc3578c1d4830c0a271ac08aa
DELETE sha256:d08b7d87bf42969dfe810de01d775eec56e32b9d170b7c96a7fa836262e01ac4
DELETE sha256:5fbf0593ce10536be59d6d81e9f0894468953a47b243b10b52a09c474a3eaba3
DELETE sha256:9b6d9807117f105a4c8dc96853f53967fe42054b0929de4f7e90951cef9e4822
DELETE sha256:f46ed641a06570d42ee992ec443e9eafb9d4b878658f1c9a9c176bccb5f9d7c2
DELETE sha256:d8ba795098528bea1e1ed68b86caf7da7280d6af8a1e86d9e0433dc403303640
DELETE sha256:1106614dbecabadb10017d8b220f0be81cdfedbf55eb69c959d99c5811b8eba2
DELETE sha256:a2bd9a3442c94785aaf09fa2a2ea060648379bfb43127e08f1b687a6af233b86
DELETE sha256:fe377e8b6466ca14e2c948ccec5a7e8d8f21389dd537cfb34e67096f38ddb7fa
DELETE sha256:da2cecce4e3b88838a5d612a3ebbce440bb5a3dffb993f5f9ee3e31270e7cd88
DELETE sha256:4c3c3001bee901ef7d8a0e8204694a19e93ea078f30c251c8c5eb2d96f0254f3
DELETE sha256:a529ad64e3b7b8e4b6d1ff4938f10ac69fb14e217928d77648b8d095fae59832
DELETE sha256:500676f2d3b0d8903107b2ef8b2a74e1b64aa917ec399b53c1d4c829795b4a4f
DELETE sha256:ae927f96aa8deadbd0979370f8d99499e693fb52f860bf55f5f447120f815ee8
DELETE sha256:a58d2a0d47e1746c9e15feed2abc3b108c17aa0bcbc3994fe6d849760dac7d7c
DELETE sha256:e834627d93cf269bae28acb64b75471b557923ed4c293414960180811ba35a55
DELETE sha256:98351905c68907ee6fb47a375be94c2670e01d66089f1243e7fdcd20ec82e08f
DELETE sha256:a9cc5f2c166b5b9af935b8d2d9eb9d24ddd645f36dad5031b011c52ae0903a92
DELETE sha256:97f6a247bff7e4bf4204639265888595ad5786794af41d7c5ba480c69de03bda
DELETE sha256:60fd9e531cfc3aaf4596d56001b7ce17fa5dcfca09dc608ed6d1fedd74ff362b
DELETE sha256:b309166981080a7e6a29101f0c5cfcf276b6132adbea21957739233ea653883b
DELETE sha256:9b2091813f13af8893c9f9f1389349c35356bcdfea86f14cbf0cedcddf564de2
DELETE sha256:b858c1cf7b12eaf269f099e844f1492352d1c7fc05cea94f06d27056c8696af4
DELETE sha256:8ac7a1b21ef06d253a5ffeea6fd52d386025c4466e1867be12865d66b4e8e3cc
DELETE sha256:4d4b7bd35dd80f2395e11506993b5109680410636d5897a38125ba980cb23466
DELETE sha256:b8dac1515ef6fbece43eae740ec9b7ac1e217d8cdbd3f8af4cd468a8df9041c1
DELETE sha256:cfa32658670a998af0e47635ee194695aa54138b833b9d7964f8ca6c32411396
DELETE sha256:4b7e3e185519e0eac2496d54e7344fbdafe4c2bd817265cb63583ffc57e106af
DELETE sha256:d70878c68b55b729320e69d172d4db8a54aecdd319f2c93d3688626fc470ad71
DELETE sha256:ab66cf5625aea9e4536a0712747367371195e7f4801f488ff3ca1f4abd06cb25
DELETE sha256:0c51f8b393644f1594e7c4481fe0dbc2ccb7a1209ba3d7db014bca0237be82d4
DELETE sha256:c9c87b33a9fdb0677422e0587bf94ec2908c6a4b8433ab704042b2d7b2ea2811
DELETE sha256:901cdc7ffc2cf0d70ca91a5fffa22d27c2b1273ed43348d2211734289b655d54
DELETE sha256:dad406b0352dd7987a9cfa78c7c470d52aa92afc57f81f2b2759228dd3d44d02
DELETE sha256:bea4c98c135acba444cb9c73dc59154256d1971f5974f6f0e635a464592f0f50
DELETE sha256:e786f51ed20a2fd111befaebb2195478c32f8b37db48b1d9c6a443535df3552d
DELETE sha256:7c7b95d4bd7ca9c2c7f61533e128fbd3d2d6d628370c2c78ad45cadc59b88617
DELETE sha256:b04f3a3f39306b637d63bbc50394ba6fd2b4f62b3335f91d92e10cdfba74f81f
DELETE sha256:84f6be64c2fb77e0ae4b7806063e9caea7e9b62d541e5402211c93d4b8c1d135
DELETE sha256:e356aede486dccb8b5129b498324e2f4e25ee3758600349495be83d1aa1ebd91
DELETE sha256:d38adb396606a721096018554077eff4fa1f2ed8b05343edf43d735dbc6323d2
DELETE sha256:9639d59a686e96521275800289762c777cacfbdf14dc0434ee0ac48f33d87f58
DELETE sha256:7e5928aa10e10eef408e1e657d02799ea16f1b83ba1e3c167e250af7c4362cdf
DELETE sha256:ab32434d3734fb454f6b345eebacc2db0db1adc990ef481953c02e1c747ff4c4
DELETE sha256:e12dcc58d8bee09ece4395cfd9325ba45e646a19899ff0bfb80cec2c34d9320b
DELETE sha256:5c9ef1291f9b0477dc8cc21b043a36cd61c1e5d90c965e2319a80608a28aeab5
DELETE sha256:129cd9268f2542b9b61be98363ce86c6a3cc7387674729af8fcbf899eeb31ec4
DELETE sha256:7da1640ae982194ad5c32b92d9024f19cc1fa3b16507016328b0fb253e6adf6f
DELETE sha256:fa7164297a6da553f95fab2e3c51f32115664afdb153219add5d879674d8fc43
DELETE sha256:ebe0c0daef21f30f0244e0244bf0bd9ffd8734ba1d6df3addc78f1fdd049dda9
DELETE sha256:73df6afcb848b74ed3a1eb8ad2e450d4c9de2b4a7604514e2754bd080a382b29
DELETE sha256:c4398f768745d8ea0d7bcd9120104136131add82e462ae747e1d4ae1103049a5
DELETE sha256:f5713261151c029cf62b93e3920c19f39e1d8f89a1a8ea8001faed2aca41ec1a
DELETE sha256:d8155d9e06681e2d82a17e38c9128deb486186190029c164195e37b2bb42ef30
DELETE sha256:e4fe83949190781c79d71bd38de18997d80e638abf2a51d8ae97fe0f12625ca9
DELETE sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
DELETE sha256:2db4ca3b312c6f4c4fad476523d224f3443330f34061a12494baccd4b3b735b5
DELETE sha256:f9202505c57535ef83349e992b042b6ea1bbffc83b12072c22328b648e0e8071
DELETE sha256:eb0ebb16774921a2b983b1bf2df163f9b0b8d7e7d45dc23af7a95def7afa361b
DELETE sha256:0b392d63fa40ac78eb1ce66d8ef290a7185a2793c141f233a6cb7da56fbfe5b4
DELETE sha256:ad6b363efe757163f82bff09e86bdb238d6796a2357cf837caf06bcf6d339ac6
DELETE sha256:9c816581d1b1d789f64cd29a52ec1a32f9c32d3837288addd6211914ac5277af
DELETE sha256:43488a5cac640b9941fbc35bc83e7a76c4818e23aa0eb4260de2fd0fd48fa59a
DELETE sha256:7b04e34b88d37cf1d8957cdaf46841c2904bd749e60c2b57dcf0b7c9fda51360
DELETE sha256:2c93bba5818d8e23be37f243159576fd8e5068cd28ad9e66bea3330de955301e
DELETE sha256:d2f98a7cbf93dea125a9d45449cca7cb09c14c59829e5c9e7745029f49969728
DELETE sha256:1f49037844d7b2c3fc630c4d22a3e99b8cac722755d65c229d4553be2e95fba9
DELETE sha256:010f8a907a124799a9b068133ccbaf3e5ad07f9d4b98719891b9d2c43ae600d1
DELETE sha256:73496f94b4099b89cc44892584a49ce49a5f4f9e445f548ed7a9ae8f99dadc3d
DELETE sha256:022244a42edb560a222c438bc0e3a7a51ee2ee512381af02fbb95dd3a03089f2
Found tags, registryctl not used, unsafe to drop
elukey@build2001:~$ python3 registry_cleaner.py registry_cleaner.yaml "rsyslog"


Connecting to Swift..
Checking Swift container docker_registry_codfw..
Safety check: no tags should be listed for prefix files/docker/registry/v2/repositories/rsyslog
Traceback (most recent call last):
  File "/home/elukey/registry_cleaner.py", line 59, in <module>
    delete_stale_objects(swift, args.repository)
  File "/home/elukey/registry_cleaner.py", line 29, in delete_stale_objects
    raise RuntimeError(
RuntimeError: Found at least one object with '/tags/' in its path, the data needs to be removed via registryctl first, aborting..
Code (prototype, it needs further refinement)
elukey@build2001:~$ cat registry_cleaner.py 
#!/usr/bin/python3

from wmflib.config import load_yaml_config
import argparse
import redis
import swiftclient

def delete_stale_keys(redis, repository):
    blobs_to_delete = []
    print("Deleting repository keys..")
    for key in redis.scan_iter(f"repository::{repository}::*"):
        blob_prefix = f"repository::{repository}::blobs::sha256:"
        if key.decode().startswith(blob_prefix):
            sha256 = key.decode().replace(blob_prefix, '')
            blobs_to_delete.append(sha256)
        print(f"DELETE {key}")
    print("Deleting sha256 blobs' metadata..")
    for blob in blobs_to_delete:
        print(f"DELETE sha256:{blob}")


def delete_stale_objects(swift, repository):
    for container in swift.get_account()[1]:
        prefix = f"files/docker/registry/v2/repositories/{repository}"
        print(f"Checking Swift container {container['name']}..")
        print(f"Safety check: no tags should be listed for prefix {prefix}")
        for data in swift.get_container(container['name'], prefix=prefix)[1]:
            if "/tags" in data['name']:
                raise RuntimeError(
                    f"Found at least one object with '/tags/' in its path, "
                    "the data needs to be removed via registryctl first, aborting.."
                )
        print(f"Proceeding in deleting data for {prefix}")
        blobs_to_delete = []
        for data in swift.get_container(container['name'], prefix=prefix)[1]:
            print(f"DELETE: {data['name']}")
            if "sha256" in data['name']:
                prefix = f"files/docker/registry/v2/repositories/{repository}/_manifests/revisions/sha256/"
                blob = data['name'].replace(prefix, '').replace('/link', '')
                blobs_to_delete.append(blob)
        for blob in blobs_to_delete:
            print(f"DELETE files/docker/registry/v2/blobs/sha256/{blob[0:2]}/{blob}/data")


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument("config", type=str)
    parser.add_argument("repository", type=str)
    args = parser.parse_args()
    config = load_yaml_config(args.config)

    print("\n\nConnecting to Swift..")
    swift = swiftclient.client.Connection(
        user=config["swift"]["user"],
        key=config["swift"]["key"],
        authurl=config["swift"]["authurl"],
        auth_version=1
    )
    delete_stale_objects(swift, args.repository)

    print("\n\nConnecting to Redis..")
    redis = redis.Redis(
        host=config["redis"]["hostname"],
        port=config["redis"]["port"],
        password=config["redis"]["password"],
    )
    delete_stale_keys(redis, args.repository)
elukey triaged this task as Medium priority.Oct 14 2024, 2:39 PM
BTullis subscribed.Nov 26 2024, 5:49 PM
elukey closed this task as Declined.Nov 27 2024, 8:48 AM

The K8s SIG reviewed this proposal and for the moment it was decided not to proceed with anything that could harm the consistency of the Registry. We'll probably form a group of people interested in maintaining the registry, and after that a decision about how to proceed will be made.

elukey mentioned this in T394476: Onboard the Docker Registry to apus.May 16 2025, 8:22 AM
elukey added a comment.May 16 2025, 2:23 PM
In T375645#10194826, @elukey wrote:

It failed with:

failed to garbage collect: failed to mark: swift: swift: failed to retrieve tags unknown repository name=dev/stretch-php73-fpm

This is an image that we deleted with requestctl, so IIUC it is present in the catalog but not on Swift. After a chat with Janis we agreed that the garbage collector feature of Docker Distribution is probably a little less experimental than the docker-distribution-pruner, so we may just test the latter in dry-run mode and see how it goes. The caveat is that it doesn't support Swift natively, but only the s3 API, so we'll see if we can adapt to it.

To keep archives happy, checked garbagecollect.go today and I found this commit on 3.x:

https://github.com/distribution/distribution/commit/601b37d98b9330bd66137cdc89a01896cf7a481d

This bit is very interesting:

if _, ok := err.(distribution.ErrManifestUnknownRevision); !ok {
     return nil
}

My reading is that if ErrManifestUnknownRevision is raised (and it happens when a repository doesn't have tags anymore, like after we registryctl-delete an image), the repository gets simply skipped and it shouldn't cause a GC failure. This is good on one side, but bad since the code doesn't handle any real clean up after all tags are deleted.

BTullis mentioned this in T413080: Design and build the next generation of container-registry service for the WMF production realm.Dec 18 2025, 1:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits