Page MenuHomePhabricator

docker-pkg should attempt to pull dependent images from the registry
Closed, ResolvedPublic

Description

On a fresh machine, docker-pkg will build images from scratch and tag them with the version from the changelog. However the freshly build image would not match the one in the registry.

  • a new image might come with different package versions
  • forces us to keep all the images on the build host until they are no more needed
  • if an image is not locally present, editing the Dockerfile.template without changing the changelog cause the local image to be entirely different

It seems to me it is better to be in sync with the Docker registry.

Event Timeline

hashar created this task.Jul 30 2018, 7:37 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 30 2018, 7:37 PM
Joe added a comment.Jul 31 2018, 10:22 AM

I guess we should add a command line switch to jump between the two behaviours.

hashar added a comment.Aug 1 2018, 8:33 AM

I would rather enforce it. If a container image:0.0.1 has been published to the registry, surely we should not rebuild one locally that will be different? docker-pkg should use the exact same image and hence pull it, else one would need to do a version bump.

I don't have a practical example, but that has hit me several time when porting the CI containers to docker-pkg. And still it me from time to time when I lack an image locally. I ended up writing a shell one liner to sync them before running docker-pkg:

docker-pkg
grep -R -P -h -o 'docker-registry.*:[.\d]+' jjb|sort|uniq|xargs --verbose -n1 docker pull
docker-pkg "$@"

(where the grep commands list the images as for example: docker-registry.wikimedia.org/releng/tox-mysqld:0.1.0.

Change 475843 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/docker-images/docker-pkg@master] Support docker build pull and cache

https://gerrit.wikimedia.org/r/475843

hashar moved this task from Backlog to Patch for review on the docker-pkg board.Nov 26 2018, 9:43 PM
hashar claimed this task.
Joe added a comment.Nov 27 2018, 10:12 AM

I would rather enforce it. If a container image:0.0.1 has been published to the registry, surely we should not rebuild one locally that will be different? docker-pkg should use the exact same image and hence pull it, else one would need to do a version bump.

I don't have a practical example, but that has hit me several time when porting the CI containers to docker-pkg. And still it me from time to time when I lack an image locally. I ended up writing a shell one liner to sync them before running docker-pkg:

docker-pkg
grep -R -P -h -o 'docker-registry.*:[.\d]+' jjb|sort|uniq|xargs --verbose -n1 docker pull
docker-pkg "$@"

(where the grep commands list the images as for example: docker-registry.wikimedia.org/releng/tox-mysqld:0.1.0.

so your proposal is to make docker-pkg do the following:
1 - check dependencies. If they're not available locally, try to pull them (at the exact tag in the changelog) from the registry. This works as long as our changelogs are updated, and it's ok I guess.

When a containers is already available locally, docker-pkg skips building it, so one always have to bump the changelog :)

Change 475843 merged by jenkins-bot:
[operations/docker-images/docker-pkg@master] Attempt to pull images before building

https://gerrit.wikimedia.org/r/475843

Joe closed this task as Resolved.Jan 20 2019, 12:03 PM