If the tests from integration/config maybe be generalized, maybe they could be added as a new lint sub command to docker-pkg?

Change 731149 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/docker-images/docker-pkg@master] Introduce lint command

https://gerrit.wikimedia.org/r/731149

Change 731149 abandoned by Hashar:

[operations/docker-images/docker-pkg@master] Introduce lint command

Reason:

I might revisit another time

https://gerrit.wikimedia.org/r/731149

I agree with @MLechvien-WMF that this is a nice-to-have only, but I would also like to add that I lost a lot of time Monday from duplicating a docker image name (see this ugly series of commits ). Although my changes were reviewed by several people, it took @RLazarus 's sharp eyes to finally spot the problem.

CI would have helped, although I would suggest we look at things more broadly. docker-pkg is WMF-specific code that doesn't seem to have been updated in some time. I was unable to get it to work in a virtualenv on my MacOS laptop. As we start looking at revamping the container registry service in T413080, maybe it's worth thinking about a more sustainable way to manage production images?

I am pretty sure Kunal filed this task referring to the few unit tests I wrote in integration/config to catch all the basic issues one can make when updating the docker-pkg definitions.

The list of tests is there https://gerrit.wikimedia.org/g/integration/config/+/refs/heads/master/tests/test_dockerfiles.py and the tests verify a few basic things pertaining to how the images are defining, without building images nor running them. That verify a few things:

• directory has a Dockerfile.template
• directory has changelog and that it parses without warnings (most contributors do not use dch)
• directory has control and that the list of images in Depends match the FROM statements. They MUST be in sync.
• another short test to ensure some image tags match the software version (eg Quibble 1.15 is in an image releng/quibble:1.15)

Those tests have been a life changer early on since it is so easy to screw up a definition by mistake and wondering why an image does not build which would happen if you forgot to bump the changelog (why? there is git log) or update the control file after you have changed the FROM statement.

I think this task would be fulfilled by copy the test file into the repo, maybe make some adjustments (integration/config does not have a tree of directories) and adding the necessary files to run the test. Or even better, those bits of python I wrote could be upstreamed to the docker-pkg command as a lint command that would be available out of the box (given we have at least 3 repos holding images definitions).

In T283855#11592466, @bking wrote:

CI would have helped, although I would suggest we look at things more broadly. docker-pkg is WMF-specific code that doesn't seem to have been updated in some time.

If you are referring to the date at the bottom of that page (©2017, Giuseppe Lavagetto), that is obviously a hardcoded date. It comes from Sphynx configuration and is hardcoded:

https://gerrit.wikimedia.org/g/operations/docker-images/docker-pkg/+/refs/heads/master/doc/conf.py#69

# General information about the project.
project = "docker-pkg"
copyright = "2017, Giuseppe Lavagetto"
author = "Giuseppe Lavagetto"

docker-pkg is maintained as seen in the git log https://gerrit.wikimedia.org/g/operations/docker-images/docker-pkg/ , I event sent a couple patches last summer which are in my local copy ( shameless advertisement to get reviews/merge https://gerrit.wikimedia.org/r/q/project:operations/docker-images/docker-pkg+is:open ).

@bking wrote:

I was unable to get it to work in a virtualenv on my MacOS laptop.

Yes MacOS support tends to be challenging unless devs also have a Mac and for docker-pkg I believe most use it on their Linux machines or on a build host. It might be an issue with the python library that is used to interfaces to Docker, see for example T380424.

But you can always file a task against docker-pkg and add ARM support (that is the magic tag to refer to MacOS arm issues which might or might not be relevant.). Assisting developers/sre having troubles installing dev tools is literally my group description (Developer Experience) I am more than happy to assist / pair on it anytime!

Thanks @hashar , I must have had my brain dripping out of my ears when I wrote that last update. As you responded, docker-pkg is actively maintained so there is no reason to pull that thread. I would ask to add a link back to the docker-pkg repo in that same infobox, happy to make a PR for that.

And I'm all in favor of adding the CI as you mentioned, especially if most of the work is done already. We built similar checks into our OpenSearch plugin Deb package repo

As far as building the virtualenv, thanks for the advice! I will try the workaround from your linked ticket and get back to you.