Page MenuHomePhabricator

Add a verify step to docker-pkg
Closed, ResolvedPublic

Description

As we want to force images created by docker-pkg to only contain numeric UIDs, we should have a way to verify that they are correct.
Some images do use users created by debian packages, so the UID of those might change without us noticing.

Event Timeline

I'm currently implementing this step, and for now my implementation idea goes as follows:

  • By default, we'll verify that a test executable is present in the image directory, and run it with the image as the only cli argument.
  • It will be possible to override the command to run, for example say you've created a test suite written with pytest, and you have a test_image.py tests file in the directory where the image resides. You will be able to declare
verify_command_template = pytest {path}/test_image.py {image}

and docker-pkg will then run something like pytest images/httpd-fcgi/test-image.py docker-registry.wikimedia.org/httpd-fcgi:1.0.1

This should allow users as much flexibility as they want to set up their testing framework as they see fit.

While I like the flexibility it brings the downside of potentially having to duplicate test code for multiple images.
We will probably have some repeated tasks in the tests, right? I'm especially thinking of the "verify-user" test that would be nice if run all the time without having to implement it in the tests.

While I like the flexibility it brings the downside of potentially having to duplicate test code for multiple images.
We will probably have some repeated tasks in the tests, right? I'm especially thinking of the "verify-user" test that would be nice if run all the time without having to implement it in the tests.

There are various ways to avoid duplication in basically any testing framework I can think of, including shell scripts :)

The point is, we might want to verify numeric IDs ourselves, but people using docker-pkg for running CI might be interested in a completely different set of tests.

So to make an example, right now production-images has some tests written in bash (yes, I know!).

We could think of adding to the root of that repository a relatively simple common-tests.sh file that just runs some basic consistency check on the run as user, or something verifying that the base image is the latest on the registry.

Yeah, sure. I just wanted to make the point that we should (be able to) provide some basic boilerplate for testing and probably have some generic tests implemented that can easily be used (or will automatically be used) on images.

So to make an example, right now production-images has some tests written in bash (yes, I know!).

We could think of adding to the root of that repository a relatively simple common-tests.sh file that just runs some basic consistency check on the run as user, or something verifying that the base image is the latest on the registry.

Yes, that :-)

Joe triaged this task as Medium priority.

Change 661138 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/docker-images/docker-pkg@master] Allow running tests on an image once it's built

https://gerrit.wikimedia.org/r/661138

I am not sure what kind of problem it is aimed at solving or what the proposed command will solve? For the CI images in integration/config.git we have two set of tests:

A) the first ones are pytest based they crawl the tree of docker-pkg definitions and catch a bunch of basic issues. tests/test_dockerfiles.py checks:

  • each directory has Dockerfile.template , changelog and control. Else they will not be found by docker-pkg.
  • changelog is valid
  • changelog package name match the name of the directory holding it
  • dependencies in control point to images that actually exist

And I wanted to add a check to ensure version increases are monotonic

Most probably, those tests can be upstream to docker-pkg in a new lint command to ensure the basic structure of the file tree is more or less valid.

B) for some images, we have plain shell scripts (dockerfiles/*/example-run.sh). We run them manually and they simply execute the image with some well crafted parameters to more or less ensure the image work.

I would love a way to ensure that all the CI images run as nobody when they have an ENTRYPOINT defined.

Not sure if that help though :/

B) for some images, we have plain shell scripts (dockerfiles/*/example-run.sh). We run them manually and they simply execute the image with some well crafted parameters to more or less ensure the image work.

I would love a way to ensure that all the CI images run as nobody when they have an ENTRYPOINT defined.

Not sure if that help though :/

That's exactly the use-case I had in mind: provide a way to force docker-pkg to run a test script with the current image tag as argument, and check that the script ends successfully.

Change 661138 merged by jenkins-bot:
[operations/docker-images/docker-pkg@master] Allow running tests on an image once it's built

https://gerrit.wikimedia.org/r/661138