I tackled that a bit yesterday and went with a very rough patch to build the dependency tree of images (based on FROM). https://gerrit.wikimedia.org/r/381761

{
   "docker-registry.wikimedia.org/wikimedia-jessie" : {
      "wmfreleng/operations-puppet" : {},
      "wmfreleng/zuul-cloner" : {},
      "wmfreleng/ci-jessie" : {
         "wmfreleng/lintr" : {},
         "wmfreleng/tox" : {},
         "wmfreleng/php" : {
            "wmfreleng/php-mediawiki" : {
               "wmfreleng/mediawiki-phpcs" : {},
               "wmfreleng/mediawiki-phan" : {}
            },
            "wmfreleng/composer" : {
               "wmfreleng/mediawiki-phan" : {},
               "wmfreleng/mediawiki-phpcs" : {}
            }
         }
      }
   }
}

So if you made a change to the php image, you gotta pull and/or rebuild the parents. And then foreach of those parents rebuild their descendants. Then you can process with the php images descendants. Note how mediawiki-phan descend from both php-mediawiki and composer.

I have the feeling we should just blindly rebuild all of them based on the order of their depth. With non wmfreleng images being pull from the upstream (eg docker-registry.wikimedia.org/wikimedia-jessie).

docker build does not support mounting a volume which make it impossible to use local caches for eg apt packages. We would need the RedHat docker flavor to support that or cherry pick whatever patch they have.

I am tempted to build the images based on a flat good old chroot + debootstrap + shell scripts. Which diskimage-builder (for Nodepool images) does.

That looks like a good start to me. I think I want to collapse the dependency tree a bit (merge php-mediawiki, composer into php) but that's a bit tangential.

Are you proposing we use a fresh/clean build environment (aka not personal laptops)? I was going to file a separate task for that too :)

Change 381761 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: lame dependency tree

https://gerrit.wikimedia.org/r/381761

That looks like a good start to me. I think I want to collapse the dependency tree a bit (merge php-mediawiki, composer into php) but that's a bit tangential.

We can probably have as many layers as needed. IIrc the php-mediawiki one is meant to add all the packages mediawiki requires but other PHP projects don't. Then I will might resurect Quibble , which potentially is intended to be an all in one test runner for MediaWiki and co.

Are you proposing we use a fresh/clean build environment (aka not personal laptops)? I was going to file a separate task for that too :)

We all use our personal laptops. But once we get the dependency madness sorted out, I guess it will be easy to automatize on post merge. From there I guess the CI job will rebuild the images without any cache. Once they all have been rebuild, it would tag and push them. (eg image:build1234). Had we got CI jobs pointing to image:stable we would then manually promote a build image by changing the stable tag. (all that are random thoughts past midnight sorry).

From https://gerrit.wikimedia.org/r/381761 there is some dependency tree which I think cover part of this change. I am tempted to just blindly rebuild all the images stages by stages (with foreign images requiring in stage 1 requiring a docker pull).

{
  "1": [
    "docker-registry.wikimedia.org/wikimedia-jessie",
    "docker-registry.wikimedia.org/wikimedia-stretch"
  ],
  "2": [
    "wmfreleng/ci-jessie",
    "wmfreleng/ci-stretch",
    "wmfreleng/operations-puppet",
    "wmfreleng/zuul-cloner"
  ],
  "3": [
    "wmfreleng/lintr",
    "wmfreleng/npm",
    "wmfreleng/php",
    "wmfreleng/tox"
  ],
  "4": [
    "wmfreleng/composer",
    "wmfreleng/php-mediawiki"
  ],
  "5": [
    "wmfreleng/composer-package",
    "wmfreleng/mediawiki-phan",
    "wmfreleng/mediawiki-phpcs"
  ]
}

{
  "docker-registry.wikimedia.org/wikimedia-stretch": {
    "wmfreleng/ci-stretch": {
      "wmfreleng/php": {
        "wmfreleng/php-mediawiki": {
          "wmfreleng/mediawiki-phpcs": {},
          "wmfreleng/composer-package": {},
          "wmfreleng/mediawiki-phan": {}
        },
        "wmfreleng/composer": {
          "wmfreleng/mediawiki-phpcs": {},
          "wmfreleng/composer-package": {},
          "wmfreleng/mediawiki-phan": {}
        }
      }
    }
  },
  "docker-registry.wikimedia.org/wikimedia-jessie": {
    "wmfreleng/zuul-cloner": {},
    "wmfreleng/operations-puppet": {},
    "wmfreleng/ci-jessie": {
      "wmfreleng/lintr": {},
      "wmfreleng/npm": {},
      "wmfreleng/tox": {}
    }
  }
}

{
  "1": [
    "docker-registry.wikimedia.org/wikimedia-jessie",
    "docker-registry.wikimedia.org/wikimedia-stretch"
  ],
  "2": [
    "wmfreleng/ci-jessie",
    "wmfreleng/ci-stretch",
    "wmfreleng/operations-puppet",
    "wmfreleng/zuul-cloner"
  ],
  "3": [
    "wmfreleng/lintr",
    "wmfreleng/npm",
    "wmfreleng/php",
    "wmfreleng/tox"
  ],
  "4": [
    "wmfreleng/composer",
    "wmfreleng/php-mediawiki"
  ],
  "5": [
    "wmfreleng/composer-package",
    "wmfreleng/mediawiki-phan",
    "wmfreleng/mediawiki-phpcs"
  ]
}

Change 381761 merged by jenkins-bot:
[integration/config@master] docker: Recursively build images in order

https://gerrit.wikimedia.org/r/381761