Between the WMF CI images for node18 and node20, we switched from Debian 11 Bullseye to Debian 12 Bookworm. This means the long-deprecated /usr/bin/python alias no longer exists, instead python2 or python3 must be used explicitly.

The "gl" package still depends on generic "python" and so fails:

20:19:08 npm error path /src/node_modules/gl
…
20:19:08 npm error gyp info find Python using Python version 3.11.2 found at "/usr/bin/python3"
…
20:19:08 npm error /bin/sh: 1: python: not found

@Krinkle wrote at change 1314972

Yes, the irony is that earlier in the same process, it actually finds python3. Oh well.

https://github.com/stackgl/headless-gl/issues/270

They recommend installing python-is-python3 https://packages.debian.org/stable/python-is-python3 as a workaround.

We are using the latest version of gl, 8.1.6 released two years ago. The next release, 9.0.0 will probalby fix this, but that may take some time.

@Samwilson wrote at change 1314972

What will it take to get CI working again?

@Krinkle wrote at change 1314972

Installing python-is-python3 should do it.

@hashar wrote at change 1314972

Done via https://gerrit.wikimedia.org/r/c/integration/config/+/1323667

@hashar wrote at change 1314972

There is some further gyp error:

Package x11 was not found in the pkg-config search path.
Perhaps you should add the directory containing `x11.pc'
to the PKG_CONFIG_PATH environment variable
Package 'x11', required by 'virtual:world', not found
Package 'xi', required by 'virtual:world', not found
Package 'xext', required by 'virtual:world', not found
gyp: Call to 'pkg-config --libs-only-L --libs-only-other x11 xi xext' returned exit status 1 while in angle/src/angle.gyp. while loading dependencies of binding.gyp while trying to load binding.gyp

Package x11 was not found in the pkg-config search path.
Perhaps you should add the directory containing `x11.pc'

That is provided by libx11-dev.

Then there is pkg-config --libs-only-L --libs-only-other x11 xi xext which would indicate libxi-dev and libxext-dev are also required.

Why is that repo requiring the X Window System client? Could it be a slop in the npm dependencies? We never had libx11-dev added in the CI image.

If we actually need those packages, I think we will want to craft a more specific image that ships the required libraries.

As Antoine says, this clearly never worked via the node-test24 Jenkins job the repo runs today, because it, nor any if its predecessors (node-test20, node-test16, etc) never installed X11. We could create a custom image for it, but let's first zoom out and try to understand when this last worked and how.

I looked at changes to package.json in the "3d2png" repo and around 2019 it looked like this (https://gerrit.wikimedia.org/r/c/3d2png/+/492373):

3d2png$ git log -p -- package.json

"deploy": {
  "node": "6.9.1",
  "target": "debian:stretch",
  "dependencies": {
    "debian": [
      "pkg-config",
      "libcairo2-dev",
      "libjpeg-dev",
      "libxi-dev",
      "libgif-dev",
      "libglu1-mesa-dev",
      "libglew-dev",
      "libpng-dev",
      "xvfb",
      "xauth",
      "libgl1-mesa-dri",
      "libgl1-mesa-glx"
    ]
  }
},

I don't know what consumes this non-standard deploy key in package.json, but it cleary isn't the standard node-test Jenkins jobs that we maintain in WMF CI today.

I then looked at our integration/config repo in Gerrit, in the history of the /zuul/layout.yaml file for any changes that mention "3D" to see if we can spot what CI jobs it used in the past.

https://gerrit.wikimedia.org/r/c/integration/config/+/417282

Around 2018, a regular routine maintenance patch by @hashar shows that it used to have a custom Jenkins job and custom Docker image that, I guess, either had these additional packages pre-installed or somehow applied the above "deploy" instructions:

- name: 3d2png
  test:
    - 3d2png-npm-node-6-docker
  gate-and-submit:
    - 3d2png-npm-node-6-docker

- name: 3d2png/deploy
  experimental:
    - 3d2png-deploy-npm-node-6-docker
    - 3d2png-deploy-npm-node-6-jessie

Then around 2019, we see a @James_F helping the repo migrate away from the legacy "deploy" repo system, to Wikimedia Blubber, as part of T211784: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects:

https://gerrit.wikimedia.org/r/c/integration/config/+/516707

- name: 3d2png
  template:
    - name: service-pipeline-test
    - name: service-pipeline-test-and-publish

That task links to T225678: Migrate 3d2png to k8s, which has a patch that remains unmerged to this day:

https://gerrit.wikimedia.org/r/c/3d2png/+/516709

This is the approach that makes sense to pick up here to make CI pass again.

But rather than create something novel, we should make CI work similarly to what we do in production whenever possible. How does this get built and deployed in production today? We did add support for 3D to Thumbor. And, we did reuse the same 3d2png application to do it. So how does Thumbor install 3d2png? And how does it test it?

https://gerrit.wikimedia.org/g/operations/software/thumbor-plugins/+/9a208172003393d37234d0c7c6c0a735004be21b/Makefile#68

3d2png: needs-docker
	git clone https://github.com/wikimedia/3d2png.git
	cd 3d2png; npm install

https://gerrit.wikimedia.org/g/operations/software/thumbor-plugins/+/9a208172003393d37234d0c7c6c0a735004be21b/.pipeline/blubber.yaml#116

    # Packages required for 3d2png
    - npm
    - pkg-config
    - libx11-dev
    - libxi-dev
    - libglx-dev
    - libgl1-mesa-dev
    - libxi-dev
copies: [local]
builder:
  command: [make, 3d2png]
  requirements:
    - Makefile

So we have a working Blubber setup in the Thumbor repo for production. I suggest we dust off https://gerrit.wikimedia.org/r/c/3d2png/+/516709 and write a dev/CI-only Blubber file (no need to build an image after each commit for production, because it isn't used as standalone service), and use the same base image and steps as Thumbor.

Change #516709 had a related patch set uploaded (by TheDJ; author: Jforrester):

[3d2png@master] build: Create pipeline configuration for CI tests

https://gerrit.wikimedia.org/r/516709

I've cleaned up the blubber config and added a dev profile so you can run 3d2png.

Change #1323849 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] Add Pipeline jobs for 3d2png

https://gerrit.wikimedia.org/r/1323849

Change #1323849 merged by jenkins-bot:

[integration/config@master] Add Pipeline jobs for 3d2png

https://gerrit.wikimedia.org/r/1323849

I have added the PipelineLib configuration to CI ( https://gerrit.wikimedia.org/r/c/integration/config/+/1323849 ) and did a recheck of the refreshed Blubber config https://gerrit.wikimedia.org/r/c/3d2png/+/516709 which missed a .pipeline/config.yaml file. It now has a single test pipeline. I guess the dev variant should be build as well, and maybe published to the Docker registry? I have commented on the change, we can follow up there.

Yeah i added a dev variant because i saw thumbor had one. But i honestly don’t really know what we do with blubber dev generally ;)
It was helpful to test if something useful came out of the image however.

Change #516709 merged by jenkins-bot:

[3d2png@master] build: Create pipeline configuration for CI tests

https://gerrit.wikimedia.org/r/516709