Change #1286896 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/deployment-charts@master] EXPERIMENT: rest-gateway: Dockerize system tests

https://gerrit.wikimedia.org/r/1286896

Change #1267959 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/deployment-charts@master] EXPERIMENT: run smokepy tests via helm test

https://gerrit.wikimedia.org/r/1267959

Change #1286896 merged by jenkins-bot:

[operations/deployment-charts@master] EXPERIMENT: rest-gateway: Dockerize system tests

https://gerrit.wikimedia.org/r/1286896

Change #1267959 merged by jenkins-bot:

[operations/deployment-charts@master] EXPERIMENT: run smokepy tests via helm test

https://gerrit.wikimedia.org/r/1267959

Change #1297666 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/deployment-charts@master] EXPERIMENT: rest-gateway: Dockerize system tests (again)

https://gerrit.wikimedia.org/r/1297666

Change #1297668 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/deployment-charts@master] EXPERIMENT: run smokepy tests via helm test (again)

https://gerrit.wikimedia.org/r/1297668

Change #1301423 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/deployment-charts@master] smokepy: Add interactive pod

https://gerrit.wikimedia.org/r/1301423

Change #1302104 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/deployment-charts@master] smokepy: use live mount for test files

https://gerrit.wikimedia.org/r/1302104

Summary of decisions that went into the proposed implementation:

Goals: Make smokepy end-to-end system tests callable via helm test.

Requirements:

1. Single environment for CI and local. Use the same pod definition for helm test (minikube), helmfile test (staging), and interactive debugging.
2. Reusable across charts. Adding smokepy to a new chart should be simple, code duplication should be minimal.
3. Support interactive debugging. Developers should be able to run individual tests, make changes to the test, and re-run them, without having to re-install the helm chart.

Move smokepy out of this repo, distribute as a docker image

Smokepy used to live inside this deployment-charts repository under python/smokepy/ and run directly on the host. To allow smokepy to be run from inside kubernetes, the test framework had to be containerized.

Smokepy was split into its own project at https://gitlab.wikimedia.org/repos/mediawiki/services/smokepy and is now consumed here as a published docker image.

Note: Chart-level render tests were decoupled from smokepy, so they can be run in CI without the somepy image. Code re-use between end-to-end tests and render tests was minimal anyway.

Run smokepy as helm test (chart) and helmfile test (service)

The previous local workflow ran the smokepy docker container on the developer's host with --network host, requiring kubectl port-forward in a parallel terminal and a bind-mount overlay of tests/smokepy/ over the chart's tests directory. This can't (or at least shouldn't) be done on a deployment host.

Instead, the tests should run in their own kubernetes pod. helm test is designed for this. It can be used directly for local testing on minikube. For end-to-end tests on the staging cluster we can then use helmfile test, which automatically uses the correct configuration for the staging environment.

Note that we don't rely on the --logs option of helm test for showing logs, since that sometimes runs into race conditions when the pod takes a while to come up (e.g. when a new image needs to be fetched).

Helm-test pod as a sextant vendor module

Using a sextant vendor module reduces the amount of copy&paste when using smokepy in a new chart. The test.smokepy module exposes test.smokepy.testPod with a small parameter surface (ctx, valuesSubset, envVarName, targetUrl) and bundles the chart's tests/smokepy/*.py files into a ConfigMap. Adopting smokepy in a new chart becomes: declare the module dependency, render the test pod template, place tests under tests/smokepy/.

Instructions for using the smokepy module can be found under modules/test/README.md.

Interactive pod derived from the helm-test pod

Interactive testing should also be possible using a kubernetes pod. This keeps the interactive environment as close as possible to the one used by helm test.

To create an interactive (sleep-forever) pod that can be used interactively, we render the chart with helm template and pipe through a yq filter that turns the automated test pod into an interactive pod for running smokepy tests. Everything else is inherited unchanged.

Live mount via minikube mount

Three approaches were considered for "developer edits visible in the pod":

Since the mount logic was making the makefile recipe complex and hard to read, the logic was movbed into a separate script, makefiles/smokepy-interactive.sh. This script takes care of creating the mount and launching the pod, and cleaning up when the script terminates.

To make the pod use the live mount, another yq filter is added that rewrites the smokepy-tests volume from a ConfigMap to a hostPath pointing at the in-VM mount target.

Note: minikube mount defaults to a random port, which is unfriendly to firewall rules. The script sets the port to 43043 and allows it to be overwritten by setting the MOKEPY_MOUNT_PORT variable.

Inline yq filter for the parameterised step

Two yq implementations exist in developer environments: mikefarah (Go) and kislyuk (Python, jq backend). They have incompatible env-var access syntaxes (strenv(X) vs env.X). The script keeps the parameter-less interactive filter as a .yq file (works in both), but the parameterised live-mount filter is built inline in the shell script with $HOST_MOUNT shell-interpolated into the expression — portable across both yqs.

Change #1298850 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/deployment-charts@master] smokepy tests: share helm-test pod via vendor module

https://gerrit.wikimedia.org/r/1298850