Specifically, it would be very nice if we could influence the MW docker-compose configuration when testing the Wikilambda extension. This way, we could introduce other services (viz. function-orchestrator and function-evaluator) for purposes of integration testing.

We do have a way to run integration tests via helm (deploying to kubernetes) in CI. I think once the mediawiki chart is ready, we could probably try to get something like this working. We'd need to make sure the staging cluster/CI namespace has enough resources to support it.

I don't know of any discussions or plans by RelEng to support docker-compose in CI, but my knowledge is limited to the Pipelinelib in that regard. Migration to gitlab could also affect all of this. @brennen for any questions about gitlab.

All of the above refers to automated testing. I'm also trying to get a preview environment project underway for the UAT/manual testing part of this, but don't have any phabricator tasks to point you to at the moment.

On the legacy CI infrastructure (Zuul / Jenkins / Docker images) the workload is run on WMCS instances that have a Docker daemon. We can not use that daemon to run arbitrary containers from a docker-compose definition. So what Jeena said: the target is to use Helm charts / Kubernetes much like we will do for production deployments.

Hey @jeena, @hashar, and friends, I'm a new software engineer in test on the Abstract Wikipedia team, and I'm staring to work on end-to-end testing for WikiLamda (and it's related services).

I would be very interested in learning more about using helm/kubernetes to deploy a CI test environment (or whatever else you would suggest today, I realize this thread is quite old.) Are there anu docs or examples you could point me to? Is there anything that would be blocking me from trying it out?

To summarize a chat that @jeena and I had. Abstract Wikipedia will start working on writing end-to-end tests that work on a helm/k8s deployed environment with helm test. We'll plan on trying to run try it in Gitlab CI when its helm/k8s capabilities come online.