When developing software for Toolforge, we've detected a few problems related to how we usually set up the local kubernetes environment. Example problem T325755: Bug: jobs-framework-api job run fails silently on local development environment. The goal of this ticket is to capture/describe the problem and potential solutions.
There are scattered instructions on how to setup the local development environment in several different READMEs (example1, example2), and a collection of opinionated and questionable setup.sh scripts (example). The development environment for a local kubernetes can also be different if using minikube, kind or if using Debian, Ubuntu or Mac. To overcome this situation I started experimenting with an ansible setup playbook here: https://gitlab.wikimedia.org/aborrero/cloud-toolforge-lima-kilo
The lima-kilo setup could include pieces like:
- setting up the core RBAC config of Toolforge kubernetes (PSP, Roles, etc)
- maintaining the [fake] context for Toolforge, like /data/project directories, fake files like /etc/wmcs-project that the different K8S components expect to exist
- setting up the different admission controllers so the local kubernetes behaves like the actual Toolforge kubernetes
- creating a few fake tool accounts (users) so they can be used for development/testing purposes
We could have playbooks to support the combo {debian,ubuntu,mac}-{kind,minikube}-{install,uninstall}.
I've started with the debian-kind combo because is the one @Raymond_Ndibe and I use. The experiments so far have show promising results.