We can reuse the sample repositories and tools to create a set of fullstack tests to be monitored in production, replacing and extending the toolschecker tests.
This should report to prometheus, where the alerts for the failure of the tests would get the data from (as opposed to alerting directly).
We could have sample tools running a webservice, that exposes a prometheus endpoint with the stats for the tool, one of them can do the full stack of tests:
- Webservice itself
- Build (if a buildservice image)
- One-off job
- Scheduled job
- Continuous job
- Envvars
- Access to NFS
- Access to replicas
- Access to toolsdb
- Access to redis
- Access to elasticsearch
While the others can do a simplified version:
- Webservice itself
- Build
- One-off Job
- Access replicas
A side-benefit of this is that it will serve as example also for users on how to run those things.
List of current sample tools:
- https://gitlab.wikimedia.org/toolforge-repos/sample-dotnet-buildpack-app
- https://gitlab.wikimedia.org/toolforge-repos/sample-php-buildpack-app
- https://gitlab.wikimedia.org/toolforge-repos/sample-static-buildpack-app
- https://gitlab.wikimedia.org/toolforge-repos/sample-rust-buildpack-app
- https://gitlab.wikimedia.org/toolforge-repos/sample-ruby-rails-buildpack-app
- https://gitlab.wikimedia.org/toolforge-repos/sample-python-buildpack-app
- https://gitlab.wikimedia.org/toolforge-repos/sample-python-buildpack-asgi
- https://gitlab.wikimedia.org/toolforge-repos/nodejs-flask-buildpack-sample
Plus we might want to add a sample tool + simple tests for each of the latest versions of the pre-built images.
- debian - bookworm
- jdk - jdk17
- mariadb
- mono - mono6.8
- node - node18
- perl - perl5.36
- php - php8.2
- python - python3.11
- ruby - ruby3.1
- tcl - tcl8.6