Currently, the Toolforge Build Service will happily build an image even when the head revision in git has broken unit tests. For automatic push-to-deploy, it would be important to run automated tests as part of the deployment process.

To reproduce, I've added an intentionally broken test case to my tool, submitted it to the code repository, and then ran "toolforge build start https://github.com/brawer/wikidata-qrank". Expected: Toolforge Build Service should have run "go test ./..." to execute unit tests; this would have found the intentionally broken test case. Observerd: Toolforge Build Service happily built an image with broken tests. I’ll roll back the intentional breakage so I can continue developing my tool, but you can easily create another code repo that has broken unit tests.

Discussion on the wikimedia-cloud IRC channel:

• lucaswerkmeister: I agree it should be part of push-to-deploy, but I’d expect it to happen outside the build – that GitLab runs tests first and then builds and deploys a new image if they succeed. [A]nd if “run *specific, well-known* test commands (e.g. npm test)” was supported, you probably would’ve noticed it already. I agree it should be part of push-to-deploy, but I’d expect it to happen outside the build – that GitLab runs tests first and then builds and deploys a new image if they succeed. [P]robably worth a phabricator task in any case

• taavi: i would agree that's not really in-scope for the build service