[tbs.beta] Create a toolforge build service beta release
Open, In Progress, MediumPublic
Actions

Assigned To

Authored By

	Legoktm
	Nov 5 2020, 9:54 PM

Description

For proof of concept and to start getting user feedback, setup a minimal build service deployment on tools.

User stories:
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Ongoing_Efforts/Toolforge_Build_Service/Overview#Beta_user_stories

Related Objects
Search...

Status	Assigned	Task
Stalled	LucasWerkmeister	T320140 Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes
Stalled	matmarex	T319707 Migrate dtcheck from Toolforge GridEngine to Toolforge Kubernetes
Open	None	T320062 Migrate steve-adder from Toolforge GridEngine to Toolforge Kubernetes
Open	None	T320011 Migrate rfa-voting-history from Toolforge GridEngine to Toolforge Kubernetes
Open	dcaro	T194332 [Epic] Make Toolforge a proper platform as a service with push-to-deploy and build packs
In Progress	dcaro	T267374 [tbs.beta] Create a toolforge build service beta release
Resolved	dcaro	T293644 [tbs] Create a subcommand for the toolforge cli to trigger a build
Resolved	dcaro	T293667 Add documentation on the buildpack POC
Open	None	T265685 [tbs] Set up CI for cloud/toolforge/buildpacks repository
Declined	dduvall	T266081 Evaluate how we might run buildpack's `pack` in CI
Open	None	T283724 Tracking task to support running arbitrary Docker images
Open	None	T265691 [tbs.maintaindbusers] Update maintain-dbusers to create a k8s secret with replica.my.cnf to be used by buildpack images
Duplicate	None	T291409 [NFS] Update maintain_dbusers so it can run on a VM
Resolved	dcaro	T265781 [tbs] Upgrade Toolforge buildpacks to api 0.8 (latest)
Resolved	Raymond_Ndibe	T316231 [tbs.cli.build-logs] Show a warning to the user if it’s not admin saying that they will not be able to read the logs
Resolved	dcaro	T316228 [tbs.cli] Define the process to build and deploy the CLI on toolforge bastions
Resolved	Slst2020	T316226 [tbs.cli]: toolforge build-show shows unknown for steps that are actually ok
Resolved	Slst2020	T316222 [tbs.cli]: Create reasonable defaults for the build command (so the users have to add only the minimum needed info)
Duplicate	None	T267062 Allow Toolforge users to manually start/restart/stop buildpack based web servers
Duplicate	None	T297089 [tbs.docs] Create user documentation (to be expanded)
Open	None	T296947 [toolforge.buildservice] Add CI checks
Duplicate	None	T297088 [tbs.docs] Create admin documentation (to be expanded)
Resolved	dcaro	T265780 [tbs.docs]Figure out naming conventions for Toolforge buildpack related stuff
Resolved	Raymond_Ndibe	T316332 [tbs.buildpack-admission-controller] Ensure we use the deploy.sh standard
Resolved	Raymond_Ndibe	T316335 [tbs.registry-admission-webhook] Ensure we use the deploy.sh standard
Resolved	Raymond_Ndibe	T316530 [tbs.harbor] Upgrade harbor to 2.5.*
Resolved	dcaro	T316541 [tbs.harbor] Rebuild toolsbeta-harborweb-2 with debian 11(bullseye)
Resolved	fnegri	T316658 [tbs.devenv] Francesco to setup the development environment
Resolved	Raymond_Ndibe	T316657 [tbs.devenv] Raymond to setup the development environment
Resolved	Slst2020	T316656 [tbs.devenv] Slavina to setup the development environment
Resolved	Slst2020	T318656 [tbs][cli] Error when there's no 'steps' in the status
Resolved	KHernandez-WMF	T320424 [tbs] Add project documentation to wikitech wiki
Resolved	Raymond_Ndibe	T320522 [buildpack-admission-controller] Rename the toolforge deployment envirnoment to tools
Resolved	komla	T321162 [buildservice] allow running deploy.sh without passing a parameter
Resolved	Raymond_Ndibe	T321188 [tbs] fix/improve the updating of the buildpack/tekton images in the local repo
Resolved	Slst2020	T322190 [tbs][cli] When a pipelinerun is created, it has no 'status' so we should not break on it
Resolved	dcaro	T322412 [tbs][buildpack-admission-controller] fix deploy on toolsbeta/tools
Resolved	dcaro	T322413 [tbs][buildpack-admission-controller] use git hash and timestamp for docker image tag
Resolved	dcaro	T324314 [harbor] puppet onlyif not working
Resolved	Raymond_Ndibe	T324785 get buildpack-admission-controller to work in development
Resolved	dcaro	T324704 Develop user stories for Toolforge Build Service.
Open	None	T324814 tbs: user-story 1 - I'm able to use the service using its documentation
Stalled	None	T324815 tbs: user-story 1 - Find a place for the user documentation
Open	None	T324816 tbs: user-story 1 - Create a tutorial (step-by-step) on how to deploy a tool using toolforge buildservice
Open	None	T324817 tbs: user-story 1 - Create an 'if something breaks' section with some of the encountered issues and guidelines on how to debug
Open	KHernandez-WMF	T324818 tbs: user-story 1 - [User-facing page] Create a "contact" section on how to ask for help
Open	None	T324819 tbs: user-story 1 - Create a "feedback" section on how to give feedback
Resolved	Slst2020	T324822 tbs: user-story 1 - Create a "reporting bugs" section on how to open a bug task
Open	None	T324827 tbs: user-story 3 - I want to cancel the build using the toolforge build cancel feature
Resolved	Raymond_Ndibe	T324828 tbs: user-story 3 - cancel cli should give the right error when the service is down
Resolved	Raymond_Ndibe	T331409 [tbs.cli] The cancel command should return a message saying that the build is already finished when trying to cancel a finished build
Resolved	Raymond_Ndibe	T331410 [tbs.cli] The cancel command should show a useful message when the connection to tekton works but returns http error (requests.exceptions.HTTPError)
Resolved	None	T332294 [tbs][cli] build cancel --help lists the -v option twice
Resolved	dcaro	T324833 tbs: user-story 5 - I want to check the status of a given build using the toolforge build show feature.
Open	None	T324830 tbs: user-story 4 - I want to check the status of the last build using the toolforge build show feature
Resolved	Raymond_Ndibe	T324831 tbs: user-story 4 - show cli should give the right error when the service is down
Stalled	dcaro	T324823 tbs: user-story 2 - Feature: I can start a new build from a git url
Resolved	Raymond_Ndibe	T323555 [tbs][cli] Implement the global --verbose flag as an env var
Resolved	Slst2020	T322286 [tbs][cli] Create proper subcommands and add 'start' subcommand
Resolved	Slst2020	T323681 [tbs.cli] Multiple bugs when running external commands
Resolved	Slst2020	T323709 [tbs.cli] Do not raise an error when an external subcommand fails
Resolved	Slst2020	T323711 [tbs.cli] Options of external subcommands are not forwarded
Resolved	Raymond_Ndibe	T323700 [toolforge-cli][build] show fails if the pipelinerun has no 'status'
Resolved	Slst2020	T324824 tbs: user-story 2 - start cli should autogenerate the image name when not passed
Resolved	Slst2020	T324825 tbs: user-story 2 - start cli should give the right error when the git url is not passed
Resolved	Slst2020	T324826 tbs: user-story 2 - start cli should give the right error when the service is down
Resolved	dcaro	T331862 [tbs.cli] build start default builder has no tag, and does not pass the admission webhook check
Resolved	dcaro	T331873 [tbs.toolsbeta] Push the heroku builder image to the harbor.toolsbeta repository
In Progress	dcaro	T331876 [tbs.cli] change the default builder to be just the heroku builder from the harbor repository
Resolved	dcaro	T331893 [tsb.cli] don't force a user when running upstream buildpacks
Resolved	dcaro	T331897 [webservce] buildservice type is shown as deprecated
Resolved	dcaro	T331723 [tbs.buildpack-admission-controller] Restrict the builders to heroku only
Resolved	Raymond_Ndibe	T324832 tbs: user-story 4 - show cli should use the latest build if no build id is passed
Open	None	T324834 tbs: user-story 6 - I want to start a webservice with the image I built earlier.
Resolved	Raymond_Ndibe	T293645 [tbs.webservice] Modify the webservice cli to allow the deployment of custom images from harbor
Stalled	Raymond_Ndibe	T316327 [tbs] Deploy on tools
Resolved	dcaro	T316325 [tbs.buildservice.toolsbeta] Deploy on toolsbeta
Resolved	Raymond_Ndibe	T316326 [tbs.buildservice] Ensure we use the deploy.sh standard
In Progress	Raymond_Ndibe	T332347 Duplicate the maintain-harbor deployment for "tools"
Resolved	fnegri	T316323 [tbs.harbor.tools] Replicate toolsbeta harbor deployment
Resolved	fnegri	T332208 [tbs] Harbor broken in toolsbeta because of missing db password
Open	None	T265681 [tbs.tools] Deploy a tool using a buildpack-based image on Toolforge
Open	None	T316330 [tbs.toolsbeta] Deploy a tool using a buildpack-based image on Toolsbeta
Resolved	Raymond_Ndibe	T316328 [tbs.registry-admission.toolsbeta] Allow images from harbor on toolsbeta
Resolved	dcaro	T267616 [tbs.harbor] Puppetize the toolsbeta installation
Resolved	Andrew	T267618 Request increased quota for toolsbeta Cloud VPS project
Resolved	Slst2020	T316232 [tbs.harbor.toolsbeta] Create a postgres DB with trove and plug the current instance there
Resolved	dcaro	T316233 [tbs.harbor.toolsbeta] Create a cinder volume and put there the storage for the container images
Resolved	Raymond_Ndibe	T293649 [tbs.harbor] Pre-create namespaces
Invalid	None	T265557 [tbs.maintainkubeusers] Create security policies for running buildpack images
Resolved	Raymond_Ndibe	T316339 [tbs.registry-admission.tools] Allow images from harbor on tools
Resolved	Raymond_Ndibe	T323689 [webservice] Allow configuring the image registry used to pull
Resolved	dcaro	T324566 [webservice] Figure out how to configure depending on the environment it's running
Open	None	T325141 tbs: user-story 6 - Build and deploy the modified webservice to toolsbeta
Open	None	T325142 tbs: user-story 6 - Build and deploy the modified webservice to tools
Resolved	dcaro	T324835 tbs: user-story 7 - I want to list all the tool builds.
Resolved	Raymond_Ndibe	T324836 tbs: user-story 7 - list cli should give the right error when the service is down
Resolved	Raymond_Ndibe	T324837 tbs: user-story 8 - I want to delete builds.
Resolved	Raymond_Ndibe	T324838 tbs: user-story 8 - delete cli should give the right error when the service is down
Open	None	T325172 tbs: user-story 11: I want to know how to debug the service
Open	None	T325174 tbs: user-story 11: Add section to admin docs on how to debug the service, how to pin-point the failing component and how to get the logs for each of them.
Open	None	T325166 tbs: user-story 10: I want to know how to manage the service
Open	komla	T325167 tbs: user-story 10: Create admin wiki page for the toolforge build service
Open	None	T325175 tbs: user-story 11: Add a runbook for each of the service alerts.
Open	None	T325160 tbs: user-story 9: I want to know when the service is down
Open	None	T325162 tbs: user-story 9: Create an alert on metricsinfra for tekton being down on toolsbeta
Open	None	T325163 tbs: user-story 9: Create an alert on metricsinfra for tekton being down on tools
Open	None	T325164 tbs: user-story 9: Create an alert on metricsinfra for harbor being down on tools
Open	None	T325165 tbs: user-story 9: Create an alert on metricsinfra for harbor being down on toolsbeta
Resolved	dcaro	T330096 tbs: user-story 9: Gather statistics for harbor uptime in toolsbeta
Resolved	dcaro	T330757 [tbs.harbor] Harbor core being down is not noticed by simple http request
Open	None	T325180 tbs: user-story 12: I want to easily run the tests for each of the codebases
Resolved	Slst2020	T325183 tbs: user-story 12: Add pre-commit to the buildservice repo
Resolved	Raymond_Ndibe	T325184 tbs: user-story 12: Add pre-commit to the buildpack-admission-controller repo
Resolved	Slst2020	T320234 tbs: user-story 12: use pre-commit for the toolforge-cli repo
Open	None	T325187 tbs: user-story 13: I want to avoid getting bugs inadvertently into the codebases
In Progress	None	T325189 tbs: user-story 13: Add ci to the buildservice repo
In Progress	None	T325192 tbs: user-story 13: Add ci to the buildpack-admission-controller repo
Resolved	Slst2020	T325194 tbs: user-story 13: Add ci to the toolforge-cli repo
Open	None	T325207 tbs: user-story 14: I want have some certainty that the service is secure
Open	None	T325208 tbs: user-story 14: Run a set of security checks on the full service
Resolved	dcaro	T330102 Decision request - What buildpacks to allow and include for toolforge build service beta
Open	dcaro	T331323 [tbs] Everybody has deployed every component once in toolsbeta
In Progress	dcaro	T331324 [tbs] David has deployed every component in toolsbeta
Resolved	dcaro	T331328 [tbs] David has deployed toolforge-cli in toolsbeta
Open	dcaro	T331329 [tbs] David has deployed bulidservice in toolsbeta
Resolved	dcaro	T331330 [tbs] David has deployed buildpack-admission-controller in toolsbeta
Open	dcaro	T331331 [tbs] David has deployed the webservice cli in toolsbeta
In Progress	Raymond_Ndibe	T331325 [tbs] Raymond has deployed every component in toolsbeta
Resolved	Raymond_Ndibe	T331332 [tbs] Raymond has deployed toolforge-cli in toolsbeta
Resolved	Raymond_Ndibe	T331333 [tbs] Raymond has deployed buildservice in toolsbeta
In Progress	Raymond_Ndibe	T331334 [tbs] Raymond has deployed buildpack-admission-controller in toolsbeta
Open	Raymond_Ndibe	T331335 [tbs] Raymond has deployed the webservice cli in toolsbeta
Open	fnegri	T331326 [tbs] Francesco has deployed every component in toolsbeta
Open	fnegri	T331336 [tbs] Francesco has deployed the webservice cli in toolsbeta
Open	fnegri	T331337 [tbs] Francesco has deployed toolforge-cli in toolsbeta
Open	fnegri	T331338 [tbs] Francesco has deployed buildservice in toolsbeta
Open	fnegri	T331339 [tbs] Francesco has deployed buildpack-admission-controller in toolsbeta
In Progress	Slst2020	T331327 [tbs] Slavina has deployed every component in toolsbeta
Resolved	Slst2020	T331340 [tbs] Slavina has deployed toolforge-cli in toolsbeta
Resolved	Slst2020	T331341 [tbs] Slavina has deployed buildservice in toolsbeta
Resolved	Slst2020	T331342 [tbs] Slavina has deployed buildpack-admission-controller in toolsbeta
In Progress	Slst2020	T331343 [tbs] Slavina has deployed the webservice cli in toolsbeta