Page MenuHomePhabricator
Create Task
Maniphest T267616

[tbs.harbor] Puppetize the toolsbeta installation
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

This needs expanding, but some things we need are:

  • Install dependencies (docker-compose, docker)
  • Configure the app [done]
  • Start docker compose containers (check if they are up, if not then run "docker-compose up -d")

This task includes:

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedLucasWerkmeisterT320140 Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes
 ResolvedmatmarexT319707 Migrate dtcheck from Toolforge GridEngine to Toolforge Kubernetes
 ResolvedLegoktmT320062 Migrate steve-adder from Toolforge GridEngine to Toolforge Kubernetes
 ResolvedLegoktmT320011 Migrate rfa-voting-history from Toolforge GridEngine to Toolforge Kubernetes
 OpendcaroT194332 [Epic,builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs
 ResolveddcaroT267374 [tbs.beta] Create a toolforge build service beta release
 ResolveddcaroT316541 [tbs.harbor] Rebuild toolsbeta-harborweb-2 with debian 11(bullseye)
 ResolvedRaymond_NdibeT324827 tbs: user-story 3 - I want to cancel the build using the toolforge build cancel feature
 ResolvedRaymond_NdibeT324828 tbs: user-story 3 - cancel cli should give the right error when the service is down
 ResolvedRaymond_NdibeT331409 [tbs.cli] The cancel command should return a message saying that the build is already finished when trying to cancel a finished build
 ResolvedRaymond_NdibeT331410 [tbs.cli] The cancel command should show a useful message when the connection to tekton works but returns http error (requests.exceptions.HTTPError)
 ResolvedNoneT332294 [tbs][cli] build cancel --help lists the -v option twice
 ResolveddcaroT324833 tbs: user-story 5 - I want to check the status of a given build using the toolforge build show feature.
 ResolveddcaroT324830 tbs: user-story 4 - I want to check the status of the last build using the toolforge build show feature
 ResolvedRaymond_NdibeT324831 tbs: user-story 4 - show cli should give the right error when the service is down
 ResolveddcaroT324823 tbs: user-story 2 - Feature: I can start a new build from a git url
 ResolvedRaymond_NdibeT323555 [tbs][cli] Implement the global --verbose flag as an env var
 ResolvedSlst2020T322286 [tbs][cli] Create proper subcommands and add 'start' subcommand
 ResolvedSlst2020T323681 [tbs.cli] Multiple bugs when running external commands
 ResolvedSlst2020T323709 [tbs.cli] Do not raise an error when an external subcommand fails
 ResolvedSlst2020T323711 [tbs.cli] Options of external subcommands are not forwarded
 ResolvedRaymond_NdibeT323700 [toolforge-cli][build] show fails if the pipelinerun has no 'status'
 ResolvedSlst2020T324824 tbs: user-story 2 - start cli should autogenerate the image name when not passed
 ResolvedSlst2020T324825 tbs: user-story 2 - start cli should give the right error when the git url is not passed
 ResolvedSlst2020T324826 tbs: user-story 2 - start cli should give the right error when the service is down
 ResolveddcaroT331862 [tbs.cli] build start default builder has no tag, and does not pass the admission webhook check
 ResolveddcaroT331873 [tbs.toolsbeta] Push the heroku builder image to the harbor.toolsbeta repository
 ResolveddcaroT331876 [tbs.cli] change the default builder to be just the heroku builder from the harbor repository
 ResolveddcaroT331893 [tsb.cli] don't force a user when running upstream buildpacks
 ResolveddcaroT331897 [webservce] buildservice type is shown as deprecated
 ResolveddcaroT331723 [tbs.buildpack-admission-controller] Restrict the builders to heroku only
 ResolvedRaymond_NdibeT324832 tbs: user-story 4 - show cli should use the latest build if no build id is passed
 ResolveddcaroT324834 tbs: user-story 6 - I want to start a webservice with the image I built earlier.
 ResolvedRaymond_NdibeT293645 [tbs.webservice] Modify the webservice cli to allow the deployment of custom images from harbor
 ResolvedRaymond_NdibeT316327 [tbs] Deploy on tools
 ResolveddcaroT316325 [tbs.buildservice.toolsbeta] Deploy on toolsbeta
 ResolvedRaymond_NdibeT316326 [tbs.buildservice] Ensure we use the deploy.sh standard
 ResolvedRaymond_NdibeT332347 Duplicate the maintain-harbor deployment for "tools"
 ResolvedfnegriT316323 [tbs.harbor.tools] Replicate toolsbeta harbor deployment
 ResolvedNoneT265681 [tbs.tools] Deploy a tool using a buildpack-based image on Toolforge
 ResolveddcaroT316330 [tbs.toolsbeta] Deploy a tool using a buildpack-based image on Toolsbeta
 ResolveddcaroT267616 [tbs.harbor] Puppetize the toolsbeta installation
 ResolvedAndrewT267618 Request increased quota for toolsbeta Cloud VPS project
 ResolvedSlst2020T316232 [tbs.harbor.toolsbeta] Create a postgres DB with trove and plug the current instance there
 ResolveddcaroT316233 [tbs.harbor.toolsbeta] Create a cinder volume and put there the storage for the container images
 ResolvedRaymond_NdibeT293649 [tbs.harbor] Pre-create namespaces
 ResolvedRaymond_NdibeT316339 [tbs.registry-admission.tools] Allow images from harbor on tools
 ResolvedRaymond_NdibeT323689 [webservice] Allow configuring the image registry used to pull
 ResolveddcaroT324566 [webservice] Figure out how to configure depending on the environment it's running
 ResolveddcaroT325141 tbs: user-story 6 - Build and deploy the modified webservice to toolsbeta
 ResolveddcaroT325142 tbs: user-story 6 - Build and deploy the modified webservice to tools

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Jul 23 2021, 8:41 PM

Change 707572 merged by Bstorm:

[operations/puppet@production] toolforge harbor: puppetize experimental base server for harbor

https://gerrit.wikimedia.org/r/707572

Maintenance_bot removed a project: Patch-For-Review.Jul 23 2021, 9:10 PM
• Bstorm added a comment.Jul 24 2021, 12:12 AM

Ok, so I have a very nice https://goharbor.io server running in toolsbeta (using independent database auth for testing) https://harbor.toolsbeta.wmflabs.org
It has APIs for adding just about everything and has robot auth, etc. It grant plenty of flexibility for image management (even quotas).

It hooks up to LDAP, but that's not necessarily helpful here. I think to do this right, users need to not have direct push rights. That needs to be handled by another build service. However, with this, that service can easily run in k8s.

• Bstorm raised the priority of this task from Medium to High.Sep 15 2021, 4:25 PM

In order to deploy this to tools soon, I'm proposing replacing tools-docker-registry with harbor running in k8s. However, unless we deploy cinder for k8s, we'll be doing that with NFS storage.

• Bstorm added a comment.Sep 16 2021, 9:01 PM

I've deployed harbor in toolsbeta, however, I want to modify the deployment a bit before replacing docker-registry with it or putting it in tools.

gerritbot added a comment.Sep 21 2021, 6:05 PM

Change 722664 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: install clients for redis and postgres

https://gerrit.wikimedia.org/r/722664

gerritbot added a project: Patch-For-Review.Sep 21 2021, 6:05 PM
gerritbot added a comment.Sep 21 2021, 6:15 PM

Change 722664 merged by Bstorm:

[operations/puppet@production] toolforge harbor: install clients for redis and postgres

https://gerrit.wikimedia.org/r/722664

Maintenance_bot removed a project: Patch-For-Review.Sep 21 2021, 7:10 PM
dcaro subscribed.Sep 22 2021, 8:16 AM
gerritbot added a comment.Sep 30 2021, 5:28 PM

Change 725048 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: add external postgres db

https://gerrit.wikimedia.org/r/725048

gerritbot added a project: Patch-For-Review.Sep 30 2021, 5:28 PM
• Bstorm added a comment.Sep 30 2021, 5:29 PM

Using trove for Postgres in the most recent iteration is terrible. You cannot control it much, and it doesn't actually allow you access to the Postgres account to create a database. This means you can have exactly one database and user. I doubt the replication still works as well. Maybe it will be improved as they settle in to their more containerized setup.

gerritbot added a comment.Oct 1 2021, 3:36 PM

Change 725048 merged by Bstorm:

[operations/puppet@production] toolforge harbor: add external postgres db

https://gerrit.wikimedia.org/r/725048

Maintenance_bot removed a project: Patch-For-Review.Oct 1 2021, 4:10 PM
gerritbot added a comment.Oct 5 2021, 11:58 PM

Change 726723 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge postgres: drop database tuning

https://gerrit.wikimedia.org/r/726723

gerritbot added a project: Patch-For-Review.Oct 5 2021, 11:58 PM
gerritbot added a comment.Oct 6 2021, 12:01 AM

Change 726723 merged by Bstorm:

[operations/puppet@production] toolforge postgres: drop database tuning

https://gerrit.wikimedia.org/r/726723

Maintenance_bot removed a project: Patch-For-Review.Oct 6 2021, 12:11 AM
gerritbot added a comment.Oct 8 2021, 12:28 AM

Change 727638 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: dockerize the config file and such

https://gerrit.wikimedia.org/r/727638

gerritbot added a project: Patch-For-Review.Oct 8 2021, 12:28 AM
dcaro added a project: User-dcaro.Oct 8 2021, 7:24 AM
gerritbot added a comment.Oct 8 2021, 3:47 PM

Change 727638 merged by Bstorm:

[operations/puppet@production] toolforge harbor: puppetize the install/compose config file and such

https://gerrit.wikimedia.org/r/727638

gerritbot added a comment.Oct 8 2021, 4:12 PM

Change 728560 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: change the permissions a bit on the dir

https://gerrit.wikimedia.org/r/728560

gerritbot added a comment.Oct 8 2021, 4:22 PM

Change 728560 merged by Bstorm:

[operations/puppet@production] toolforge harbor: change the permissions a bit on the dir

https://gerrit.wikimedia.org/r/728560

gerritbot added a comment.Oct 8 2021, 5:03 PM

Change 728566 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: install docker-compose with puppet

https://gerrit.wikimedia.org/r/728566

gerritbot added a comment.Oct 8 2021, 5:06 PM

Change 728566 merged by Bstorm:

[operations/puppet@production] toolforge harbor: install docker-compose with puppet

https://gerrit.wikimedia.org/r/728566

gerritbot added a comment.Oct 8 2021, 5:43 PM

Change 728578 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: add small customization to prepare script here

https://gerrit.wikimedia.org/r/728578

gerritbot added a comment.Oct 8 2021, 5:58 PM

Change 728578 abandoned by Bstorm:

[operations/puppet@production] toolforge harbor: add small customization to prepare script here

Reason:

changing approach

https://gerrit.wikimedia.org/r/728578

gerritbot added a comment.Oct 8 2021, 5:58 PM

Change 728581 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: clean up the certs setup a bit better

https://gerrit.wikimedia.org/r/728581

gerritbot added a comment.Oct 8 2021, 6:06 PM

Change 728581 merged by Bstorm:

[operations/puppet@production] toolforge harbor: clean up the certs setup a bit better

https://gerrit.wikimedia.org/r/728581

gerritbot added a comment.Oct 8 2021, 7:39 PM

Change 728629 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: update certs with acmechief

https://gerrit.wikimedia.org/r/728629

dcaro renamed this task from Set up docker-registry and image builder infra in toolsbeta to Set up docker-registry (harbor) in toolsbeta.Oct 18 2021, 2:13 PM
dcaro claimed this task.
dcaro edited projects, added Toolforge Build Service; removed Toolforge.
dcaro updated the task description. (Show Details)
dcaro added a subscriber: • Bstorm.
dcaro added a project: Cloud-Services-Worktype-Maintenance.Nov 24 2021, 2:29 PM
dcaro added a project: Cloud-Services-Origin-Team.
dcaro added a project: Cloud-Services-Worktype-Project.Nov 24 2021, 3:25 PM
dcaro removed a project: Cloud-Services-Worktype-Maintenance.
dcaro changed the task status from Open to In Progress.Apr 14 2022, 8:37 AM
dcaro moved this task from To refine to Doing on the User-dcaro board.
dcaro changed the task status from In Progress to Open.Aug 23 2022, 8:14 AM
dcaro moved this task from Doing to Refined on the User-dcaro board.
dcaro added a parent task: T315720: DWL irc-buster VM did not come back up from unexpected cloudvirt reboot.Aug 26 2022, 8:22 AM
dcaro renamed this task from Set up docker-registry (harbor) in toolsbeta to [tbs.harbor] Puppetize the toolsbeta installation.Aug 26 2022, 8:24 AM
dcaro removed dcaro as the assignee of this task.
dcaro updated the task description. (Show Details)
dcaro removed a subscriber: • Bstorm.
dcaro added a subtask: T316232: [tbs.harbor.toolsbeta] Create a postgres DB with trove and plug the current instance there.Aug 26 2022, 8:25 AM
dcaro added a subtask: T316233: [tbs.harbor.toolsbeta] Create a cinder volume and put there the storage for the container images.
dcaro added a parent task: T316323: [tbs.harbor.tools] Replicate toolsbeta harbor deployment.Aug 26 2022, 8:27 AM
dcaro removed a parent task: T315720: DWL irc-buster VM did not come back up from unexpected cloudvirt reboot.Aug 26 2022, 8:48 AM
dcaro removed a parent task: T267374: [tbs.beta] Create a toolforge build service beta release.
dcaro added a parent task: T316330: [tbs.toolsbeta] Deploy a tool using a buildpack-based image on Toolsbeta.Aug 26 2022, 9:44 AM
dcaro closed subtask T316233: [tbs.harbor.toolsbeta] Create a cinder volume and put there the storage for the container images as Resolved.Aug 29 2022, 1:37 PM
dcaro added a parent task: T316541: [tbs.harbor] Rebuild toolsbeta-harborweb-2 with debian 11(bullseye).Aug 29 2022, 1:42 PM
Slst2020 subscribed.Aug 30 2022, 9:33 AM

This is the latest version from previous work on this, right?

https://gerrit.wikimedia.org/r/c/operations/puppet/+/728586/1/modules/profile/manifests/toolforge/harbor.pp

dcaro added a comment.Aug 30 2022, 10:14 AM

Probably, you can get the log of the file/directory if you prefer too

dcaro updated the task description. (Show Details)Oct 11 2022, 10:33 AM
dcaro updated the task description. (Show Details)
dcaro moved this task from Backlog to To be Discussed on the Toolforge Build Service board.Oct 19 2022, 12:33 PM
Slst2020 closed subtask T316232: [tbs.harbor.toolsbeta] Create a postgres DB with trove and plug the current instance there as Resolved.Oct 20 2022, 9:41 AM
dcaro updated the task description. (Show Details)Oct 25 2022, 1:38 PM
KHernandez-WMF assigned this task to Slst2020.Oct 25 2022, 1:43 PM
KHernandez-WMF moved this task from To be Discussed to Discussed/Estimated on the Toolforge Build Service board.
KHernandez-WMF set the point value for this task to 5.
KHernandez-WMF moved this task from Discussed/Estimated to Iteration 03 on the Toolforge Build Service board.Oct 25 2022, 3:37 PM
KHernandez-WMF edited projects, added Toolforge Build Service (Iteration 03); removed Toolforge Build Service.
KHernandez-WMF moved this task from Iteration 03 to Iteration 04 on the Toolforge Build Service board.Nov 8 2022, 11:26 PM
KHernandez-WMF edited projects, added Toolforge Build Service (Iteration 04); removed Toolforge Build Service (Iteration 03).
Slst2020 moved this task from Next Up to In Progress on the Toolforge Build Service (Iteration 04) board.Nov 10 2022, 10:20 AM
Slst2020 removed Slst2020 as the assignee of this task.Nov 10 2022, 2:41 PM
Slst2020 moved this task from In Progress to Next Up on the Toolforge Build Service (Iteration 04) board.
dcaro moved this task from Next Up to In Progress on the Toolforge Build Service (Iteration 04) board.Nov 10 2022, 2:43 PM
dcaro changed the task status from Open to In Progress.Nov 10 2022, 2:44 PM
dcaro claimed this task.
dcaro moved this task from Refined to Doing on the User-dcaro board.
KHernandez-WMF moved this task from Iteration 04 to Iteration 05 on the Toolforge Build Service board.Nov 22 2022, 5:50 PM
KHernandez-WMF edited projects, added Toolforge Build Service (Iteration 05); removed Toolforge Build Service (Iteration 04).
KHernandez-WMF moved this task from Next Up to In Progress on the Toolforge Build Service (Iteration 05) board.
gerritbot added a comment.Nov 24 2022, 4:47 PM

Change 860623 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] harbor: remove support for <bullseye

https://gerrit.wikimedia.org/r/860623

gerritbot added a comment.Nov 24 2022, 5:05 PM

Change 860627 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] harbor: remove unused harbor::db module/role

https://gerrit.wikimedia.org/r/860627

gerritbot added a comment.Nov 25 2022, 2:20 PM

Change 860896 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] harbor: ensure that it's started

https://gerrit.wikimedia.org/r/860896

dcaro moved this task from In Progress to In Review on the Toolforge Build Service (Iteration 05) board.Nov 25 2022, 2:21 PM
dcaro moved this task from In Review to In Progress on the Toolforge Build Service (Iteration 05) board.Nov 30 2022, 5:26 PM
dcaro moved this task from In Progress to In Review on the Toolforge Build Service (Iteration 05) board.Nov 30 2022, 5:32 PM
dcaro moved this task from In Review to In Progress on the Toolforge Build Service (Iteration 05) board.
gerritbot added a comment.Dec 1 2022, 9:54 AM

Change 860623 merged by David Caro:

[operations/puppet@production] harbor: remove support for <bullseye

https://gerrit.wikimedia.org/r/860623

gerritbot added a comment.Dec 1 2022, 9:54 AM

Change 860627 merged by David Caro:

[operations/puppet@production] harbor: remove unused harbor::db module/role

https://gerrit.wikimedia.org/r/860627

gerritbot added a comment.Dec 1 2022, 9:54 AM

Change 728629 merged by David Caro:

[operations/puppet@production] toolforge harbor: update certs with acmechief

https://gerrit.wikimedia.org/r/728629

gerritbot added a comment.Dec 1 2022, 9:54 AM

Change 860896 merged by David Caro:

[operations/puppet@production] harbor: ensure that it's started

https://gerrit.wikimedia.org/r/860896

dcaro updated the task description. (Show Details)Dec 1 2022, 9:58 AM
dcaro closed this task as Resolved.Dec 1 2022, 9:58 AM
dcaro moved this task from Doing to Done on the User-dcaro board.
dcaro moved this task from In Progress to Done on the Toolforge Build Service (Iteration 05) board.Dec 1 2022, 9:58 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL