Page MenuHomePhabricator
Create Task
Maniphest T267616

Set up docker-registry (harbor) in toolsbeta
Open, In Progress, HighPublic
Actions

Description

For testing and building out the CD pipeline in toolsbeta, we need to have a separate docker-registry hosted by harbor.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+47 -1toolforge harbor: update certs with acmechief
operations/puppetproduction+86 -4toolforge harbor: clean up the certs setup a bit better
operations/puppetproduction+78 -0toolforge harbor: add small customization to prepare script here
operations/puppetproduction+2 -2toolforge harbor: install docker-compose with puppet
operations/puppetproduction+2 -2toolforge harbor: change the permissions a bit on the dir
operations/puppetproduction+233 -0toolforge harbor: puppetize the install/compose config file and such
operations/puppetproduction+0 -2toolforge postgres: drop database tuning
operations/puppetproduction+70 -0toolforge harbor: add external postgres db
operations/puppetproduction+2 -0toolforge harbor: install clients for redis and postgres
operations/puppetproduction+12 -0toolforge harbor: puppetize experimental base server for harbor
Customize query in gerrit

Related Objects
Search...

Event Timeline

Legoktm created this task.Nov 9 2020, 10:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 9 2020, 10:31 PM
Legoktm mentioned this in T267618: Request increased quota for toolsbeta Cloud VPS project.Nov 9 2020, 10:55 PM
Legoktm added a subtask: T267618: Request increased quota for toolsbeta Cloud VPS project.Nov 9 2020, 11:05 PM
Stashbot added a comment.Nov 10 2020, 6:27 PM

Mentioned in SAL (#wikimedia-cloud) [2020-11-10T18:27:06Z] <legoktm> creating toolsbeta-docker-imagebuilder-01 (T267616)

Andrew closed subtask T267618: Request increased quota for toolsbeta Cloud VPS project as Resolved.Nov 10 2020, 6:44 PM
Andrew triaged this task as Medium priority.Dec 8 2020, 5:38 PM
Andrew moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
Legoktm removed Legoktm as the assignee of this task.Mar 8 2021, 11:06 PM
aborrero assigned this task to Bstorm.May 11 2021, 4:45 PM
aborrero moved this task from Doing to Soon! on the cloud-services-team (Kanban) board.
gerritbot added a comment.Jul 23 2021, 8:39 PM

Change 707572 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: puppetize experimental base server for harbor

https://gerrit.wikimedia.org/r/707572

gerritbot added a project: Patch-For-Review.Jul 23 2021, 8:39 PM

Change 707572 merged by Bstorm:

[operations/puppet@production] toolforge harbor: puppetize experimental base server for harbor

https://gerrit.wikimedia.org/r/707572

Maintenance_bot removed a project: Patch-For-Review.Jul 23 2021, 9:10 PM
Bstorm added a comment.Jul 24 2021, 12:12 AM

Ok, so I have a very nice https://goharbor.io server running in toolsbeta (using independent database auth for testing) https://harbor.toolsbeta.wmflabs.org
It has APIs for adding just about everything and has robot auth, etc. It grant plenty of flexibility for image management (even quotas).

It hooks up to LDAP, but that's not necessarily helpful here. I think to do this right, users need to not have direct push rights. That needs to be handled by another build service. However, with this, that service can easily run in k8s.

Bstorm raised the priority of this task from Medium to High.Sep 15 2021, 4:25 PM

In order to deploy this to tools soon, I'm proposing replacing tools-docker-registry with harbor running in k8s. However, unless we deploy cinder for k8s, we'll be doing that with NFS storage.

Bstorm added a comment.Sep 16 2021, 9:01 PM

I've deployed harbor in toolsbeta, however, I want to modify the deployment a bit before replacing docker-registry with it or putting it in tools.

gerritbot added a comment.Sep 21 2021, 6:05 PM

Change 722664 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: install clients for redis and postgres

https://gerrit.wikimedia.org/r/722664

gerritbot added a project: Patch-For-Review.Sep 21 2021, 6:05 PM
gerritbot added a comment.Sep 21 2021, 6:15 PM

Change 722664 merged by Bstorm:

[operations/puppet@production] toolforge harbor: install clients for redis and postgres

https://gerrit.wikimedia.org/r/722664

Maintenance_bot removed a project: Patch-For-Review.Sep 21 2021, 7:10 PM
dcaro added a subscriber: dcaro.Sep 22 2021, 8:16 AM
gerritbot added a comment.Sep 30 2021, 5:28 PM

Change 725048 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: add external postgres db

https://gerrit.wikimedia.org/r/725048

gerritbot added a project: Patch-For-Review.Sep 30 2021, 5:28 PM
Bstorm added a comment.Sep 30 2021, 5:29 PM

Using trove for Postgres in the most recent iteration is terrible. You cannot control it much, and it doesn't actually allow you access to the Postgres account to create a database. This means you can have exactly one database and user. I doubt the replication still works as well. Maybe it will be improved as they settle in to their more containerized setup.

gerritbot added a comment.Oct 1 2021, 3:36 PM

Change 725048 merged by Bstorm:

[operations/puppet@production] toolforge harbor: add external postgres db

https://gerrit.wikimedia.org/r/725048

Maintenance_bot removed a project: Patch-For-Review.Oct 1 2021, 4:10 PM
gerritbot added a comment.Oct 5 2021, 11:58 PM

Change 726723 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge postgres: drop database tuning

https://gerrit.wikimedia.org/r/726723

gerritbot added a project: Patch-For-Review.Oct 5 2021, 11:58 PM
gerritbot added a comment.Oct 6 2021, 12:01 AM

Change 726723 merged by Bstorm:

[operations/puppet@production] toolforge postgres: drop database tuning

https://gerrit.wikimedia.org/r/726723

Maintenance_bot removed a project: Patch-For-Review.Oct 6 2021, 12:11 AM
gerritbot added a comment.Oct 8 2021, 12:28 AM

Change 727638 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: dockerize the config file and such

https://gerrit.wikimedia.org/r/727638

gerritbot added a project: Patch-For-Review.Oct 8 2021, 12:28 AM
dcaro added a project: User-dcaro.Oct 8 2021, 7:24 AM
gerritbot added a comment.Oct 8 2021, 3:47 PM

Change 727638 merged by Bstorm:

[operations/puppet@production] toolforge harbor: puppetize the install/compose config file and such

https://gerrit.wikimedia.org/r/727638

gerritbot added a comment.Oct 8 2021, 4:12 PM

Change 728560 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: change the permissions a bit on the dir

https://gerrit.wikimedia.org/r/728560

gerritbot added a comment.Oct 8 2021, 4:22 PM

Change 728560 merged by Bstorm:

[operations/puppet@production] toolforge harbor: change the permissions a bit on the dir

https://gerrit.wikimedia.org/r/728560

gerritbot added a comment.Oct 8 2021, 5:03 PM

Change 728566 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: install docker-compose with puppet

https://gerrit.wikimedia.org/r/728566

gerritbot added a comment.Oct 8 2021, 5:06 PM

Change 728566 merged by Bstorm:

[operations/puppet@production] toolforge harbor: install docker-compose with puppet

https://gerrit.wikimedia.org/r/728566

gerritbot added a comment.Oct 8 2021, 5:43 PM

Change 728578 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: add small customization to prepare script here

https://gerrit.wikimedia.org/r/728578

gerritbot added a comment.Oct 8 2021, 5:58 PM

Change 728578 abandoned by Bstorm:

[operations/puppet@production] toolforge harbor: add small customization to prepare script here

Reason:

changing approach

https://gerrit.wikimedia.org/r/728578

gerritbot added a comment.Oct 8 2021, 5:58 PM

Change 728581 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: clean up the certs setup a bit better

https://gerrit.wikimedia.org/r/728581

gerritbot added a comment.Oct 8 2021, 6:06 PM

Change 728581 merged by Bstorm:

[operations/puppet@production] toolforge harbor: clean up the certs setup a bit better

https://gerrit.wikimedia.org/r/728581

gerritbot added a comment.Oct 8 2021, 7:39 PM

Change 728629 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] toolforge harbor: update certs with acmechief

https://gerrit.wikimedia.org/r/728629

dcaro renamed this task from Set up docker-registry and image builder infra in toolsbeta to Set up docker-registry (harbor) in toolsbeta.Oct 18 2021, 2:13 PM
dcaro claimed this task.
dcaro edited projects, added Toolforge Build Service; removed Toolforge.
dcaro updated the task description. (Show Details)
dcaro added a subscriber: Bstorm.
dcaro added a project: Cloud-Services-Worktype-Maintenance.Nov 24 2021, 2:29 PM
dcaro added a project: Cloud-Services-Origin-Team.
dcaro added a project: Cloud-Services-Worktype-Project.Nov 24 2021, 3:25 PM
dcaro removed a project: Cloud-Services-Worktype-Maintenance.
dcaro changed the task status from Open to In Progress.Apr 14 2022, 8:37 AM
dcaro moved this task from To refine to Doing on the User-dcaro board.
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL