toolforge: consider relocating core k8s components out of puppet into its own repository
Open, MediumPublic
Actions

Assigned To

None

Authored By

	aborrero
	Feb 1 2023, 10:51 AM

Description

As of this ticket, some core Toolforge k8s component configuration live in the ops/puppet.git tree. We're expected to load them all by hand into k8s. Puppet doesn't do it.

List of stuff (probably not complete):

RBAC (toolforge::k8s::config i.e, modules/toolforge/files/k8s/toolforge-tool-roles.yaml)
PSP (i.e modules/kubeadm/files/psp/base-pod-security-policies.yaml)
calico (i.e modules/kubeadm/templates/calicoctl.yaml.erb) now live at https://gitlab.wikimedia.org/repos/cloud/toolforge/calico

I don't think we have a lot of value having all that YAML coupled to the puppet git tree. Like what happened with the ingress component, we could move all that to a separate repository maintained as helm charts or whatever.

Some docs:

Details

	Subject	Repo	Branch	Lines +/-
	toolforge: kubeadm: drop calico deployment	operations/puppet	production	+0 -4 K

Customize query in gerrit

	Title	Reference	Author	Source Branch	Dest Branch
	calico: delete old manifests	repos/cloud/toolforge/calico!2	aborrero	arturo-calico-delete-old-manif	main
	calico: initial code dump	repos/cloud/toolforge/calico!1	aborrero	initial-code-dump	main

Customize query in GitLab

Related Objects

Mentioned In: T341084: [toolforge] Move all the components to the gitlab ci/cd flow
T329530: Convert all Toolforge custom components to standardized Helm based deployment
Mentioned Here: T320667: Cloud services enhancement proposal: Toolforge Kubernetes component workflow improvements

Event Timeline

aborrero created this task.Feb 1 2023, 10:51 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 1 2023, 10:51 AM

aborrero mentioned this in T329530: Convert all Toolforge custom components to standardized Helm based deployment.Feb 14 2023, 1:28 PM

aborrero triaged this task as Medium priority.Feb 14 2023, 1:32 PM

aborrero moved this task from Inbox to Soon! on the cloud-services-team board.

The base RBAC is a good candidate for moving into the maintain-kubeusers repository, I think.

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/1

calico: initial code dump

gerritbot added a project: Patch-For-Review.Feb 20 2023, 12:01 PM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/1

calico: initial code dump

Maintenance_bot removed a project: Patch-For-Review.Mar 6 2023, 12:10 PM

Mentioned in SAL (#wikimedia-cloud) [2023-03-06T12:16:38Z] <arturo> delete calico deployment, redeploy from https://gitlab.wikimedia.org/repos/cloud/toolforge/calico (T328539)

aborrero updated the task description. (Show Details)Mar 6 2023, 12:28 PM

aborrero updated the task description. (Show Details)

In T328539#8614259, @taavi wrote:

The base RBAC is a good candidate for moving into the maintain-kubeusers repository, I think.

The file only has a tool-observer role, and a PSP for the build service.

Perhaps is worth folding into some kind of security repo (no PSP keyword in the name of the repo given the rework that will happen to PSP soon in a few k8s versions)

What do you think?

Will this include creating a repo for toolforge where we bundle up all these components or similar? (something like a toolforge repo with a helmfile pulling all the others)

I'm a bit concern on the sprawl of components without keeping track of the combinations that are deployed.

If so, that might belong to that repository.

I'd prefer to keep those in an existing repository. I think tool-observer should be in maintain-kubeusers which has everything else for tool and user rbac setups. And the build service PSP either there or in the build service repository.

In T328539#8668284, @dcaro wrote:

Will this include creating a repo for toolforge where we bundle up all these components or similar? (something like a toolforge repo with a helmfile pulling all the others)

That is planned as a future part of T320667: Cloud services enhancement proposal: Toolforge Kubernetes component workflow improvements.

In T328539#8668284, @dcaro wrote:

Will this include creating a repo for toolforge where we bundle up all these components or similar? (something like a toolforge repo with a helmfile pulling all the others)

I'm a bit concern on the sprawl of components without keeping track of the combinations that are deployed.

If so, that might belong to that repository.

Yes, that's described in T320667: Cloud services enhancement proposal: Toolforge Kubernetes component workflow improvements, i.e, adopt some kind of gitops. But first we need to get stuff out of the puppet git tree.

Change 894641 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] toolforge: kubeadm: drop calico deployment

https://gerrit.wikimedia.org/r/894641

gerritbot added a project: Patch-For-Review.Mar 6 2023, 1:25 PM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/2

calico: delete old manifests

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/2

calico: delete old manifests

Change 894641 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] toolforge: kubeadm: drop calico deployment

https://gerrit.wikimedia.org/r/894641