⚓ T341084 [toolforge] Move all the components to the gitlab ci/cd flow

	Subject	Repo	Branch	Lines +/-
	MOVED TO GITLAB	labs/tools/maintain-kubeusers	master	+9 -12 K
	jobs-framework-emailer: move to gitlab	cloud/toolforge/jobs-framework-emailer	main	+1 -2 K

Title	Reference	Author	Source Branch	Dest Branch
README: update the docs enumerating the components	repos/cloud/toolforge/toolforge-deploy!96	dcaro	update_readme	main
deploy: use toolforge cd flows	repos/cloud/toolforge/ingress-admission!2	dcaro	move_to_toolforge_deploy	master
builds-builder: add component	repos/cloud/toolforge/toolforge-deploy!81	dcaro	add_builds_builder	main
deploy: move to helmfile	repos/cloud/toolforge/builds-builder!9	dcaro	move_to_standard_ci	main
maintain_kubeusers: add component	repos/cloud/toolforge/toolforge-deploy!80	dcaro	add_maintain_kubeusers	main
deploy: move to toolforge-deploy	repos/cloud/toolforge/maintain-kubeusers!1	dcaro	move_to_toolforge_deploy	main
registry-admission: add component	repos/cloud/toolforge/toolforge-deploy!68	dcaro	add_registry_admission	main
components: add jobs-emailer	repos/cloud/toolforge/toolforge-deploy!64	aborrero	arturo-components-add-jobs-ema	main
jobs-framework-emailer: migrate to modern workflow	repos/cloud/toolforge/jobs-emailer!4	aborrero	arturo-jobs-framework-emailer	main
builds-admission: bump to 0.0.63	repos/cloud/toolforge/toolforge-deploy!62	dcaro	build-admission_bump_0.0.64	main
api-gateway: bump to 0.0.16	repos/cloud/toolforge/toolforge-deploy!59	dcaro	api-gateway_bump_0.0.16	main
ingress-admission: add component	repos/cloud/toolforge/toolforge-deploy!57	dcaro	add_ingress-admission	main
deploy: use toolforge cd flows	repos/cloud/toolforge/ingress-admission!1	dcaro	move_to_toolforge_deploy	master
calico: add component	repos/cloud/toolforge/toolforge-deploy!54	dcaro	add_calico	main
deploy: move to toolforge-deploy	repos/cloud/toolforge/calico!5	dcaro	move_to_toolforge_deploy	main
kubernetes-metrics: add component	repos/cloud/toolforge/toolforge-deploy!53	dcaro	kuberenetes-metrcis_move_to_deploy_repo	main
deploy: move to the toolforge-deploy repository	repos/cloud/toolforge/wmcs-k8s-metrics!3	dcaro	move_to_toolforge_deploy	main
ingres-nginx: add component	repos/cloud/toolforge/toolforge-deploy!52	dcaro	ingress-nginx_move_to_delpoy_repo	main
image-config: bump to 0.0.14	repos/cloud/toolforge/toolforge-deploy!51	dcaro	image-config_bump_0.0.14	main
volume-admission: bump to 0.0.40	repos/cloud/toolforge/toolforge-deploy!50	dcaro	volume-admission_bump_0.0.40	main

Status	Assigned	Task
Resolved	taavi	T320446 The following container images did not match any of the allowed registries ([['docker-registry.tools.wmflabs.org']])
Resolved	taavi	T320476 do not use :latest for toolforge infrastructure components
Resolved	taavi	T320667 Cloud services enhancement proposal: Toolforge Kubernetes component workflow improvements
Resolved	dcaro	T336130 Automatically build Toolforge infrastructure container images in GitLab
Open	None	T334399 Move Toolforge PipelineLib repositories to GitLab
Resolved	LucasWerkmeister	T320140 Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes
Resolved	matmarex	T319707 Migrate dtcheck from Toolforge GridEngine to Toolforge Kubernetes
Resolved	Legoktm	T320062 Migrate steve-adder from Toolforge GridEngine to Toolforge Kubernetes
Resolved	Legoktm	T320011 Migrate rfa-voting-history from Toolforge GridEngine to Toolforge Kubernetes
Open	dcaro	T194332 [Epic,builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs
Resolved	dcaro	T327057 [toolforge] repositories move to gitlab
Resolved	dcaro	T341084 [toolforge] Move all the components to the gitlab ci/cd flow

dcaro updated the task description. (Show Details)Jul 11 2023, 9:41 AM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/1

deploy: use toolforge cd flows

Mentioned in SAL (#wikimedia-cloud) [2023-07-11T10:14:02Z] <dcaro> deploy ingress-admission 0.0.38, ci rebuild (T341084)

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/57

ingress-admission: add component

dcaro updated the task description. (Show Details)Jul 11 2023, 10:16 AM

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/59

api-gateway: bump to 0.0.16

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/59

api-gateway: bump to 0.0.16

aborrero moved this task from Next to Doing on the User-aborrero board.Jul 12 2023, 9:47 AM

Mentioned in SAL (#wikimedia-cloud) [2023-07-12T10:06:42Z] <dcaro> deployed api-gateway 0.0.16, no changes, ci rebuild (T341084)

dcaro updated the task description. (Show Details)Jul 12 2023, 10:07 AM

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/62

builds-admission: bump to 0.0.63

Mentioned in SAL (#wikimedia-cloud) [2023-07-12T12:46:39Z] <arturo> deployed builds-admission 0.0.63-20230712120152-2ef80a7c (T341084)

Mentioned in SAL (#wikimedia-cloud) [2023-07-12T12:46:46Z] <arturo> deployed builds-admission 0.0.63-20230712120152-2ef80a7c (T341084)

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/62

builds-admission: bump to 0.0.63

dcaro updated the task description. (Show Details)Jul 12 2023, 12:49 PM

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager/-/merge_requests/3

Move to toolforge-deploy

I'll be taking a bit of a break on this until I'm back on pto, if anyone wants to keep at it while I'm away I'll be more than happy, feel free :)

dcaro updated the task description. (Show Details)Jul 13 2023, 1:53 PM

dcaro added a parent task: T334399: Move Toolforge PipelineLib repositories to GitLab.Jul 14 2023, 6:29 AM

dcaro mentioned this in T332953: Migrate PipelineLib repos to GitLab.Jul 14 2023, 8:09 AM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-framework-emailer/-/merge_requests/4

jobs-framework-emailer: migrate to modern workflow

aborrero triaged this task as Medium priority.Jul 17 2023, 12:21 PM

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/64

components: add jobs-emailer

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-framework-emailer/-/merge_requests/4

jobs-framework-emailer: migrate to modern workflow

Mentioned in SAL (#wikimedia-cloud) [2023-07-18T10:45:50Z] <arturo> redeploy jobs-emailer into k8s (T341084)

Mentioned in SAL (#wikimedia-cloud) [2023-07-18T11:02:20Z] <arturo> redeploy jobs-emailer 0.0.41-20230718103342-3dddcfb8 into k8s (T341084)

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/64

components: add jobs-emailer

aborrero updated the task description. (Show Details)Jul 18 2023, 11:09 AM

aborrero moved this task from Doing to Radar on the User-aborrero board.Jul 18 2023, 12:03 PM

fnegri added projects: cloud-services-team (FY2023/2024-Q1-Q2), Goal.Jul 26 2023, 3:57 PM

fnegri moved this task from Backlog to Planned Goals on the cloud-services-team (FY2023/2024-Q1-Q2) board.

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/68

registry-admission: add component

dcaro updated the task description. (Show Details)Aug 2 2023, 9:28 AM

dcaro updated the task description. (Show Details)

dcaro updated the task description. (Show Details)Aug 2 2023, 9:31 AM

Change 945753 had a related patch set uploaded (by David Caro; author: David Caro):

[labs/tools/maintain-kubeusers@master] MOVED TO GITLAB

https://gerrit.wikimedia.org/r/945753

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/1

deploy: move to toolforge-deploy

dcaro updated https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/80

maintain_kubeusers: add component

Change 945753 merged by jenkins-bot:

[labs/tools/maintain-kubeusers@master] MOVED TO GITLAB

https://gerrit.wikimedia.org/r/945753

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/68

registry-admission: add component

dcaro updated the task description. (Show Details)Sep 5 2023, 1:35 PM

dcaro mentioned this in rLTMK2993a16dde66: MOVED TO GITLAB.Sep 5 2023, 1:56 PM

dcaro updated the task description. (Show Details)Sep 8 2023, 1:07 PM

dcaro added a parent task: T327057: [toolforge] repositories move to gitlab.

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/9

deploy: move to helmfile

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/81

builds-builder: add component

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/3

deploy: move to the toolforge-deploy repository

Mentioned in SAL (#wikimedia-cloud) [2023-09-11T11:36:29Z] <dcaro> deploy kubernetes-metrics (T341084)

For kubernetes-metrics, I had to manually update the helm manifest to match the existing resources.

I think this happened when we upgraded to k8s 1.22, where clusterrole and clusterrolebinding moved from v1beta1 to v1, we might have done it by hand and helm did not see the change.

Anyhow, the procedure for that is:

Download the existing manifest:

root@toolsbeta-test-k8s-control-5:/srv/git/toolforge-deploy# kubectl get secret -o json -n metrics sh.helm.release.v1.wmcs-metrics.v1 | jq '.data.release' -r | base64 -d | base64 -d | gunzip > metrics.helm-manifest

# edit the manifest and change rbac.authorization.k8s.io/v1beta1 -> rbac.authorization.k8s.io/v1

root@toolsbeta-test-k8s-control-5:/srv/git/toolforge-deploy# DATA=$(cat metrics.helm-manifest  | gzip -c | base64 --wrap=0| base64 --wrap=0)
root@toolsbeta-test-k8s-control-5:/srv/git/toolforge-deploy# kubectl patch secret -n metrics sh.helm.release.v1.wmcs-metrics.v1 --type='json' -p="[{\"op\":\"replace\",\"path\":\"/data/release\",\"value\":\"$DATA\"}]"

Mentioned in SAL (#wikimedia-cloud) [2023-09-11T12:34:27Z] <dcaro> deploy kubernetes-metrics (T341084)

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/53

kubernetes-metrics: add component

dcaro updated the task description. (Show Details)Sep 11 2023, 1:58 PM

Mentioned in SAL (#wikimedia-cloud) [2023-09-11T16:05:17Z] <dcaro> deploy builds-builder (T341084)

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/1

deploy: move to toolforge-deploy

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T10:27:31Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T10:27:56Z] <wm-bot2> dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T10:29:24Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T10:30:25Z] <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T10:34:32Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T10:35:20Z] <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T10:38:18Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T10:38:44Z] <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T10:41:34Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T10:41:42Z] <wm-bot2> dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T12:37:08Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T12:37:23Z] <wm-bot2> dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T12:37:33Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T12:38:24Z] <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T12:40:20Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T12:40:23Z] <wm-bot2> dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T12:41:07Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T12:41:09Z] <wm-bot2> dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T12:51:07Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T12:51:30Z] <wm-bot2> dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T16:40:25Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T16:41:10Z] <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T17:13:16Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-13T17:14:03Z] <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone (T341084)

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/80

maintain_kubeusers: add component

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-14T08:09:33Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone (T341084)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-09-14T08:10:18Z] <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone (T341084)

dcaro updated the task description. (Show Details)Sep 15 2023, 3:20 PM

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/2

deploy: use toolforge cd flows

dcaro closed https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/2

deploy: use toolforge cd flows

dcaro updated the task description. (Show Details)Sep 21 2023, 2:46 PM

dcaro updated the task description. (Show Details)

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/96

README: update the docs enumerating the components

dcaro updated the task description. (Show Details)Sep 21 2023, 3:56 PM

dcaro moved this task from In Progress to Toolforge iteration 00 on the Toolforge board.Sep 22 2023, 8:52 AM

dcaro edited projects, added Toolforge (Toolforge iteration 00); removed Toolforge.

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/9

deploy: move to helmfile

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/81

builds-builder: add component

dcaro changed the task status from Open to In Progress.Sep 26 2023, 12:34 PM

dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 00) board.

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/96

README: update the docs enumerating the components

dcaro closed this task as Resolved.Oct 4 2023, 7:25 AM

dcaro moved this task from In Progress to Done on the Toolforge (Toolforge iteration 00) board.

fnegri moved this task from Planned Goals to Done on the cloud-services-team (FY2023/2024-Q1-Q2) board.Oct 5 2023, 4:35 PM

[toolforge] Move all the components to the gitlab ci/cd flow
Closed, ResolvedPublic
Actions

Description

Details

Related Objects
Search...

Event Timeline

	dcaro
	Jul 4 2023, 4:20 PM

[toolforge] Move all the components to the gitlab ci/cd flowClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

[toolforge] Move all the components to the gitlab ci/cd flow
Closed, ResolvedPublic
Actions

Related Objects
Search...