Page MenuHomePhabricator
Create Task
Maniphest T251305

Migrate to helm v3
Closed, ResolvedPublic
Actions

Description

Meta-task tracking things needed for helm 3 migration

Pre-Migration tasks

  • import and build helm3
  • ensure compatibility of helm plugins
  • add helm3 stuff to puppet
  • figure a way to gradually migrate helmfile.d/services (this should only be done after k8s has been upgraded to >= 1.16 in all clusters)
  • alternatively find a way to migrate all services at once (on a depooled cluster), verify success and do so on the second one
  • We need an alternative to recreate pods
  • We might need additional annotations to helm tests (see comments)
  • Find replacement and new workflow for missing Tiller serviceaccount (new RBAC rules and modifications in helmfiles)
  • Create and verify plan for migration of codfw and eqiad (see T251305#7492328)

Migration tasks:

  • verify migration path on staging-codfw
  • re-deploy and verify staging
  • re-deploy and verify codfw
  • re-deploy and verify eqiad

Post migration tasks:

  • remove tiller and tiller service accounts (742989)
  • uninstall helm2 from hosts (deploy, releases,contint), make helm3 default (753026)
  • Cleanup boilderplate code in helmfiles (737034)
  • remove environment state values helmBinary (751067)
  • bump charts to v2 api version T295750
  • remove helm2 from CI and rake files (746864, 747147, 747487, 747814, 748701)
  • remove depricated "helm.sh/hook": test-success annotation (757877 and also related to T276949)

Details

SubjectRepoBranchLines +/-
Remove the tiller imageoperations/docker-images/production-imagesmaster+0 -205
helmfile.d: Remove all reference to tillerNamespaceoperations/deployment-chartsmaster+0 -38
charts: remove depricated helm test annotation, fix hook-delete-policyoperations/deployment-chartsmaster+81 -57
deployment_server,::helm: remove helm2 supportoperations/puppetproduction+36 -37
changeprop/eventgate: bump kafka-dev dependencie to 0.1.0operations/deployment-chartsmaster+10 -10
charts: update charts to api v2operations/deployment-chartsmaster+144 -177
deployment_server: remove obsolete value helmBinaryoperations/puppetproduction+0 -7
services: cleanup helmfiles, update SAL loggingoperations/deployment-chartsmaster+549 -590
Rakefile: check only client helm versionoperations/deployment-chartsmaster+1 -1
helmfile.d/admin_ng: fix subjects of rolebinding in namespacesoperations/deployment-chartsmaster+2 -2
Rakefile/rake_modules: remove unused function helm_version() and cleanupoperations/deployment-chartsmaster+2 -19
Rakefile: remove helm2 from Rakefile, bump scaffold to v2 apioperations/deployment-chartsmaster+17 -13
Update utils.rb's helm_version functionoperations/deployment-chartsmaster+4 -5
jjb: update helm-linter job to releng/helm-linter:0.3.0integration/configmaster+1 -1
helm-linter: remove helm2 from Docker imageintegration/configmaster+10 -3
admin_ng: remove tilleroperations/deployment-chartsmaster+1 -180
hiera::role::common::deployment_server update helmBinary eqiadoperations/puppetproduction+2 -0
hiera::role::common::deployment_server update helmBinary codfwoperations/puppetproduction+2 -0
hiera::role::common::deployment_server update helmBinary stagingoperations/puppetproduction+2 -0
charts: bump common_templates to 0.4 and chart versionsoperations/deployment-chartsmaster+141 -95
services: add support to deploy all services with helm3operations/deployment-chartsmaster+755 -37
blubberoid: bump common_templates to 0.4 and chart versionoperations/deployment-chartsmaster+6 -4
services: deploy services with helm3operations/deployment-chartsmaster+21 -1
hiera::deployment_server add missing mathoid helm3 deploy useroperations/puppetproduction+1 -0
hiera:kubernetes:deployment_server add deploy users for helm3operations/puppetproduction+106 -35
hiera::deployment_server add helm3 deploy user to deployment serveroperations/puppetproduction+105 -35
hiera::role::common::kubernetes add helm3 deploy userslabs/privatemaster+74 -0
hier::common::deployment_server add environment helmfile-defaultsoperations/puppetproduction+1 -0
helmfile.d: move ml-services to the new helm3 deploy user/token configoperations/deployment-chartsmaster+2 -2
kubernetes: add token config for revscoring-editquality-deployoperations/puppetproduction+3 -1
helmfile.d admin add dedicated deploy useroperations/deployment-chartsmaster+48 -6
helmfile.d/admin make tiller components configurable per environmentoperations/deployment-chartsmaster+12 -2
helmfile.d admin rename view rbac resourcesoperations/deployment-chartsmaster+5 -5
jjb: update job to releng/helm-linter:0.2.11integration/configmaster+1 -1
helm-linter: Update helm and components, add helm3integration/configmaster+19 -3
New upsteam version 2.17.0operations/debs/helmmaster+10 -1
helm: Fix environment syntax in execoperations/puppetproduction+5 -5
helm: Add helm3 support to helm moduleoperations/puppetproduction+46 -2
New upstream version 3.1.3 plus helm3 supportoperations/debs/helm-diffmaster+11 -2
Import new helmfile upstream version 0.135.0operations/debs/helmfilemaster+168 -7
New upstream version v3.4.1operations/debs/helm3master+19 -0
Make debian-glue voting for helm3integration/configmaster+5 -1
Add debian directory and .gitreviewoperations/debs/helm3master+239 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Nov 24 2021, 9:14 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:14:43Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on mobileapps.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:14 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:14:45Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on proton.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:14 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:14:48Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on proton.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:14 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:14:51Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on push-notifications.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:14 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:14:54Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on push-notifications.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:14 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:14:57Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on recommendation-api.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:00Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on recommendation-api.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:03Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on sessionstore.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:06Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on sessionstore.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:09Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on shellbox.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:12Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on shellbox.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:15Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on shellbox-constraints.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:18Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on shellbox-constraints.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:21Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on shellbox-media.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:25Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on shellbox-media.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:28Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on shellbox-syntaxhighlight.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:31Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on shellbox-syntaxhighlight.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:34Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on shellbox-timeline.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:36Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on shellbox-timeline.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:40Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on similar-users.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:43Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on similar-users.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:46Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on tegola-vector-tiles.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:49Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on tegola-vector-tiles.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:52Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on termbox.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:55Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on termbox.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:15:58Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on wikifeeds.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:16 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:16:01Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on wikifeeds.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:16 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:16:03Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on zotero.svc.codfw.wmnet with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 24 2021, 9:16 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T09:16:07Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on zotero.svc.codfw.wmnet with reason: helm3 de-deploy T251305

LSobanski subscribed.Nov 24 2021, 10:00 AM
Stashbot added a comment.Nov 24 2021, 10:06 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-24T10:06:55Z] <jelto> downtime PyBal backends health check for helm3 de-deploy T251305. I'm keeping an eye on icing and remove downtime as soon as I'm finished

gerritbot added a comment.Nov 24 2021, 12:54 PM

Change 736822 merged by Jelto:

[operations/puppet@production] hiera::role::common::deployment_server update helmBinary codfw

https://gerrit.wikimedia.org/r/736822

Jelto added a comment.EditedNov 24 2021, 2:08 PM

The re-deploy of codfw was successful. Some take-aways are added here which came up in the codfw migration. The plan to migrate eqiad Kubernetes to helm3:

  • Announce maintenance some days ahead on ops list
  • Downtime PyBal backends health check (and keep one eye on icinga, because this downtime is quite generic)
  • Downtime Kubernetes services in eqiad (according to T277740).
cookbook sre.hosts.downtime -r "helm3 de-deploy T251305" -H 3 --force 'apertium.svc.eqiad.wmnet,api-gateway.svc.eqiad.wmnet,apple-search.svc.eqiad.wmnet,blubberoid.svc.eqiad.wmnet,citoid.svc.eqiad.wmnet,cxserver.svc.eqiad.wmnet,echostore.svc.eqiad.wmnet,eventgate-analytics.svc.eqiad.wmnet,eventgate-analytics-external.svc.eqiad.wmnet,eventgate-logging-external.svc.eqiad.wmnet,eventgate-main.svc.eqiad.wmnet,eventstreams.svc.eqiad.wmnet,eventstreams-internal.svc.eqiad.wmnet,linkrecommendation.svc.eqiad.wmnet,mathoid.svc.eqiad.wmnet,mobileapps.svc.eqiad.wmnet,mwdebug.svc.eqiad.wmnet,proton.svc.eqiad.wmnet,push-notifications.svc.eqiad.wmnet,recommendation-api.svc.eqiad.wmnet,sessionstore.svc.eqiad.wmnet,shellbox.svc.eqiad.wmnet,shellbox-constraints.svc.eqiad.wmnet,shellbox-media.svc.eqiad.wmnet,shellbox-syntaxhighlight.svc.eqiad.wmnet,shellbox-timeline.svc.eqiad.wmnet,similar-users.svc.eqiad.wmnet,tegola-vector-tiles.svc.eqiad.wmnet,termbox.svc.eqiad.wmnet,toolhub.svc.eqiad.wmnet,wikifeeds.svc.eqiad.wmnet,zotero.svc.eqiad.wmnet'
  • depool all Kubernetes services in eqiad (except mwdebug and toolhub):
confctl --object-type discovery select 'name=eqiad,dnsdisc=(apertium|api-gateway|apple-search|blubberoid|citoid|cxserver|echostore|eventgate-analytics|eventgate-analytics-external|eventgate-logging-external|eventstreams|eventstreams-internal|linkrecommendation|mathoid|mobileapps|proton|push-notifications|recommendation-api|sessionstore|shellbox|shellbox-constraints|shellbox-media|shellbox-syntaxhighlight|shellbox-timeline|similar-users|tegola-vector-tiles|termbox|wikifeeds|zotero)' set/pooled=false
  • validate services are fine and served by codfw only
  • dump all manifests from flink to save configmaps not needed
  • create lock for mwdebug auto deploy service flock /var/lib/deploy-mwdebug/flock sleep infinity
  • run redeploy: P17693
  • validate service redeploy went fine in eqiad
  • Switch services to both datacenters again:
confctl --object-type discovery select 'name=eqiad,dnsdisc=(apertium|api-gateway|apple-search|blubberoid|citoid|cxserver|echostore|eventgate-analytics|eventgate-analytics-external|eventgate-logging-external|eventstreams|eventstreams-internal|linkrecommendation|mathoid|mobileapps|proton|push-notifications|recommendation-api|sessionstore|shellbox|shellbox-constraints|shellbox-media|shellbox-syntaxhighlight|shellbox-timeline|similar-users|tegola-vector-tiles|termbox|wikifeeds|zotero)' set/pooled=true
  • bump environment state value helmBinary to helm3 (see 741681)
  • remove lock /var/lib/deploy-mwdebug/flock

Feel free to add any thoughts or additional steps in case I missed something.

Jelto updated the task description. (Show Details)Nov 24 2021, 2:08 PM
gerritbot added a comment.Nov 24 2021, 2:10 PM

Change 741681 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] hiera::role::common::deployment_server update helmBinary eqiad

https://gerrit.wikimedia.org/r/741681

Stashbot added a comment.Nov 25 2021, 7:09 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-25T07:09:49Z] <jelto> start re-deploy procedure in eqiad Kubernetes T251305

Stashbot added a comment.Nov 25 2021, 7:10 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-25T07:10:00Z] <jelto> downtime PyBal backends health check on lvs1015 and lvs1016 for helm3 de-deploy T251305. I'm keeping an eye on icing and remove downtime as soon as I'm finished

Stashbot added a comment.Nov 25 2021, 7:17 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-25T07:17:22Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on 32 hosts with reason: helm3 de-deploy T251305

Stashbot added a comment.Nov 25 2021, 7:17 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-25T07:17:46Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 32 hosts with reason: helm3 de-deploy T251305

gerritbot added a comment.Nov 25 2021, 10:05 AM

Change 741681 merged by Jelto:

[operations/puppet@production] hiera::role::common::deployment_server update helmBinary eqiad

https://gerrit.wikimedia.org/r/741681

Jelto added a comment.Nov 25 2021, 10:18 AM

cc from ops list:

The re-deploy for all services in the eqiad Kubernetes cluster was successful. However this time we had an impact on service availability. Planned reduced service availability happened for mwdebug and toolhub, which are only available on eqiad. These services weren't available for around 3 minutes. Unplanned reduced service availability happened for eventgate-main service due to a pooling mistake on my side. eventgate-main service was not available between 7:32 and 7:35 UTC and generated around ~25k exceptions (see Grafana).

All Kubernetes environments are running with helm3 now. For day-to-day deployments nothing should change for you. For low-level troubleshooting please keep in mind to use helm3 client instead of helm for the next time (until cleanup happened).

I'll proceed with cleanup steps (see task description) next week. You may also see some related follow-up tasks.

Jelto updated the task description. (Show Details)Nov 25 2021, 10:18 AM
Jelto updated the task description. (Show Details)Nov 25 2021, 10:40 AM
gerritbot added a comment.Dec 1 2021, 5:53 PM

Change 742989 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] admin_ng: remove tiller

https://gerrit.wikimedia.org/r/742989

Jelto updated the task description. (Show Details)Dec 1 2021, 5:54 PM
JMeybohm updated the task description. (Show Details)Dec 7 2021, 3:09 PM
gerritbot added a comment.Dec 10 2021, 2:26 PM

Change 742989 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng: remove tiller

https://gerrit.wikimedia.org/r/742989

Jelto added a comment.Dec 10 2021, 3:27 PM

First cleanup task is finished:

  • remove tiller and tiller service accounts (742989)

Tiller deployments and RBAC resources are removed from all Kubernetes environments (staging, codfw and eqiad).

cd /srv/deployment-charts/helmfile.d/admin_ng
kube_env admin staging-codfw
helmfile -e staging-codfw diff
helmfile -e staging-codfw -l name=namespaces apply # only namespaces because jayme is working on cert-manager 
helmfile -e staging-codfw -l name=rbac-rules apply # only rbac-rules because jayme is working on cert-manager

kube_env admin staging
helmfile -e staging-eqiad diff
helmfile -e staging-eqiad apply

kube_env admin codfw
helmfile -e codfw diff
helmfile -e codfw apply

kube_env admin eqiad
helmfile -e eqiad diff
helmfile -e eqiad apply
Jelto updated the task description. (Show Details)Dec 10 2021, 3:27 PM
gerritbot added a comment.Dec 13 2021, 1:28 PM

Change 746864 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] Rakefile: remove helm2 from Rakefile

https://gerrit.wikimedia.org/r/746864

Jelto updated the task description. (Show Details)Dec 13 2021, 1:31 PM
gerritbot added a comment.Dec 14 2021, 3:58 PM

Change 747147 had a related patch set uploaded (by Jelto; author: Jelto):

[integration/config@master] helm-linter: remove helm2 from Docker image

https://gerrit.wikimedia.org/r/747147

gerritbot added a comment.Dec 14 2021, 4:02 PM

Change 747148 had a related patch set uploaded (by Jelto; author: Jelto):

[integration/config@master] jjb: update helm-linter job to releng/helm-linter:0.3.0

https://gerrit.wikimedia.org/r/747148

gerritbot added a comment.Dec 15 2021, 8:23 AM

Change 747147 merged by jenkins-bot:

[integration/config@master] helm-linter: remove helm2 from Docker image

https://gerrit.wikimedia.org/r/747147

Stashbot added a comment.Dec 15 2021, 8:31 AM

Mentioned in SAL (#wikimedia-releng) [2021-12-15T08:31:01Z] <hashar> Update helm-lint job https://gerrit.wikimedia.org/r/c/integration/config/+/747148 T251305

gerritbot added a comment.Dec 15 2021, 8:32 AM

Change 747148 merged by jenkins-bot:

[integration/config@master] jjb: update helm-linter job to releng/helm-linter:0.3.0

https://gerrit.wikimedia.org/r/747148

gerritbot added a comment.Dec 15 2021, 9:02 AM

Change 747460 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] Update utils.rb's helm_version function

https://gerrit.wikimedia.org/r/747460

gerritbot added a comment.Dec 15 2021, 9:14 AM

Change 747460 abandoned by Elukey:

[operations/deployment-charts@master] Update utils.rb's helm_version function

Reason:

https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/746864

https://gerrit.wikimedia.org/r/747460

gerritbot added a comment.Dec 15 2021, 9:18 AM

Change 746864 merged by jenkins-bot:

[operations/deployment-charts@master] Rakefile: remove helm2 from Rakefile, bump scaffold to v2 api

https://gerrit.wikimedia.org/r/746864

gerritbot added a comment.Dec 15 2021, 9:57 AM

Change 747487 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] Rakefile/rake_modules: remove unused function helm_version() and cleanup

https://gerrit.wikimedia.org/r/747487

• dduvall subscribed.Dec 15 2021, 5:22 PM

The removal of tiller has broken PipelineLib's deploy functionality. For example, https://integration.wikimedia.org/ci/job/blubber-pipeline-rehearse/84/console

We'll need to refactor PipelineLib to use helm3 ASAP. Filing a task.

• dduvall mentioned this in T297809: PipelineLib deploy is broken and needs refactoring to use helm3.Dec 15 2021, 5:28 PM
gerritbot added a comment.Dec 16 2021, 11:31 AM

Change 747819 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] helmfile.d/admin_ng: fix subjects of rolebinding in namespaces

https://gerrit.wikimedia.org/r/747819

gerritbot added a comment.Dec 16 2021, 1:36 PM

Change 747487 merged by jenkins-bot:

[operations/deployment-charts@master] Rakefile/rake_modules: remove unused function helm_version() and cleanup

https://gerrit.wikimedia.org/r/747487

gerritbot added a comment.Dec 17 2021, 8:52 AM

Change 747819 merged by jenkins-bot:

[operations/deployment-charts@master] helmfile.d/admin_ng: fix subjects of rolebinding in namespaces

https://gerrit.wikimedia.org/r/747819

gerritbot added a comment.Dec 20 2021, 11:08 AM

Change 748701 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] Rakefile: check only client helm version

https://gerrit.wikimedia.org/r/748701

gerritbot added a comment.Dec 20 2021, 2:43 PM

Change 748701 merged by jenkins-bot:

[operations/deployment-charts@master] Rakefile: check only client helm version

https://gerrit.wikimedia.org/r/748701

Jelto updated the task description. (Show Details)Dec 22 2021, 2:59 PM
gerritbot added a comment.Jan 3 2022, 8:46 AM

Change 751067 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] deployment_server: remove obsolete value helmBinary

https://gerrit.wikimedia.org/r/751067

Jelto updated the task description. (Show Details)Jan 3 2022, 8:47 AM
gerritbot added a comment.Jan 3 2022, 9:29 AM

Change 751070 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] charts: update charts to api v2

https://gerrit.wikimedia.org/r/751070

gerritbot added a comment.Jan 3 2022, 1:38 PM

Change 751120 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] changeprop/eventgate: bump kafka-dev dependencie to 1.0.0

https://gerrit.wikimedia.org/r/751120

gerritbot added a comment.Jan 5 2022, 11:52 AM

Change 737034 merged by jenkins-bot:

[operations/deployment-charts@master] services: cleanup helmfiles, update SAL logging

https://gerrit.wikimedia.org/r/737034

gerritbot added a comment.Jan 5 2022, 1:17 PM

Change 751067 merged by Jelto:

[operations/puppet@production] deployment_server: remove obsolete value helmBinary

https://gerrit.wikimedia.org/r/751067

Jelto updated the task description. (Show Details)Jan 5 2022, 1:36 PM
gerritbot added a comment.Jan 6 2022, 10:39 AM

Change 751070 merged by jenkins-bot:

[operations/deployment-charts@master] charts: update charts to api v2

https://gerrit.wikimedia.org/r/751070

gerritbot added a comment.Jan 6 2022, 11:15 AM

Change 751120 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop/eventgate: bump kafka-dev dependencie to 0.1.0

https://gerrit.wikimedia.org/r/751120

Jelto closed subtask T295750: Helm chart dependencies no longer in requirements.yaml as Resolved.Jan 6 2022, 11:20 AM
Jelto updated the task description. (Show Details)
gerritbot added a comment.Jan 11 2022, 12:28 PM

Change 753026 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] deployment_server,::helm: remove helm2 support

https://gerrit.wikimedia.org/r/753026

Jelto updated the task description. (Show Details)Jan 11 2022, 12:28 PM
gerritbot added a comment.Jan 12 2022, 2:33 PM

Change 753026 merged by Jelto:

[operations/puppet@production] deployment_server,::helm: remove helm2 support

https://gerrit.wikimedia.org/r/753026

Stashbot added a comment.Jan 12 2022, 2:40 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-12T14:40:11Z] <jelto> remove helm2 from deployment_server T251305 https://gerrit.wikimedia.org/r/c/operations/puppet/+/753026

Jelto added a comment.Jan 12 2022, 3:20 PM

I removed helm2 from deploy1001 and deploy2001 by merging https://gerrit.wikimedia.org/r/753026. I tested the removal before on WMCS and a temporary pontoon setup before (see details here).

The removal of the systemd timer helm-repo-update.timer failed on the machines deploy1001 and contint2001 due to some race condition. The puppet execution, which removed the timer components ran during the execution of the helm-repo-update.timer. So the timer failed with:

systemctl status helm-repo-update.timer
● helm-repo-update.timer
   Loaded: not-found (Reason: Unit helm-repo-update.timer not found.)
   Active: failed (Result: resources) since Wed 2022-01-12 14:35:50 UTC; 19min ago
  Trigger: n/a

Jan 12 14:35:50 deploy1002 systemd[1]: helm-repo-update.timer: Failed to queue unit startup job: Unit helm-repo-update.service not found

I executed sudo systemctl reset-failed helm-repo-update.timer manually on deploy1001 and contint2001, because additional puppet runs could not clean up the obsolete/unmanaged timer entry.

helm2 is now removed from hosts deploy1001, deploy2001, contint1001 and contint2001:

$ helm2 version
-bash: helm2: command not found

helm links to helm3 now:

$ helm version
version.BuildInfo{Version:"v3.6.3", GitCommit:"", GitTreeState:"", GoVersion:"go1.15.9"}
Jelto updated the task description. (Show Details)Jan 12 2022, 3:21 PM
gerritbot added a comment.Jan 28 2022, 10:35 AM

Change 757877 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] charts: remove depricated helm test annotation, fix hook-delete-policy

https://gerrit.wikimedia.org/r/757877

Jelto mentioned this in T276949: Run helm test after deploy.Jan 28 2022, 1:01 PM
gerritbot added a comment.Jan 28 2022, 1:14 PM

Change 757877 merged by jenkins-bot:

[operations/deployment-charts@master] charts: remove depricated helm test annotation, fix hook-delete-policy

https://gerrit.wikimedia.org/r/757877

Jelto updated the task description. (Show Details)Jan 28 2022, 1:21 PM
Jelto closed this task as Resolved.Jan 28 2022, 1:30 PM

With the removal of deprecated helm2 test annotations in https://gerrit.wikimedia.org/r/757877 all (known) cleanup steps are finished. There is some open work regarding automatic testing of services in T276949. But this is not really related to the migration, but was also depended on helm3.

So I'm going to close this task. If you find some helm2 debris, feel free to re-open the task and link it here.

Thanks all for the support!

taavi awarded a token.Jan 28 2022, 1:30 PM
gerritbot added a comment.Apr 20 2022, 12:49 PM

Change 784227 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] helmfile.d: Remove all reference to tillerNamespace

https://gerrit.wikimedia.org/r/784227

gerritbot added a comment.Apr 20 2022, 1:03 PM

Change 784227 merged by jenkins-bot:

[operations/deployment-charts@master] helmfile.d: Remove all reference to tillerNamespace

https://gerrit.wikimedia.org/r/784227

gerritbot added a comment.Apr 21 2022, 12:00 AM

Change 784791 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] kubernetes::deployment_server: add new service image-suggestion

https://gerrit.wikimedia.org/r/784791

akosiaris removed a subtask: T277849: Convert helm releases to the new release naming schema.May 4 2022, 3:15 PM
gerritbot added a comment.Mon, Apr 15, 1:42 PM

Change #1019809 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/docker-images/production-images@master] Remove the tiller image

https://gerrit.wikimedia.org/r/1019809

gerritbot added a comment.Mon, Apr 15, 2:10 PM

Change #1019809 merged by JMeybohm:

[operations/docker-images/production-images@master] Remove the tiller image

https://gerrit.wikimedia.org/r/1019809

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL