Page MenuHomePhabricator

Error when executing helmfile commands for the termbox service
Closed, ResolvedPublic

Description

We are currently unable to deploy a new version of the termbox service. We get the following error when executing any helmfile command. See e.g. here:

ladsgroup@deploy1001:/srv/deployment-charts/helmfile.d/services/staging/termbox$ source .hfenv; helmfile --selector name=test diff
Adding repo stable https://releases.wikimedia.org/charts/
"stable" has been added to your repositories

Updating repo
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈ 

helmfile.yaml: basePath=.
Comparing test stable/termbox
in ./helmfile.yaml: failed processing release test: helm exited with status 1:
  Error: forwarding ports: error upgrading connection: pods "tiller-deploy-6d464bb9c7-f7jjl" is forbidden: User "termbox" cannot create resource "pods/portforward" in API group "" in the namespace "termbox"

Event Timeline

Joe triaged this task as High priority.
Joe subscribed.

@Jakob_WMDE this is a result of our temporary fix for a CVE affecting kubernetes. We will try to revert the situation tomorrow. Thanks for your patience.

@Pablo-WMDE not sure if you want to have this flagged up as this week's incident manager. I think being unable to deploy urgent bugfixes counts as an incident.

@Tarrow if it's an urgent bugfix we can just revert the change to let you deploy immediately. Please let's coordinate on IRC, and sorry for the inconvenience :)

Discussed that with Jakob yesterday and am watching this, was under the assumption that it would unlock itself during the course of today - one way or the other. Seeing T236677: Impossible to uncollapose termbox changes deployed would be a blast.

@Tarrow @Pablo-WMDE can someone try the release to staging? I should have fixed the rbac roles there. It should've fixed your issues.

I am proceeding with releasing the change on the main clusters too, in the meanwhile.

I have just tested and I can easily run helmfile diff on termbox now, in all environments. Resolving for now