Page MenuHomePhabricator

Toolforge kubernetes kube-proxy broken by recent production changes
Closed, ResolvedPublic

Description

Toolforge is not ready for full upgrade of the Kubernetes cluster, and we need the old cluster live while we migrate.

https://gerrit.wikimedia.org/r/c/operations/puppet/+/554036 has introduced an error that prevents any new web services being launched:
Dec 02 18:21:11 tools-proxy-05 kube-proxy[30558]: unknown flag: --metrics-bind-address

Details

Related Gerrit Patches:
operations/puppet : productionkube-proxy: Fix toolforge kube-proxy

Event Timeline

Bstorm triaged this task as Unbreak Now! priority.Dec 2 2019, 10:51 PM
Bstorm created this task.
Restricted Application added subscribers: Liuxinyu970226, Aklapper. · View Herald TranscriptDec 2 2019, 10:51 PM

Change 554178 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] kube-proxy: Fix toolforge kube-proxy

https://gerrit.wikimedia.org/r/554178

Change 554178 merged by Bstorm:
[operations/puppet@production] kube-proxy: Fix toolforge kube-proxy

https://gerrit.wikimedia.org/r/554178

Checked that my patch had no effect on production hosts (cc @akosiaris )

So at this point, the proxy seems to work right, but the worker nodes are unable to disable this flag still.

Fixed on the nodes now.

Bstorm closed this task as Resolved.Dec 3 2019, 12:42 AM
Joutbis removed a subscriber: Joutbis.Dec 3 2019, 1:35 AM

Thanks @Bstorm

My two tools came up nicely yesterday evening/night.