Toolforge kubernetes kube-proxy broken by recent production changes
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Bstorm
	Dec 2 2019, 10:51 PM

Description

Toolforge is not ready for full upgrade of the Kubernetes cluster, and we need the old cluster live while we migrate.

https://gerrit.wikimedia.org/r/c/operations/puppet/+/554036 has introduced an error that prevents any new web services being launched:
Dec 02 18:21:11 tools-proxy-05 kube-proxy[30558]: unknown flag: --metrics-bind-address

	Subject	Repo	Branch	Lines +/-
	kube-proxy: Fix toolforge kube-proxy	operations/puppet	production	+14 -9

• Bstorm triaged this task as Unbreak Now! priority.Dec 2 2019, 10:51 PM

• Bstorm created this task.

Change 554178 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] kube-proxy: Fix toolforge kube-proxy

Change 554178 merged by Bstorm:
[operations/puppet@production] kube-proxy: Fix toolforge kube-proxy

Checked that my patch had no effect on production hosts (cc @akosiaris )

So at this point, the proxy seems to work right, but the worker nodes are unable to disable this flag still.

Fixed on the nodes now.

My two tools came up nicely yesterday evening/night.