Toolforge: refresh puppet code for proxy (dynamicproxy) to support Debian Buster
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	aborrero
	Oct 9 2019, 11:59 AM

Description

The puppet code for the Toolforge general proxy (dynamicproxy) only supports Debian Jessie.

This ticket is to track related work to add support for Debian Buster and general refresh:

support for buster in puppet manifests
missing packages for buster (flannel, k8s, etc)
use new puppet tree namespace (toolforge vs toollabs)

Details

Subject	Repo	Branch	Lines +/-
toolforge: introduce new proxy role	operations/puppet	production	+807 -0
toolforge: aptly: add buster-toolsbeta repository	operations/puppet	production	+4 -3
toolforge: aptly: add buster-tools repository	operations/puppet	production	+3 -0
toolforge: apt_pinning: add Buster support	operations/puppet	production	+23 -2

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
		Restricted Task
Resolved	• Bstorm	T246122 Upgrade the Toolforge Kubernetes cluster to v1.16
		Restricted Task
Resolved	bd808	T232536 Toolforge Kubernetes internal API down, causing `webservice` and other tooling to fail
Resolved	• Bstorm	T236565 "tools" Cloud VPS project jessie deprecation
Resolved	aborrero	T101651 Set up toolsbeta more fully to help make testing easier
Resolved	• Bstorm	T166949 Homedir/UID info breaks after a while in Tools Kubernetes (can't read replica.my.cnf)
Resolved	• Bstorm	T246059 Add admin account creation to maintain-kubeusers
Resolved	• Bstorm	T154504 Make webservice backend default to kubernetes
Declined	None	T245230 Investigate cpu/ram requests and limits for DaemonSets pods
Resolved	• Bstorm	T214513 Deploy and migrate tools to a Kubernetes v1.15 or newer cluster
Resolved	aborrero	T215531 Deploy upgraded Kubernetes to toolsbeta
Resolved	aborrero	T228500 Toolforge: evaluate ingress mechanism
Resolved	aborrero	T234037 Toolforge ingress: decide on final layout of north-south proxy setup
Resolved	aborrero	T235059 Toolforge: refresh puppet code for proxy (dynamicproxy) to support Debian Buster

Event Timeline

aborrero triaged this task as Medium priority.Oct 9 2019, 11:59 AM

aborrero created this task.

aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

Mentioned in SAL (#wikimedia-cloud) [2019-10-09T11:59:47Z] <arturo> re-create toolsbeta-test-proxy-01 as Debian Buster (T235059)

Change 508560 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: refactor proxy role from toollabs

https://gerrit.wikimedia.org/r/508560

gerritbot added a project: Patch-For-Review.Oct 9 2019, 12:59 PM

Change 541854 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: apt_pinning: add Buster support

https://gerrit.wikimedia.org/r/541854

Change 541854 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: apt_pinning: add Buster support

https://gerrit.wikimedia.org/r/541854

Mentioned in SAL (#wikimedia-operations) [2019-10-10T13:16:47Z] <arturo> added flannel 0.5.5-4 to buster-wikimedia (T235059)

Change 542334 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: aptly: add buster-tools repository

https://gerrit.wikimedia.org/r/542334

Change 542334 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: aptly: add buster-tools repository

https://gerrit.wikimedia.org/r/542334

Change 542348 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: aptly: add buster-toolsbeta repository

https://gerrit.wikimedia.org/r/542348

Change 542348 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: aptly: add buster-toolsbeta repository

https://gerrit.wikimedia.org/r/542348

Mentioned in SAL (#wikimedia-cloud) [2019-10-11T10:46:07Z] <arturo> added logster_0.0.10-2~jessie1_all.deb to buster-tools and buster-toolsbeta (aptly) for T235059

Mentioned in SAL (#wikimedia-cloud) [2019-10-11T10:51:38Z] <arturo> added docker-engine_1.12.6-0~debian-jessie_amd64.deb to buster-tools and buster-toolsbeta (aptly) for T235059

Mentioned in SAL (#wikimedia-cloud) [2019-10-11T10:53:07Z] <arturo> added kubernetes-node_1.4.6-7_amd64.deb to buster-tools and buster-toolsbeta (aptly) for T235059

Mentioned in SAL (#wikimedia-cloud) [2019-10-11T11:55:48Z] <arturo> create tools-test-proxy-01 VM for testing T235059 and a puppet prefix for it

I think the patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/508560 is mostly ready.

Things I've tested:

kube-proxy seems happy
nginx seems happy
flannel seems happy (although need to update the list of proxy VMs in hiera so ferm allows the connection)

Something interesting is this error by kube2proxy:

Oct 11 12:44:46 tools-test-proxy-01 kube2proxy[12927]: redis.exceptions.ReadOnlyError: You can't write against a read only replica.
Oct 11 12:44:46 tools-test-proxy-01 systemd[1]: kube2proxy.service: Main process exited, code=exited, status=1/FAILURE

It is true that the redis version in jessie (2:2.8.17-1+deb8u7) is way different than the version in buster (5:5.0.3-4+deb10u1). Not sure what's going on yet.

aborrero mentioned this in T234037: Toolforge ingress: decide on final layout of north-south proxy setup.Oct 14 2019, 4:13 PM

Change 508560 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: introduce new proxy role

https://gerrit.wikimedia.org/r/508560

Maintenance_bot removed a project: Patch-For-Review.Oct 15 2019, 12:10 PM

This is done.

aborrero mentioned this in T235627: Toolforge: upgrade main proxy servers to Debian Buster.Oct 16 2019, 11:30 AM

aborrero mentioned this in T229237: Set up tools-buster repository in aptly to allow toolforge servers to be installed on buster.Oct 23 2019, 3:30 PM