Page MenuHomePhabricator
Create Task
Maniphest T235627

Toolforge: upgrade main proxy servers to Debian Buster
Closed, ResolvedPublic
Actions

Description

We have now support in the puppet tree for building Debian Buster based proxy servers in Toolforge (related: T235059: Toolforge: refresh puppet code for proxy (dynamicproxy) to support Debian Buster)

Currently, tools-proxy-03 and tools-proxy-04 are running Debian Jessie, so they need to be rebuild and switched from the old role (role::toollabs::proxy) to the new one (role::wmcs::toolforge::proxy),
The work for doing such rebuild has been scheduled for 2019-10-28 14:30 UTC. An announcement for the operation was published already: https://lists.wikimedia.org/pipermail/cloud-announce/2019-October/000226.html.

A checklist and concrete operation steps will be added to this task previous to the operation window.

We agreed on both @Bstorm and @JHedden supervising this operation.

Details

SubjectRepoBranchLines +/-
Fix labsaliaser script to be executableoperations/puppetproduction+4 -4
Revert "cloudvps: ignore stderr in labs-ip-alias-dump.py"operations/puppetproduction+1 -1
toollabs: delete unused proxy codeoperations/puppetproduction+0 -482
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
 Resolved• BstormT246122 Upgrade the Toolforge Kubernetes cluster to v1.16
Restricted Task
 Resolvedbd808T232536 Toolforge Kubernetes internal API down, causing `webservice` and other tooling to fail
 ResolvedaborreroT101651 Set up toolsbeta more fully to help make testing easier
 Resolved• BstormT166949 Homedir/UID info breaks after a while in Tools Kubernetes (can't read replica.my.cnf)
 Resolved• BstormT246059 Add admin account creation to maintain-kubeusers
 Resolved• BstormT154504 Make webservice backend default to kubernetes
 DeclinedNoneT245230 Investigate cpu/ram requests and limits for DaemonSets pods
 Resolved• BstormT236565 "tools" Cloud VPS project jessie deprecation
 Resolved• BstormT214513 Deploy and migrate tools to a Kubernetes v1.15 or newer cluster
 Resolved• BstormT111914 Setup DNS for kubernetes services
 ResolvedaborreroT142862 Setup Kubernetes Masters in a HA setup
 ResolvedaborreroT215663 Stand up upgraded Toolforge etcd clusters
 ResolvedaborreroT215530 Sort out the best method of spinning up multiple toolforge kubernetes masters
 ResolvedaborreroT215679 Sort out and test deploying the worker nodes in a sane fashion
 ResolvedaborreroT215975 Package/copy kubeadm, kubelet, docker-ce and kubectl to Toolforge Aptly or Reprepro
 ResolvedJproramaT172855 Create visual diagram of documented components of Toolforge Kubernetes cluster
 ResolvedaborreroT215531 Deploy upgraded Kubernetes to toolsbeta
 Resolved• BstormT215529 Puppetize/stand up a load balancer for K8s API servers
 ResolvedaborreroT226098 Toolforge: modernize deployment for etcd in k8s
 ResolvedaborreroT228267 Toolforge: iptables flavor for Debian Buster-based k8s cluster
 ResolvedaborreroT228500 Toolforge: evaluate ingress mechanism
 ResolvedaborreroT234032 Toolforge ingress: create a default landing page for unknown/default URLs
 ResolvedaborreroT234037 Toolforge ingress: decide on final layout of north-south proxy setup
 ResolvedaborreroT235059 Toolforge: refresh puppet code for proxy (dynamicproxy) to support Debian Buster
 Resolved• BstormT234231 Toolforge ingress: decide on how ingress configuration objects will be managed
 Resolved• dduvallT236203 Add CI checks for golang admission controllers
 Resolved• BstormT254293 Change to admission controller readme.md failed to pass gate-and-submit jobs
 ResolvedaborreroT228660 Toolforge: new k8s: issues with the initial coredns setup
 Resolved• BstormT228887 Update pause container in our internal registry
 Resolved• BstormT229009 Proposal: ditching the master name in kubernetes servers
 Resolved• BstormT234702 Review and establish configurable quotas for users in the new Kubernetes cluster
 ResolvedaborreroT236074 Toolforge: rebuild the new k8s toolsbeta deployment and write final docs
 ResolvedaborreroT236249 Toolforge: new k8s: upload internal docker images to our registry
 ResolvedaborreroT236824 Toolforge: new k8s: get new deb packages for 1.15.4 or 1.15.5
 ResolvedaborreroT237443 toolsbeta: new k8s: deploy a front proxy (dynamicproxy)
 Resolved• BstormT237541 CoreDNS in the new k8s cluster cannot talk to the Cloud recursors
 DeclinedNoneT238641 toolforge: some additional testing before final migration
 ResolvedaborreroT239403 toolforge: new k8s: scale up a bit the cluster before final tests and initial migrations
 OpenNoneT239404 toolforge: new k8s: evaluate DNS (coredns) autoscale options
 ResolvedaborreroT239405 toolforge: new k8s: evaluate ingress controller reload behaviour
 StalledNoneT239406 toolforge: new k8s: evalute and test firewalling via calico
 ResolvedaborreroT238655 toolforge: new k8s: issues with the apiserver and etcd
 Resolved• BstormT215553 Figure out cert management for Toolforge kubernetes and make it clear in documents, etc. for the upgrade
 Resolved• BstormT169287 etcd config depends on puppet certs, but puppet doesn't know
 ResolvedyuvipandaT119814 Figure out how to deal with SSL cert issues for kubernetes masters
 DuplicateNoneT144153 Move kubernetes authentication to using X.509 client certs
 Resolved• BstormT215678 Replace each of the custom controllers with something in a new Toolforge Kubernetes setup
 Resolved• BstormT227290 Design and document how to integrate the new Toolforge k8s cluster with PodSecurityPolicy
 Resolved• BstormT238162 Establish a process for renewing TLS certs for the 2 webhook controllers
 DuplicateNoneT224273 Toolforge: develop new k8s cluster in toolsbeta
 Resolved• BstormT233372 Create a "novaobserver" equivalent for Toolforge Kubernetes cluster inspection
Restricted Task
 ResolvedaborreroT235627 Toolforge: upgrade main proxy servers to Debian Buster
 DuplicateNoneT235756 Toolforge: webservice utility: add support for thew new k8s setup
 Resolved• BstormT236202 Modify webservice and maintain-kubeusers to allow switching to the new cluster
 Resolved• BstormT228499 Toolforge: changes to maintain-kubeusers
 Resolved• BstormT229058 Replace the nslcd mount in containers from the old Toolforge cluster with something that will work with sssd in the new one
 Resolved• BstormT237836 `webservice restart` regression with backend=kubernetes in webservice 0.51
 ResolvedaborreroT236826 Toolforge: new k8s: initial build of the new kubernetes cluster
 ResolvedaborreroT237633 Request increased quota for tools Cloud VPS project
 ResolvedaborreroT237643 toolforge: new k8s: figure out metrics / observability
 ResolvedaborreroT237557 new proxy and etcd nodes unreachable by ssh for tools-prometheus
 ResolvedaborreroT238058 toolforge: prometheus-node-exporter not working on tools-proxy-06
 ResolvedaborreroT238096 Toolforge: prometheus: refresh setup
 ResolvedaborreroT245180 Document and test failing over prometheus
 ResolvedaborreroT240402 Deploy or consciously decide not to deploy metrics-server in toolforge kubernetes
 ResolvedaborreroT241853 Move metrics-server and kube-state-metrics into the new metrics namespace
 Resolved• BstormT237784 Document migration plans and timelines
 Resolved• BstormT237789 Document (and execute) the upgrade process for the new Toolforge K8s cluster
 Resolved• BstormT238654 toolforge: new k8s: issues with routing interfering with DNS in the cluster as well as the webhook controllers
 DuplicateNoneT239407 toolforge: new k8s: package newer/more convenient python3 k8s client libs
 ResolvedaborreroT239409 toolforge: new k8s: introduce more robust controls for deb pkg versions
Restricted Task
 OpenNoneT272905 Reduce privs of metrics pods where we can
 Resolved• BstormT240922 Change name of commons_describer tool or provide some workaround for Kubernetes/DNS
 Resolvedbd808T240923 Fix toolschecker's insistence that a kubeconfig is json
 InvalidaborreroT240925 `kubectl get pods` fails after switching to new k8s cluster
 Resolvedbd808T241008 New k8s cluster routing behaving strangely for bd808-test tool
 Resolvedbd808T241310 Kubernetes ingress passes it's port & proto to apps rather than the port & proto from the front proxy
Restricted Task
 ResolvedSecurity• BstormT242067 Error joining new worker node to Toolforge Kubernetes cluster
 ResolvedaborreroT242719 https://tools.wmflabs.org/{toolname} no longer redirects to https://tools.wmflabs.org/{toolname}/ on new k8s cluster
 Resolvedbd808T242824 Tool account cannot list all namespaced objects in its Kubernetes namespace
 DuplicateNoneT243468 Add smarter resourcing logic to kubernetes backend of webservice
 Resolved• BstormT244289 Improve limit range management in webservice for Kubernetes
 Resolvedbd808T244293 Add a function to webservice called "migrate" that will push a tool from the old cluster on Kubernetes to the new one
 Resolvedbd808T244791 Scale up 2020 Kubernetes cluster for final migration of legacy cluster workloads

Event Timeline

aborrero triaged this task as Medium priority.Oct 16 2019, 11:29 AM
aborrero created this task.
aborrero updated the task description. (Show Details)
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero added a comment.Oct 16 2019, 11:33 AM

Please @Bstorm and @JHedden confirm the scheduled operation window works for you both, thanks!

• JHedden added a comment.Oct 16 2019, 1:31 PM
In T235627#5579254, @aborrero wrote:

Please @Bstorm and @JHedden confirm the scheduled operation window works for you both, thanks!

2019-10-28 14:30 UTC works for me. Thanks

• Bstorm added a comment.Oct 16 2019, 4:37 PM

If I set an alarm, I can be dressed and at a computer by then. I just have to set a reminder for the day before (am in PDT).

aborrero added a comment.Oct 17 2019, 8:50 AM

cool thanks! I will work on the operation steps soon for you to review.

aborrero added a comment.Oct 21 2019, 10:19 AM

Operation announcement sent: https://lists.wikimedia.org/pipermail/cloud-announce/2019-October/000226.html

aborrero updated the task description. (Show Details)Oct 21 2019, 10:19 AM
aborrero added a comment.Oct 21 2019, 10:35 AM

Proposed operation steps:

  • downtime monitoring, etc
  • disable puppet in tools-proxy-03/tools-proxy-04
  • change role and hiera keys in the puppet section in horizon for the tools-proxy prefix. We need to delete the role role::toollabs::proxy and add the new one role::wmcs::toolforge::proxy.
  • create 2 new VMs: tools-proxy-05 and tools-proxy-06 using Debian Buster as base image.
  • wait for puppet to complete on the new VMs.
  • ensure Redis data is replicated into the new VMs (read only though)
  • refresh hiera keys that specify the active proxy (Hiera:tools and horizon)
  • reallocate the floating IP 185.15.56.5 from tools-proxy-03 to tools-proxy-05.
  • run puppet everywhere in toolforge
  • check kube2proxy is active in tools-proxy-05 (and happy).
  • Redis is r/w in tools-proxy-05.
  • Check that everything else is working (webservices, etc)
  • Shutdown or delete old VMs tools-proxy-03 and tools-proxy-04.
  • done.
Stashbot added a comment.Oct 28 2019, 2:34 PM

Mentioned in SAL (#wikimedia-cloud) [2019-10-28T14:34:31Z] <arturo> icinga downtime toolschecker for 1h (T235627)

Stashbot added a comment.Oct 28 2019, 2:42 PM

Mentioned in SAL (#wikimedia-cloud) [2019-10-28T14:42:08Z] <arturo> deleted role::toollabs::proxy from the tools-proxy puppet profile (T235627)

Stashbot added a comment.Oct 28 2019, 2:43 PM

Mentioned in SAL (#wikimedia-cloud) [2019-10-28T14:43:03Z] <arturo> adding role::wmcs::toolforge::proxy to the tools-proxy puppet prefix (T235627)

Stashbot added a comment.Oct 28 2019, 2:45 PM

Mentioned in SAL (#wikimedia-cloud) [2019-10-28T14:45:26Z] <arturo> created VMs tools-proxy-05 and tools-proxy-06 (T235627)

Stashbot added a comment.Oct 28 2019, 2:58 PM

Mentioned in SAL (#wikimedia-cloud) [2019-10-28T14:58:55Z] <arturo> added webproxy security group to tools-proxy-05 and tools-proxy-06 (T235627)

Stashbot added a comment.Oct 28 2019, 3:14 PM

Mentioned in SAL (#wikimedia-cloud) [2019-10-28T15:14:43Z] <arturo> refresh hiera to use tools-proxy-05 as active proxy T235627

Stashbot added a comment.Oct 28 2019, 3:16 PM

Mentioned in SAL (#wikimedia-cloud) [2019-10-28T15:16:47Z] <arturo> tools-proxy-05 has now the 185.15.56.5 floating IP as active proxy T235627

Stashbot added a comment.Oct 28 2019, 3:54 PM

Mentioned in SAL (#wikimedia-cloud) [2019-10-28T15:54:29Z] <arturo> shutting down tools-proxy-03 T235627

Stashbot added a comment.Oct 28 2019, 3:54 PM

Mentioned in SAL (#wikimedia-cloud) [2019-10-28T15:54:57Z] <arturo> tools-proxy-05 has now the 185.15.56.11 floating IP as active proxy. Old one 185.15.56.6 has been freed T235627

• JHedden mentioned this in T236703: OpenStack server list shows outdated floating ip information.Oct 28 2019, 3:55 PM
aborrero closed this task as Resolved.Oct 28 2019, 4:03 PM

This has been done.

bd808 added a parent task: T236565: "tools" Cloud VPS project jessie deprecation.Oct 28 2019, 4:08 PM
gerritbot added a comment.Oct 28 2019, 4:16 PM

Change 546640 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: delete unused proxy code

https://gerrit.wikimedia.org/r/546640

gerritbot added a project: Patch-For-Review.Oct 28 2019, 4:16 PM
• Bstorm awarded a token.Oct 28 2019, 4:17 PM
bd808 mentioned this in T236565: "tools" Cloud VPS project jessie deprecation.Oct 28 2019, 4:23 PM
gerritbot added a comment.Oct 28 2019, 4:28 PM

Change 546640 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toollabs: delete unused proxy code

https://gerrit.wikimedia.org/r/546640

Maintenance_bot removed a project: Patch-For-Review.Oct 28 2019, 5:11 PM
zhuyifei1999 subscribed.Oct 28 2019, 9:48 PM

No idea how it used to work (T56052 is old), but zhuyifei1999@tools-sgebastion-08: ~$ curl tools.wmflabs.org now hangs.

• Bstorm reopened this task as Open.Oct 28 2019, 9:55 PM

I still see this on my laptop, which may not be a coincidence. I cannot reach any tools since we changed (even on a VPN so far). DNS resolution works correctly.

• Bstorm added a comment.EditedOct 28 2019, 10:00 PM

To demonstrate something is weird:

[bstorm@icinga1001]:~ $ curl tools.wmflabs.org
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.14.2</center>
</body>

It works from prod hosts but not cloud (and my local network for some reason).

Krenair subscribed.Oct 28 2019, 10:09 PM

I don't know about your local network but it breaking specifically within cloud may indicate a labsaliaser problem - resolving tools.wmflabs.org should give the internal IP of tools-proxy-05?

gerritbot added a comment.Oct 28 2019, 10:23 PM

Change 546755 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Revert "cloudvps: ignore stderr in labs-ip-alias-dump.py"

https://gerrit.wikimedia.org/r/546755

gerritbot added a project: Patch-For-Review.Oct 28 2019, 10:23 PM
• Bstorm added a comment.Oct 28 2019, 10:52 PM

As for my local laptop, it was a random /etc/hosts entry I had from some old troubleshooting.

gerritbot added a comment.Oct 28 2019, 10:53 PM

Change 546756 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Fix labsaliaser script to be executable

https://gerrit.wikimedia.org/r/546756

Stashbot added a comment.Oct 28 2019, 10:55 PM

Mentioned in SAL (#wikimedia-cloud) [2019-10-28T22:55:54Z] <jeh> run labs-ip-alias-dump on cloudservices1003 and cloudservices1004 T235627

gerritbot added a comment.Oct 29 2019, 1:47 AM

Change 546756 merged by Andrew Bogott:
[operations/puppet@production] Fix labsaliaser script to be executable

https://gerrit.wikimedia.org/r/546756

gerritbot added a comment.Oct 29 2019, 1:48 AM

Change 546755 merged by Andrew Bogott:
[operations/puppet@production] Revert "cloudvps: ignore stderr in labs-ip-alias-dump.py"

https://gerrit.wikimedia.org/r/546755

Maintenance_bot removed a project: Patch-For-Review.Oct 29 2019, 2:10 AM
Stashbot added a comment.Oct 29 2019, 10:07 AM

Mentioned in SAL (#wikimedia-cloud) [2019-10-29T10:07:31Z] <arturo> deleting old jessie VMs tools-proxy-03/04 T235627

aborrero closed this task as Resolved.Oct 29 2019, 10:08 AM

Thanks everyone for the followup with the split DNS situation. Closing task again now.

bd808 mentioned this in T237080: Toolviews data loading from Toolforge front proxy access log stopped on 2019-10-28.Nov 1 2019, 2:08 AM
Krenair mentioned this in T237150: paws-public.wmflabs.org returns 502 Bad Gateway.Nov 2 2019, 12:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL