We need to try migration stuff in toolsbeta.
Best way to do it is by setting up a new proxy in toolsbeta.
We need to try migration stuff in toolsbeta.
Best way to do it is by setting up a new proxy in toolsbeta.
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Restricted Task | |||||
Resolved | Bstorm | T246122 Upgrade the Toolforge Kubernetes cluster to v1.16 | |||
Restricted Task | |||||
Resolved | bd808 | T232536 Toolforge Kubernetes internal API down, causing `webservice` and other tooling to fail | |||
Resolved | Bstorm | T236565 "tools" Cloud VPS project jessie deprecation | |||
Resolved | aborrero | T101651 Set up toolsbeta more fully to help make testing easier | |||
Resolved | Bstorm | T166949 Homedir/UID info breaks after a while in Tools Kubernetes (can't read replica.my.cnf) | |||
Resolved | Bstorm | T246059 Add admin account creation to maintain-kubeusers | |||
Resolved | Bstorm | T154504 Make webservice backend default to kubernetes | |||
Declined | None | T245230 Investigate cpu/ram requests and limits for DaemonSets pods | |||
Resolved | Bstorm | T214513 Deploy and migrate tools to a Kubernetes v1.15 or newer cluster | |||
Resolved | aborrero | T215531 Deploy upgraded Kubernetes to toolsbeta | |||
Resolved | aborrero | T237443 toolsbeta: new k8s: deploy a front proxy (dynamicproxy) |
Change 549072 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: proxy: make https fully optional
Change 549072 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: proxy: make https fully optional
Change 549080 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: toolviews: add default hiera values
Change 549080 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: toolviews: add default hiera values
Mentioned in SAL (#wikimedia-cloud) [2019-11-06T12:44:17Z] <arturo> created VM toolsbeta-proxy-1 (T237443)
This is mostly done.
This is a reply by nginx in the front proxy:
<html> <head><title>503 Service Temporarily Unavailable</title></head> <body bgcolor="white"> <center><h1>503 Service Temporarily Unavailable</h1></center> <hr><center>nginx/1.14.2</center> </body> </html>
This is a reply by nginx-ingress in the new k8s cluster:
aborrero@toolsbeta-puppetmaster-02:~ $ curl toolsbeta.wmflabs.org/openstack-browser <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>openresty/1.15.8.1</center> </body> </html>
This happens because I live-hacked https://gerrit.wikimedia.org/r/c/operations/puppet/+/543135 into toolsbeta puppetmaster, until T234032: Toolforge ingress: create a default landing page for unknown/default URLs (or similar) is figured out.
Had to flip a bunch of hiera switches to get the proxy a bit more functional. kube2proxy is still failing due to some redis thing. Checking that.
Mentioned in SAL (#wikimedia-cloud) [2019-11-06T18:46:19Z] <bstorm_> rebooting toolsbeta-proxy-1 trying to convince redis it is not a read replica T237443
Mentioned in SAL (#wikimedia-cloud) [2019-11-06T18:53:07Z] <bstorm_> launching toolsbeta-proxy-2 on a hunch that the config doesn't work well as a standalone T237443
Mentioned in SAL (#wikimedia-cloud) [2019-11-06T19:09:10Z] <bstorm_> added profile::toolforge::proxies in global hiera to try and figure out why it won't let anything use redis T237443
That did it! I moved the proxy config out of the prefix puppet into the project puppet, added a second node, restarted flannel and switched the master name from a FQDN to hostname. That made it a master, opened flannel etc.
Nov 6 19:11:52 toolsbeta-proxy-1 kube2proxy[20343]: 2019-11-06 19:11:52,268 Service test/test/.* is ADDED
It's behaving like a proxy now.
toolsbeta.test@toolsbeta-sgebastion-04:~$ curl http://toolsbeta.wmflabs.org/test/ Hello World!
Now we can fully test migration! Thanks @aborrero