Page MenuHomePhabricator

toolsbeta: new k8s: deploy a front proxy (dynamicproxy)
Closed, ResolvedPublic

Description

We need to try migration stuff in toolsbeta.

Best way to do it is by setting up a new proxy in toolsbeta.

Event Timeline

Change 549072 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: proxy: make https fully optional

https://gerrit.wikimedia.org/r/549072

Change 549072 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: proxy: make https fully optional

https://gerrit.wikimedia.org/r/549072

Change 549080 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: toolviews: add default hiera values

https://gerrit.wikimedia.org/r/549080

Change 549080 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: toolviews: add default hiera values

https://gerrit.wikimedia.org/r/549080

Mentioned in SAL (#wikimedia-cloud) [2019-11-06T12:44:17Z] <arturo> created VM toolsbeta-proxy-1 (T237443)

This is mostly done.

This is a reply by nginx in the front proxy:

<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body bgcolor="white">
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>

This is a reply by nginx-ingress in the new k8s cluster:

aborrero@toolsbeta-puppetmaster-02:~ $ curl toolsbeta.wmflabs.org/openstack-browser
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>openresty/1.15.8.1</center>
</body>
</html>

This happens because I live-hacked https://gerrit.wikimedia.org/r/c/operations/puppet/+/543135 into toolsbeta puppetmaster, until T234032: Toolforge ingress: create a default landing page for unknown/default URLs (or similar) is figured out.

Had to flip a bunch of hiera switches to get the proxy a bit more functional. kube2proxy is still failing due to some redis thing. Checking that.

Mentioned in SAL (#wikimedia-cloud) [2019-11-06T18:46:19Z] <bstorm_> rebooting toolsbeta-proxy-1 trying to convince redis it is not a read replica T237443

Mentioned in SAL (#wikimedia-cloud) [2019-11-06T18:53:07Z] <bstorm_> launching toolsbeta-proxy-2 on a hunch that the config doesn't work well as a standalone T237443

Mentioned in SAL (#wikimedia-cloud) [2019-11-06T19:09:10Z] <bstorm_> added profile::toolforge::proxies in global hiera to try and figure out why it won't let anything use redis T237443

That did it! I moved the proxy config out of the prefix puppet into the project puppet, added a second node, restarted flannel and switched the master name from a FQDN to hostname. That made it a master, opened flannel etc.

Nov  6 19:11:52 toolsbeta-proxy-1 kube2proxy[20343]: 2019-11-06 19:11:52,268 Service test/test/.* is ADDED

It's behaving like a proxy now.

toolsbeta.test@toolsbeta-sgebastion-04:~$ curl http://toolsbeta.wmflabs.org/test/
Hello World!

Now we can fully test migration! Thanks @aborrero

Thanks for double checking everything and finishing the setup!