Page MenuHomePhabricator

paws-deploy-hook is unreachable
Closed, ResolvedPublic

Description

There are some A records that point to the PAWS k8s master (208.80.155.224) and are not working:

  • jenkins.paws.tools.wmflabs.org.
  • paws-deploy-hook.tools.wmflabs.org.
  • paws-jenkins.tools.wmflabs.org.
  • paws.tools.wmflabs.org.
$ curl -vL http://paws-deploy-hook.tools.wmflabs.org
* Rebuilt URL to: http://paws-deploy-hook.tools.wmflabs.org/
*   Trying 208.80.155.224...
* TCP_NODELAY set
* connect to 208.80.155.224 port 80 failed: Connection refused
* Failed to connect to paws-deploy-hook.tools.wmflabs.org port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to paws-deploy-hook.tools.wmflabs.org port 80: Connection refused

$ curl -vL https://paws-deploy-hook.tools.wmflabs.org
* Rebuilt URL to: https://paws-deploy-hook.tools.wmflabs.org/
*   Trying 208.80.155.224...
* TCP_NODELAY set
* connect to 208.80.155.224 port 443 failed: Connection refused
* Failed to connect to paws-deploy-hook.tools.wmflabs.org port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to paws-deploy-hook.tools.wmflabs.org port 443: Connection refused

Event Timeline

GTirloni triaged this task as Medium priority.Mar 15 2019, 2:43 AM
GTirloni created this task.

I've added ports 80/443 to the paws-master security group but that's not the problem. There is nothing actually listening on those ports on tools-paws-master-01.

Per T195217 the configuration should be a webproxy pointing from https://paws-deploy-hook.tools.wmflabs.org to tools-paws-master-01 at 32612. No idea why it isn't the case currently or how it worked for the past months prior to the region move for PAWS.

At any rate, seems to me the easiest route is to move the paws-deploy-hook.tools.wmflabs.org dns record to point to paws-proxy-02 add a new server entry in the nginx there, add it to certbot (or whatever is renewing the certs), and point it to tools-paws-master-01 at 32612.

@GTirloni I can do most of it, but the dns record move will need tools admin power.

The other records can be safely deleted:

  • jenkins.paws.tools.wmflabs.org.
  • paws-jenkins.tools.wmflabs.org.
  • paws.tools.wmflabs.org.

Created the nginx server block, we now need to move the paws-deploy-hook.tools.wmflabs.org dns record to point to paws-proxy-02. Then we can get a certificate for it with certbot --nginx -d paws-deploy-hook.tools.wmflabs.org

Deleted old records, pointed paws-deploy-hook to paws-proxy-02 and generated certificate.

Created paws-beta.wmflabs.org pointing to paws-proxy-02.

GTirloni removed a subscriber: GTirloni.Mar 23 2019, 8:45 PM

Mentioned in SAL (#wikimedia-cloud) [2019-05-18T11:13:26Z] <chicocvenancio> point paws-proxy-02 to tools-paws-worker-1006 on paws-deploy-hook hostname (T218380)

Bstorm added a subscriber: Bstorm.Apr 15 2020, 11:11 PM

Is this ticket still current @Chicocvenancio ?

No. Automation seems broken due to the movement in repo and to travis.com. Probably all that is necessary is to regenerate a .travis.yml with encrypted secrets.

Chicocvenancio closed this task as Resolved.May 26 2020, 11:46 PM
Chicocvenancio claimed this task.

@Chicocvenancio Good to hear, overall, I haven't enabled the repo for travis yet (nothing stops it from working as is with the current file) because too many balls are in the air to automate it from there. I don't want to automatically break the existing cluster. The travis config points back at the old repo location right now...and that's just fine until a couple more pieces are done! Thanks!