Page MenuHomePhabricator

paws-deploy-hook is unreachable
Open, NormalPublic

Description

There are some A records that point to the PAWS k8s master (208.80.155.224) and are not working:

  • jenkins.paws.tools.wmflabs.org.
  • paws-deploy-hook.tools.wmflabs.org.
  • paws-jenkins.tools.wmflabs.org.
  • paws.tools.wmflabs.org.
$ curl -vL http://paws-deploy-hook.tools.wmflabs.org
* Rebuilt URL to: http://paws-deploy-hook.tools.wmflabs.org/
*   Trying 208.80.155.224...
* TCP_NODELAY set
* connect to 208.80.155.224 port 80 failed: Connection refused
* Failed to connect to paws-deploy-hook.tools.wmflabs.org port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to paws-deploy-hook.tools.wmflabs.org port 80: Connection refused

$ curl -vL https://paws-deploy-hook.tools.wmflabs.org
* Rebuilt URL to: https://paws-deploy-hook.tools.wmflabs.org/
*   Trying 208.80.155.224...
* TCP_NODELAY set
* connect to 208.80.155.224 port 443 failed: Connection refused
* Failed to connect to paws-deploy-hook.tools.wmflabs.org port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to paws-deploy-hook.tools.wmflabs.org port 443: Connection refused

Related Objects

Event Timeline

GTirloni triaged this task as Normal priority.Mar 15 2019, 2:43 AM
GTirloni created this task.

I've added ports 80/443 to the paws-master security group but that's not the problem. There is nothing actually listening on those ports on tools-paws-master-01.

Per T195217 the configuration should be a webproxy pointing from https://paws-deploy-hook.tools.wmflabs.org to tools-paws-master-01 at 32612. No idea why it isn't the case currently or how it worked for the past months prior to the region move for PAWS.

At any rate, seems to me the easiest route is to move the paws-deploy-hook.tools.wmflabs.org dns record to point to paws-proxy-02 add a new server entry in the nginx there, add it to certbot (or whatever is renewing the certs), and point it to tools-paws-master-01 at 32612.

@GTirloni I can do most of it, but the dns record move will need tools admin power.

The other records can be safely deleted:

  • jenkins.paws.tools.wmflabs.org.
  • paws-jenkins.tools.wmflabs.org.
  • paws.tools.wmflabs.org.

Created the nginx server block, we now need to move the paws-deploy-hook.tools.wmflabs.org dns record to point to paws-proxy-02. Then we can get a certificate for it with certbot --nginx -d paws-deploy-hook.tools.wmflabs.org

Deleted old records, pointed paws-deploy-hook to paws-proxy-02 and generated certificate.

Created paws-beta.wmflabs.org pointing to paws-proxy-02.

GTirloni removed a subscriber: GTirloni.Mar 23 2019, 8:45 PM

Mentioned in SAL (#wikimedia-cloud) [2019-05-18T11:13:26Z] <chicocvenancio> point paws-proxy-02 to tools-paws-worker-1006 on paws-deploy-hook hostname (T218380)