Page MenuHomePhabricator

Move cache text cluster from nginx to ats-tls
Open, NormalPublic

Description

Current status:

  • eqiad:
    • cp1075
    • cp1077
    • cp1079
    • cp1081
    • cp1083
    • cp1085
    • cp1087
    • cp1089
  • codfw:
    • cp2001
    • cp2004
    • cp2006
    • cp2007
    • cp2012
    • cp2013
    • cp2016
    • cp2019
    • cp2023
  • esams:
    • cp3030
    • cp3032
    • cp3033
    • cp3040
    • cp3041
    • cp3042
    • cp3043
  • ulsfo:
    • cp4027
    • cp4028
    • cp4029
    • cp4030
    • cp4031
    • cp4032
  • eqsin:
    • cp5007
    • cp5008
    • cp5009
    • cp5010
    • cp5011
    • cp5012

Event Timeline

Vgutierrez triaged this task as Normal priority.Aug 30 2019, 6:02 AM
Vgutierrez created this task.
Vgutierrez moved this task from Triage to TLS on the Traffic board.

Change 533379 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Add known websocket endpoints to the TLS instance mapping rules

https://gerrit.wikimedia.org/r/533379

Change 533379 merged by Vgutierrez:
[operations/puppet@production] ATS: Add known websocket endpoints to the TLS instance mapping rules

https://gerrit.wikimedia.org/r/533379

Change 537993 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] cache: Deploy ats-tls in the text cluster

https://gerrit.wikimedia.org/r/537993

Change 537994 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] hiera: Move nginx from port 443 to 4443 on cp5007

https://gerrit.wikimedia.org/r/537994

Change 537995 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] hiera: Move ats-tls from port 8443 to 443 on cp5007

https://gerrit.wikimedia.org/r/537995

Change 538231 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Provide HTTPS check

https://gerrit.wikimedia.org/r/538231

Change 538231 merged by Vgutierrez:
[operations/puppet@production] ATS: Provide HTTPS check

https://gerrit.wikimedia.org/r/538231

Change 537993 merged by Vgutierrez:
[operations/puppet@production] cache: Deploy ats-tls in the text cluster

https://gerrit.wikimedia.org/r/537993

Change 538471 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Move libhwloc5 pin to trafficserver class

https://gerrit.wikimedia.org/r/538471

Change 538471 merged by Vgutierrez:
[operations/puppet@production] ATS: Move libhwloc5 pin to trafficserver class

https://gerrit.wikimedia.org/r/538471

Change 538587 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] hiera: Move nginx from port 443 to 4443 on cp4027

https://gerrit.wikimedia.org/r/538587

Change 537995 abandoned by Vgutierrez:
hiera: Move ats-tls from port 8443 to 443 on cp4027

https://gerrit.wikimedia.org/r/537995

Change 538589 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] hiera: Move ats-tls from port 8443 to 443 on cp4027

https://gerrit.wikimedia.org/r/538589

Mentioned in SAL (#wikimedia-operations) [2019-09-23T11:42:03Z] <vgutierrez> switching cp4027 from nginx to ats-tls - T231627

Change 538587 merged by Vgutierrez:
[operations/puppet@production] hiera: Move nginx from port 443 to 4443 on cp4027

https://gerrit.wikimedia.org/r/538587

Change 538589 merged by Vgutierrez:
[operations/puppet@production] hiera: Move ats-tls from port 8443 to 443 on cp4027

https://gerrit.wikimedia.org/r/538589

CDanis added a subscriber: CDanis.Mon, Sep 23, 8:08 PM

I depooled cp4027 today when T233657: Users editing from 127.0.0.1 (due to experimenting with ATS terminating TLS) surfaced.

Gonna guess that the ATS TLS termination is missing a special client IP header that nginx knows to insert?

Thanks for the depool @CDanis, so it looks like a combination of things: nginx/ATS set X-Client-IP and X-Forwarded-For. The behavior for X-Client-IP is the same and there is no problem there. But for X-Forwarded-For nginx identifies itself with the IP of the NIC, so 10.128.0.127 for cp4027 and ATS with 127.0.0.1 and for some reason mediawiki is expecting a 10.0.0.0/8 IP instead of stopping at the first non-routable IP address.

Vgutierrez updated the task description. (Show Details)Thu, Sep 26, 10:41 AM

Change 539300 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] hiera: Move ats-tls from port 8443 to 443 on cp5007

https://gerrit.wikimedia.org/r/539300

Mentioned in SAL (#wikimedia-operations) [2019-09-26T10:48:11Z] <vgutierrez> switching from nginx to ats-tls on cp5007 - T231627

Change 537994 merged by Vgutierrez:
[operations/puppet@production] hiera: Move nginx from port 443 to 4443 on cp5007

https://gerrit.wikimedia.org/r/537994

Change 539300 merged by Vgutierrez:
[operations/puppet@production] hiera: Move ats-tls from port 8443 to 443 on cp5007

https://gerrit.wikimedia.org/r/539300

Vgutierrez updated the task description. (Show Details)Thu, Sep 26, 11:08 AM

Change 539446 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] ATS: Gather metrics regarding parent servers

https://gerrit.wikimedia.org/r/539446

Change 539446 merged by Vgutierrez:
[operations/puppet@production] ATS: Gather metrics regarding parent servers

https://gerrit.wikimedia.org/r/539446

Change 544635 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] hiera: Move nginx from port 443 to 4443 on cp3030

https://gerrit.wikimedia.org/r/544635

Change 544636 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] hiera: Move ats-tls from port 8443 to 443 on cp3030

https://gerrit.wikimedia.org/r/544636

Mentioned in SAL (#wikimedia-operations) [2019-10-21T06:46:21Z] <vgutierrez> Switch from nginx to ats-tls on cp3030 - T231627

Change 544635 merged by Vgutierrez:
[operations/puppet@production] hiera: Move nginx from port 443 to 4443 on cp3030

https://gerrit.wikimedia.org/r/544635

Change 544636 merged by Vgutierrez:
[operations/puppet@production] hiera: Move ats-tls from port 8443 to 443 on cp3030

https://gerrit.wikimedia.org/r/544636

Vgutierrez updated the task description. (Show Details)Mon, Oct 21, 6:56 AM

Change 544648 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] hiera: Move nginx from port 443 to port 4443 on cp2001

https://gerrit.wikimedia.org/r/544648

Change 544649 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] hiera: Move ats-tls from port 8443 to port 443 on cp2001

https://gerrit.wikimedia.org/r/544649

Mentioned in SAL (#wikimedia-operations) [2019-10-21T06:59:56Z] <vgutierrez> Switch from nginx to ats-tls on cp2001 - T231627

Change 544648 merged by Vgutierrez:
[operations/puppet@production] hiera: Move nginx from port 443 to port 4443 on cp2001

https://gerrit.wikimedia.org/r/544648

Change 544649 merged by Vgutierrez:
[operations/puppet@production] hiera: Move ats-tls from port 8443 to port 443 on cp2001

https://gerrit.wikimedia.org/r/544649

Vgutierrez updated the task description. (Show Details)Mon, Oct 21, 7:10 AM

Change 544655 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] hiera: Move nginx from port 443 to 4443 on cp1075

https://gerrit.wikimedia.org/r/544655

Change 544656 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] hiera: Move ats-tls from port 8443 to port 443 on cp1075

https://gerrit.wikimedia.org/r/544656

Mentioned in SAL (#wikimedia-operations) [2019-10-21T07:15:12Z] <vgutierrez> Switch from nginx to ats-tls on cp1075 - T231627

Change 544655 merged by Vgutierrez:
[operations/puppet@production] hiera: Move nginx from port 443 to 4443 on cp1075

https://gerrit.wikimedia.org/r/544655

Change 544656 merged by Vgutierrez:
[operations/puppet@production] hiera: Move ats-tls from port 8443 to port 443 on cp1075

https://gerrit.wikimedia.org/r/544656

Vgutierrez updated the task description. (Show Details)Mon, Oct 21, 7:30 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-21T07:43:02Z] <vgutierrez> Switch from nginx to ats-tls on cp3045 - T231627

Mentioned in SAL (#wikimedia-operations) [2019-10-21T07:57:02Z] <vgutierrez> Switch from nginx to ats-tls on cp3046 - T231627

Mentioned in SAL (#wikimedia-operations) [2019-10-21T08:09:02Z] <vgutierrez> Switch from nginx to ats-tls on cp2018 - T231627

Mentioned in SAL (#wikimedia-operations) [2019-10-21T08:20:28Z] <vgutierrez> Switch from nginx to ats-tls on cp2020 - T231627

Mentioned in SAL (#wikimedia-operations) [2019-10-21T08:34:16Z] <vgutierrez> Switch from nginx to ats-tls on cp2022 - T231627