Page MenuHomePhabricator

Setup a dedicated HTTPS terminating proxy for maps project
Closed, ResolvedPublic

Description

The maps project uses a number of https reverse proxy service names for its tile servers. This is fine *most* of the time, but occasionally the rate of requests for tiles rises to a point where the current globally shared reverse proxy becomes saturated. This is not great for the maps tile server users, but it is even worse for the large number of other http endpoints in other projects that share the same reverse proxy.

It would be nice to separate the tile server's reverse proxy from the general purpose reverse proxy to isolate overload failures. Because we want TLS for this, the new solution would need to either use Let's Encrypt to get certificates if the proxy lives in the maps project, or the proxy would need to live in the project-proxy project where we have the existing wildcard certificate. Single points of failure are not fun, so a replacement should have redundant nodes. Ideally this redundancy would move traffic between 2 proxy instances automatically based on some health monitoring. The current shared proxies are manually failed over so that would work here too, but it might also be a good opportunity to experiment with some automatic HA solution that could later be propagated to domainproxy & urlproxy.

Event Timeline

bd808 created this task.Sep 26 2019, 10:34 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 26 2019, 10:34 PM
bd808 triaged this task as Medium priority.Oct 4 2019, 9:55 PM

Discussed in the 2019-10-01 WMCS team meeting. Consensus was that this would be a nice thing to have, but that it is not high enough priority to displace current work by the team so it will wait for now. Building this could be used as a prototype for a newer proxy setup for more general use.

bd808 renamed this task from Consider setup of a dedicated HTTPS terminating proxy for maps project to Setup a dedicated HTTPS terminating proxy for maps project.Oct 4 2019, 9:55 PM
bd808 moved this task from Needs discussion to Soon! on the cloud-services-team (Kanban) board.
bd808 moved this task from Backlog to Dynamicproxy on the Cloud-VPS board.Nov 10 2019, 11:32 PM
bd808 added a comment.Apr 30 2020, 7:32 PM

Related: T161256: multi-component wmflabs.org subdomains doesn't work under simple wildcard TLS cert

So a dedicated reverse proxy with proper LE certs could kill that one off too I think.

The long-term solution for this is T252721 -- in the meantime I'm going to split maps into its own proxy with the existing (semi-broken) TLS support.

Change 596328 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add profile::wmcs::proxy::static

https://gerrit.wikimedia.org/r/596328

Change 596328 merged by Andrew Bogott:
[operations/puppet@production] Add profile::wmcs::proxy::static

https://gerrit.wikimedia.org/r/596328

I now have a pair of proxies (active/passive) in the 'project-proxy' project: maps-proxy-01 and maps-proxy-02. Proxies are configured via the 'maps-proxy' hiera prefix, like so:

profile::wmcs::proxy::static::proxy_mappings:
  0.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  1.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  2.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  3.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  4.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  5.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  6.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  7.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  8.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  a.tiles.wmflabs.org: maps-tiles1.maps.eqiad1.wikimedia.cloud
  label.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  maps.wmflabs.org: maps-tiles1.maps.eqiad1.wikimedia.cloud
  tiles.wmflabs.org: maps-tiles1.maps.eqiad1.wikimedia.cloud
  warper.wmflabs.org: maps-warper3.maps.eqiad1.wikimedia.cloud
  wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  wma0.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  wma1.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  wma2.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  wma3.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  wma4.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  wma5.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  wma6.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  wma7.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud

It's not ideal that updating this will require access to the project-proxy project, but for now the proxy needs to live there because that's where the ssl certs are.

Also, note that the three-dot hostnames (e.g. 6.wma.wmflabs.org) don't work with ssl -- that's the same as it was before, and won't be fixed until we have a better LE setup here.

The next step is to switch DNS to actually point to this thing. I'm planning to move all of the DNS ownership into the 'maps' project, as currently all those domains are in the wmflabsdotorg project.

Andrew claimed this task.May 14 2020, 2:03 PM
Andrew added a subscriber: TheDJ.

a.tiles.wmflabs.org: maps-tiles1.maps.eqiad1.wikimedia.cloud

where are b and c ?

a.tiles.wmflabs.org: maps-tiles1.maps.eqiad1.wikimedia.cloud

where are b and c ?

You're right! Amended list is:

profile::wmcs::proxy::static::proxy_mappings:
  0.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  1.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  2.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  3.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  4.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  5.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  6.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  7.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  a.tiles.wmflabs.org: maps-tiles1.maps.eqiad1.wikimedia.cloud
  b.tiles.wmflabs.org: maps-tiles1.maps.eqiad1.wikimedia.cloud
  c.tiles.wmflabs.org: maps-tiles1.maps.eqiad1.wikimedia.cloud
  label.wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  maps.wmflabs.org: maps-tiles1.maps.eqiad1.wikimedia.cloud
  tiles.wmflabs.org: maps-tiles1.maps.eqiad1.wikimedia.cloud
  warper.wmflabs.org: maps-warper3.maps.eqiad1.wikimedia.cloud
  wma.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  wma0.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  wma1.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  wma2.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  wma3.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  wma4.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  wma5.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  wma6.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud
  wma7.wmflabs.org: maps-wma.maps.eqiad1.wikimedia.cloud

Change 596884 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] profile::wmcs::proxy::static: pass on $host to the backend

https://gerrit.wikimedia.org/r/596884

Change 596884 merged by Andrew Bogott:
[operations/puppet@production] profile::wmcs::proxy::static: pass on $host to the backend

https://gerrit.wikimedia.org/r/596884

Change 596885 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] profile::wmcs::proxy::static: blank out x-forwarded-for and proxy headers

https://gerrit.wikimedia.org/r/596885

Change 596885 merged by Andrew Bogott:
[operations/puppet@production] profile::wmcs::proxy::static: blank out x-forwarded-for and proxy headers

https://gerrit.wikimedia.org/r/596885

This seems to be working -- we'll see if anyone complains! Docs are at

https://wikitech.wikimedia.org/wiki/OSM_Tileserver#DSN%2FWeb_proxies

Andrew closed this task as Resolved.May 17 2020, 10:01 PM