Page MenuHomePhabricator

Migrate tools.wmflabs.org to https only (and set HSTS)
Open, LowPublic

Tokens
"Like" token, awarded by Steinsplitter."Like" token, awarded by Bawolff."Baby Tequila" token, awarded by Liuxinyu970226."Love" token, awarded by Framawiki."Baby Tequila" token, awarded by tom29739.
Assigned To
None
Authored By
yuvipanda, Jun 14 2015

Description

In line with switching over everything else.

Should be fairly simple to do.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Should be fairly simple to do.

I doubt this is possible. Several tools do not support https.

Restricted Application added a project: Cloud-Services. · View Herald TranscriptJun 18 2015, 4:57 AM
valhallasw triaged this task as Low priority.Jul 2 2015, 7:40 PM
valhallasw moved this task from Triage to Backlog on the Toolforge board.
valhallasw added a subscriber: valhallasw.

Change 218117 merged by BBlack:
dynamicproxy: Allow proxies to be https only

https://gerrit.wikimedia.org/r/218117

Restricted Application added a project: Operations. · View Herald TranscriptFeb 23 2016, 6:12 PM
Andrew claimed this task.Apr 18 2016, 6:07 PM

Should be fairly simple to do.

I doubt this is possible. Several tools do not support https.

Webservices don't directly interact with HTTPS - it is the proxy that does.

BBlack moved this task from Triage to BadHerald on the Traffic board.Oct 4 2016, 1:29 PM
abian added a subscriber: abian.Oct 10 2016, 11:23 AM
Andrew removed Andrew as the assignee of this task.Oct 17 2016, 5:52 PM
Abbe98 added a subscriber: Abbe98.Nov 27 2016, 1:17 AM

I've been using HTTPS Everywhere for a long time. This extension has a rule for tools.wmflabs.org, ensuring that all my connections with the subdomain use HTTPS, and I haven't experienced any issues for now.

This is an important task. I think that we should define a deadline and contact all the developers of Tool Labs to explain that, if their tools only support HTTP, they could stop running in X months if they aren't adapted.

An HTTPS-only Tool Labs could be a nice point in our wishlist for 2017, don't you think?

I really think we could just flip the switch at the ingress proxy and then deal with the fallout. Mixed content warnings/errors are really all that I can think of that may come up. As @yuvipanda pointed out in T102367#2214936, TLS termination happens in front of the tools themselves.

I really think we could just flip the switch at the ingress proxy and then deal with the fallout. Mixed content warnings/errors are really all that I can think of that may come up. As @yuvipanda pointed out in T102367#2214936, TLS termination happens in front of the tools themselves.

The place where this breaks is request cycles that start with an HTTP POST. There is no mechanism in the HTTP protocol to redirect a POST.

We could do the same thing as what we did in prod. Allow HTTP POST for a while, then make a percentage of requests fail, and then finally make them all fail.

scfc added a subscriber: scfc.Feb 13 2017, 10:16 AM

I really think we could just flip the switch at the ingress proxy and then deal with the fallout. Mixed content warnings/errors are really all that I can think of that may come up. As @yuvipanda pointed out in T102367#2214936, TLS termination happens in front of the tools themselves.

We could do the same thing as what we did in prod. Allow HTTP POST for a while, then make a percentage of requests fail, and then finally make them all fail.

Is there any resistance to redirecting GET requests from http to https at the proxy?

Is there any resistance to redirecting GET requests from http to https at the proxy?

Long overdue.

Framawiki added a subscriber: Framawiki.

Mentioned in SAL (#wikimedia-cloud) [2017-10-15T20:15:18Z] <bd808> Updated tool-admin-web from 83d3174 to 9994980 (Redirect http -> https & set STS header with 1day duration) T102367

bd808 added a comment.Oct 15 2017, 8:20 PM

What you can do to help modern browsers, though, without taking the redirect and/or POST-breakage risks, is universally add a long-duration HSTS header so that they lock onto HTTPS if they've ever visited once via HTTPS.

I have added a PHP based redirect from http -> https into https://tools.wmflabs.org/admin/tool/admin. I have also added a logic to that app that will set Strict-Transport-Security: max-age:86400; includeSubDomains; preload on all responses. This will be a small experiment towards the idea of adding an HSTS to all dynamicproxy responses.

Be careful with preload. It's only purpose is to signal to the Chromium list maintainers that it's ok to you preload you (making HSTS more-or-less permanent), and once you're emitting anyone can submit your domain for preload inclusion.

Be careful with preload. It's only purpose is to signal to the Chromium list maintainers that it's ok to you preload you (making HSTS more-or-less permanent), and once you're emitting anyone can submit your domain for preload inclusion.

Yikes. I removed that flag from the header.

If I have my tool set a strict-transport-security: max-age=86400 header, will that impact other tools as well since they're on the same subdomain? Or will it just affect my tool?

Krenair added a subscriber: Krenair.EditedDec 30 2017, 2:54 AM

If I have my tool set a strict-transport-security: max-age=86400 header, will that impact other tools as well since they're on the same subdomain? Or will it just affect my tool?

I would think it will break other tools that don't support HTTPS. We should probably add this header to proxy_hide_header in modules/dynamicproxy/templates/urlproxy.conf until we're ready to start using HSTS across tools.wmflabs.org.
Long-term solution to prevent that class of problems: T125589

Edit: Turns out this was done above anyway... I'd still be cautious and to have a go at T128409 first

bd808 added a comment.Dec 30 2017, 3:49 AM

If I have my tool set a strict-transport-security: max-age=86400 header, will that impact other tools as well since they're on the same subdomain? Or will it just affect my tool?

It will effect all of tools.wmflabs.org, but ... I turned this header on for the admin tool in October (T102367#3686094) and we have not had any widespread reports of problems caused by it. I think we really can start setting this header globally at the proxy and also redirecting GET requests from http to https. We should not redirect non-HTTPS POST requests as the behavior of that is undefined for clients (some detail in T128409#2233337).

Edit: Turns out this was done above anyway... I'd still be cautious and to have a go at T128409 first

I think I tend to agree with @scfc's comment in T128409#2073647 that we really can't reliably certify that everything will work over HTTPS before trying to switch. We can however be ready to respond to reports of breakage by helping tool maintainers update their code. The most common causes I can think of would be (a) hardcoded http protocols on URLs and (b) frameworks generating absolute links which are not respecting the X-Forwarded-Proto header sent by the proxy. We should try to document how to fix (b) for common frameworks before flipping this switch globally.

HSTS should help with (a)

Change 432935 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] toolforge: Redirect GET & HEAD to https

https://gerrit.wikimedia.org/r/432935

Change 481979 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] toolforge: Redirect tools-static to https

https://gerrit.wikimedia.org/r/481979

Change 481979 merged by Andrew Bogott:
[operations/puppet@production] toolforge: Redirect tools-static to https

https://gerrit.wikimedia.org/r/481979

Change 432935 merged by Andrew Bogott:
[operations/puppet@production] toolforge: Redirect GET & HEAD to https

https://gerrit.wikimedia.org/r/432935

Change 482142 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] toolforge: Redirect GET & HEAD to https (take 2)

https://gerrit.wikimedia.org/r/482142

Change 482142 merged by Andrew Bogott:
[operations/puppet@production] toolforge: Redirect GET & HEAD to https (take 2)

https://gerrit.wikimedia.org/r/482142

Currently tools.wmflabs.org is violating RFC 6797 section 7.2 by sending the HSTS header over HTTP:

An HSTS Host MUST NOT include the STS header field in HTTP responses conveyed over non-secure transport.

$ curl -v http://tools.wmflabs.org
* Rebuilt URL to: http://tools.wmflabs.org/
*   Trying 185.15.56.5...
* TCP_NODELAY set
* Connected to tools.wmflabs.org (185.15.56.5) port 80 (#0)
> GET / HTTP/1.1
> Host: tools.wmflabs.org
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.13.6
< Date: Tue, 26 Mar 2019 06:13:57 GMT
< Content-Type: text/html
< Content-Length: 185
< Connection: keep-alive
< Location: https://tools.wmflabs.org/
< Strict-Transport-Security: max-age=86400

It's also violating RFC 6797 section 7.1 by sending the HSTS header twice over HTTPS:

If an STS header field is included, the HSTS Host MUST include only one such header field.

curl --http1.1 -v https://tools.wmflabs.org -o /dev/null
* Rebuilt URL to: https://tools.wmflabs.org/
Trying 185.15.56.5...
* TCP_NODELAY set
* Connected to tools.wmflabs.org (185.15.56.5) port 443 (#0)
[output omitted]
> GET / HTTP/1.1
> Host: tools.wmflabs.org
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.13.6
< Date: Tue, 26 Mar 2019 06:22:52 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 8719
< Connection: keep-alive
< Set-Cookie: _s=eclunr0i35p2nebqbnq70venbf; path=/; HttpOnly
< Vary: Cookie
< X-Frame-Options: DENY
< Content-Security-Policy: default-src 'self' https://tools-static.wmflabs.org; child-src 'none'; object-src 'none'; img-src 'self' data: https://tools-static.wmflabs.org
< Strict-Transport-Security: max-age=86400; includeSubDomains
< Strict-Transport-Security: max-age=86400

I assume this is something in our nginx proxy, right? @Vgutierrez Could you please help us review the config?

I assume this is something in our nginx proxy, right? @Vgutierrez Could you please help us review the config?

so from a quick review of https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/dynamicproxy/templates/domainproxy.conf, line 56 is responsible of sending the HSTS header twice over HTTPS as the HSTS header is also set as part of line 39.
I assume that line 56 again is responsible for sending the header over HTTP as the same virtualhost is being used for serving http and https traffic

bd808 added a comment.Mar 27 2019, 8:06 PM

Live config from tools-proxy-03.tools.eqiad.wmflabs shows only the add_header Strict-Transport-Security "max-age=86400"; in the nginx config. Looking at the logic of how I added that, I think I assumed (incorrectly apparently) that the return 301 https://$host$request_uri; redirect would stop all further processing of the request. It should be possible to adjust the logic so that this header is only sent when the connection is using TLS.

The Strict-Transport-Security: max-age=86400; includeSubDomains header is actually being sent by the "admin" application (R1922:7be2a095ce41: Strict-Transport-Security: remove "preload" flag). This is a leftover from the experiments prior to adding this header at the proxy level. We should remove this from the app. We should probably also add a proxy_hide_header Strict-Transport-Security; config line to the proxy to keep any other tool from sending arbitrary STS headers to the client.

Mentioned in SAL (#wikimedia-cloud) [2019-03-27T21:49:20Z] <bd808> Updated to a03551b (Remove STS header and http->https redirect) T102367

Change 499669 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/puppet@production] dynamicproxy: Prevent STS header from non-TLS connections

https://gerrit.wikimedia.org/r/499669

Change 499669 merged by Bstorm:
[operations/puppet@production] dynamicproxy: Prevent STS header from non-TLS connections

https://gerrit.wikimedia.org/r/499669

bd808 added a comment.Apr 8 2019, 11:14 PM

Currently tools.wmflabs.org is violating RFC 6797 section 7.2 by sending the HSTS header over HTTP:

My attempt at fixing this in https://gerrit.wikimedia.org/r/499669 was reverted in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/502308/.

Reasons for the revert:

  1. @bd808 did not actually test the nginx config changes and they were not syntactically correct.
  2. @bd808 did not remember that this exact same syntax problem is why we have the header sent over http in the first place. (See https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/432935/ and its replacement in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/482142/).

We can't use the if construct in nginx config to guard the add_header setting. This in turn means that the entire nginx config used by domain-proxy and url-proxy will need to be rewritten if we want to ensure that the STS header is only presented to TLS secured connections. The current config for both mixes http and https handling in the same server config block leaving us no way to decide whether or not to emit the STS header. Splitting http and https into separate server blocks and only placing the STS header in the https server will fix that problem. It will require some mechanism to duplicate almost all of the other logic in the current config as long as we continue to support the POST loophole however.