Page MenuHomePhabricator

Setup real ssl certs for Beta Cluster using a restricted project
Closed, InvalidPublic

Description

This issue has been biting us a lot recently (where "this issue" == "people's browsers requesting https: beta address and it failing"). Let's fix this finally.

Proposal:
(feel free to edit this description with clarifications/additions)

  1. buy real certs (T50501 and https://rt.wikimedia.org/Ticket/Display.html?id=6116)
    • Still needs cost approval, @greg can work on that
    • glorious future of letsencrypt.org would reduce this cost to $0, but we shouldn't wait on that ("Arriving Summer 2015")
  2. make new labs project, restrict access a lot
  3. just ssl terminate there, proxy back to regular deployment-prep setup

This gets us:

Event Timeline

greg created this task.Nov 25 2014, 7:57 PM
greg raised the priority of this task from to Needs Triage.
greg updated the task description. (Show Details)
greg changed Security from none to None.
greg added subscribers: greg, yuvipanda, hashar and 6 others.
greg triaged this task as Medium priority.Nov 25 2014, 8:19 PM
greg moved this task from To Triage to Backlog on the Beta-Cluster-Infrastructure board.
bd808 added a comment.Nov 25 2014, 8:37 PM

make new labs project, restrict access a lot

The only downside I see for this plan is that it becomes "yet another way that beta is not like production." Other than that I think it should nicely solve the rest of the problems.

A similar but slightly different approach would be to use YuviProxy™ to terminate the ssl for beta. That would keep the private keys in Ops hands and only make the beta/prod difference that the nginx instances in beta never see TLS traffic.

Can't actually use dynamicproxy, since the cert there is just for *.wmflabs.org. Also, this terminator could be much simpler, since all it needs to do is forward it to the existing deployment-prep setup, without any dynamic routing.

I loose track on the rather long T50501: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org , but I commented about using a certificate authority to generate free certs for beta then inject that CA in the browsers used by SauceLabs.

Calm down, what we need is to use the 'Labs CA' certificate authority to generate all the certificates we need, then add that authority as a trusted one in the browsers.
I would do it myself if I had access to the 'Labs CA'.

Ryan Lane echoed:

rlane32 wrote:
Has anyone considered creating a CA (http://pages.cs.wisc.edu/~zmiller/ca-howto/), generating a * cert, and adding that certificate to the browser's trust? Is that a possibility using the selenium service we're using?
I'm still not opposed to getting a proper cert, but that requires some infrastructure and permissions changes in deployment-prep...

I kind of stopped following the bug after that.

There is some doc at: https://github.com/nicolaepetridean/sauceLabsTry/wiki/Test-systems-with-self-signed-certificates

https://web.archive.org/web/20121231011833/http://saucelabs.com/docs/https seems to mention SauceOndemand supports self signed.

And there is some more doc at https://docs.saucelabs.com/reference/test-configuration/#avoiding-the-selenium-proxy which mention:

By default, Sauce routes traffic from all Selenium RC and some WebDriver browsers through the Selenium HTTP proxy server so that HTTPS connections with self-signed certificates work everywhere.

So to me, this task is a dupe of T50501 and we should get a task to investigate usage of Sauce with self signed certificate. Then we can avoid throwing money at some SSL authority and will not have to figure out a policy to avoid having the certs leaked.

My 0.02 €

greg added a comment.Nov 25 2014, 10:48 PM

I loose track on the rather long T50501: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org , but I commented about using a certificate authority to generate free certs for beta then inject that CA in the browsers used by SauceLabs.

Except that only helps browser tests, not individual/random contributors who will only be greeted with a big red "SOMEONE'S TRYING TO HAXOR YOU!" warning from their browser. We have no way of putting an interstitial/modifying that warning.

Self-signed is not the way to go if we want humans and random contributors to use Beta Cluster (we do, and will more).

Consider three things we want beta to have:

  1. Same as prod (including SSL Certs)
  2. sudo for as many people as possible

They're incompatible, since as *few* people as possible have sudo in prod. Right now we've something akin to worst of both worlds - sudo is restricted, and we have no SSL. Doing this the same as prod sacrifices some things from (1) but gains us other things from (1) (ssl), and gives us (2). I'd say worth it.

Can't actually use dynamicproxy, since the cert there is just for *.wmflabs.org. Also, this terminator could be much simpler, since all it needs to do is forward it to the existing deployment-prep setup, without any dynamic routing.

Additionally that will bypass the nginx protocol proxy we have set up on beta which are build using the manifests using in production. Ie beta cluster will vary with production.

I'll also note that from a security perspective, since people without NDA *had* root in the past, it would've been trivial for them to acquire sudo again by, uh, planning ahead when they had root (not that I think anyone would have :)). So while still somewhat solid, without re-generating all the machines we can not be completely sure that there are no root exploits :)

As I said, being liberal with sudo and being closer to prod are conflicting requirements, and we've to pick one or the other :)

In T75919#786302, @greg wrote:

I loose track on the rather long T50501: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org , but I commented about using a certificate authority to generate free certs for beta then inject that CA in the browsers used by SauceLabs.

Except that only helps browser tests, not individual/random contributors who will only be greeted with a big red "SOMEONE'S TRYING TO HAXOR YOU!" warning from their browser. We have no way of putting an interstitial/modifying that warning.
Self-signed is not the way to go if we want humans and random contributors to use Beta Cluster (we do, and will more).

What is the issue with end users? If we use the Labs CA authority to create self-signed certificates for the beta cluster, human would be prompted with the security issue to accept that untrusted CA for the given domain. That is what we had originally. I looked at my browser keychain and I still have:

Cert name: Labs CA
Sha1 fingerprint: 23 63 65 50 93 24 C4 B5 2A BB 24 C6 85 79 BE 3A BE E5 52 9E
Trusted for:
- en.wikipedia.beta.wmflabs.org
- bits.wikipedia.beta.wmflabs.org
- upload.wikimedia.beta.wmflabs.org

We could FAQ it and have humans accept the cert for the beta.wmflabs.org domains. I am pretty sure that is what we used to do and it worked fine. It would be slightly annoying on first use, but I think it is an acceptable annoyance compared to buy ssl certs and managing a policy to keep them really private.

greg added a comment.Nov 26 2014, 3:53 PM
In T75919#786302, @greg wrote:

I loose track on the rather long T50501: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org , but I commented about using a certificate authority to generate free certs for beta then inject that CA in the browsers used by SauceLabs.

Except that only helps browser tests, not individual/random contributors who will only be greeted with a big red "SOMEONE'S TRYING TO HAXOR YOU!" warning from their browser. We have no way of putting an interstitial/modifying that warning.
Self-signed is not the way to go if we want humans and random contributors to use Beta Cluster (we do, and will more).

What is the issue with end users? If we use the Labs CA authority to create self-signed certificates for the beta cluster, human would be prompted with the security issue to accept that untrusted CA for the given domain.

Right, and when someone wants a random user to do a quick user test on Beta Cluster, that's a show stopper.

We could FAQ it and have humans accept the cert for the beta.wmflabs.org domains. I am pretty sure that is what we used to do and it worked fine. It would be slightly annoying on first use, but I think it is an acceptable annoyance compared to buy ssl certs and managing a policy to keep them really private.

And only helps those who know where to look before hand. I want a solution that will cause the least amount of friction to the average drive by contributor as possible.

In T75919#786302, @greg wrote:

I loose track on the rather long T50501: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org , but I commented about using a certificate authority to generate free certs for beta then inject that CA in the browsers used by SauceLabs.

Except that only helps browser tests, not individual/random contributors who will only be greeted with a big red "SOMEONE'S TRYING TO HAXOR YOU!" warning from their browser. We have no way of putting an interstitial/modifying that warning.
Self-signed is not the way to go if we want humans and random contributors to use Beta Cluster (we do, and will more).

What is the issue with end users? If we use the Labs CA authority to create self-signed certificates for the beta cluster, human would be prompted with the security issue to accept that untrusted CA for the given domain. That is what we had originally. I looked at my browser keychain and I still have:

Cert name: Labs CA
Sha1 fingerprint: 23 63 65 50 93 24 C4 B5 2A BB 24 C6 85 79 BE 3A BE E5 52 9E
Trusted for:
- en.wikipedia.beta.wmflabs.org
- bits.wikipedia.beta.wmflabs.org
- upload.wikimedia.beta.wmflabs.org

We could FAQ it and have humans accept the cert for the beta.wmflabs.org domains. I am pretty sure that is what we used to do and it worked fine. It would be slightly annoying on first use, but I think it is an acceptable annoyance compared to buy ssl certs and managing a policy to keep them really private.

I can understand greg's point, but this is pretty much standard for Dev QA. Either way it will not be really secure but the cost and process of treating beta lab certs the same as production never seems to add up.

bd808 added a comment.Nov 26 2014, 4:57 PM

Self-signed SSL certificate chains have historically been a huge pain for automated browser testing. @Cmcmahon has pointed this out elsewhere I think.

greg added a comment.Nov 26 2014, 4:59 PM

I can understand greg's point, but this is pretty much standard for Dev QA. Either way it will not be really secure but the cost and process of treating beta lab certs the same as production never seems to add up.

We'll probably need to figure it out at some point as UX/Design/Product/etc want to do more user testing on dev environments. Maybe that use-case is better handled by a non-Beta Cluster labs instance (so covered by *.wmflabs.org) though.

faidon removed a subscriber: faidon.Nov 26 2014, 6:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 24 2015, 2:10 PM
Paladox added a subscriber: Paladox.Mar 7 2016, 4:57 PM

letsencrypt.org is now available but in beta mode.

we shouldn't wait on that ("Arriving Summer 2015")

With summer *2016* fast approaching, I think it's time to scrap this plan and just go with Let's Encrypt.
Here are some older tickets on the subject: T50501: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org, T70387: Beta Cluster no longer listens for HTTPS - can we close this in favour of one of them?
This ticket is much newer but we can't do anything with it because it's restricted access: {T97593}
(I'm going to get that restricted ticket either made public or closed.)