Page MenuHomePhabricator

Chrome warns about insecure certificate on gerrit.wikimedia.org
Closed, ResolvedPublic

Description

Chrome:

The identity of this website has been verified by RapidSSL CA but it does not have public audit records.
The site is using outdated security settings that may prevent future versions of Chrome from being able to safely access it.

Event Timeline

Krinkle created this task.Dec 3 2014, 7:01 AM
Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle changed Security from none to None.
Krinkle added projects: Gerrit, HTTPS, acl*sre-team.
Krinkle added a subscriber: Krinkle.
Krenair added a subscriber: Krenair.Dec 3 2014, 8:47 AM
Aklapper triaged this task as Medium priority.Dec 3 2014, 2:06 PM

Also see T73156: Replace SHA1 certificates with SHA256 about another Chrome warning for Gerrit.

Aklapper raised the priority of this task from Medium to High.Dec 3 2014, 6:21 PM
Seb35 added a subscriber: Seb35.Dec 13 2014, 5:38 PM

The certificate *.wikimedia.org used on phabricator.wikimedia.org and other sites can be used; it has a SHA-256 signature.

Dzahn added a subscriber: Dzahn.Dec 22 2014, 1:44 PM

The certificate *.wikimedia.org used on phabricator.wikimedia.org and other sites can be used; it has a SHA-256 signature.

but we don't want to copy that cert to the gerrit host.

Dzahn added a comment.Dec 22 2014, 1:55 PM

ways to solve this:

  • resolve T18
  • buy a new cert
  • move gerrit behind misc-web varnish?? (without caching)
Dzahn added a comment.Dec 22 2014, 2:01 PM

The certificate *.wikimedia.org used on phabricator.wikimedia.org and other sites can be used; it has a SHA-256 signature.

The screenshot doesn't make it look like that is the actual issue but instead that it comes from RapidSSL. ?

I agree with this task being a relatively high priority.

I discussed this issue with @Krenair earlier today. The warning in Chrome is unsightly. He says we just need a new certificate for gerrit.wikimedia.org that doesn't rely on outdated algorithms, maybe?

en.wikipedia.org and lists.wikimedia.org both seem unaffected by this issue. @Dzahn: it feels like gerrit.wikimedia.org is already using its own certificate... does a new one need to be purchased or could we just fix the current one?

RobH added a subscriber: RobH.Jan 7 2015, 5:51 PM
RobH added a subscriber: demon.Jan 7 2015, 10:57 PM

So we have two options (that I see):

  • Push gerrit behind misc-web-lb, so it uses the wildcard certifiate (globalsign) and doesn't have this issue.
  • Order a new certificate for gerrit (since we don't use RapidSSL for any updated deployments).

We purchased a gerrit.wikimedia.org certificate per Chad's request on old RT#4976, and I know he has explained why its better on its own cert, but I cannot find said reasoning. I've CC'd Chad so perhaps he can reply back with why it needs its own. (If it no longer does, we can put behind misc-web; if it does I'll create a procurement ticket for it.)

demon added a comment.Jan 7 2015, 11:46 PM

We tried putting it behind misc-web-lb but we had problems with the git protocol behaving through the proxy if memory serves.

The only reason we bought it it's own cert was because we wanted to stop using *.wikimedia.org on misc services (and if memory serves we bought several similar ones at the time). It can very easily use a star or combined cert.

@mark, it sounds like we need to buy a certificate to make this go away, is that something you would approve?

mark added a comment.Jan 8 2015, 3:18 PM
In T76562#961213, @Chad wrote:

We tried putting it behind misc-web-lb but we had problems with the git protocol behaving through the proxy if memory serves.

With normal HTTP proxying by Varnish I can imagine that. But has VCL return(pipe) been attempted as well, such that Varnish acts as simple TCP proxy for the remainer of the TCP connection?

demon added a comment.Jan 8 2015, 3:27 PM

I don't remember to be honest. We could certainly test it.

It is relevant for Gerrit-Migration either way, so we definitely should then

RobH added a comment.Jan 8 2015, 6:46 PM

I'd suggest we test and use the misc-web-lb for this, since it means one less certificate and therefore less general overhead (since we already have the commitment to maintain misc-web-lb).

RobH claimed this task.Jan 8 2015, 9:39 PM

I'll test this out with setting up the misc web to handle it and locally hacking my /etc/hosts, not changing dns

RobH added a comment.Jan 8 2015, 10:41 PM

So this won't work behind misc-web-lb until after the gerrit dependency on using sshd on the same interface is happening (since misc-web-lb redirects all backend traffic to port 8080.)

The sshd port could be configured to a second interface to work around this. I'm not sure if it is worth a lot more cycles of work, but it would be nice to have all these misc services behind a centrally configured ssl termination cluster.

Krinkle added a comment.EditedJan 9 2015, 2:27 AM

The certificate *.wikimedia.org used on phabricator.wikimedia.org and other sites can be used; it has a SHA-256 signature.

The screenshot doesn't make it look like that is the actual issue but instead that it comes from RapidSSL. ?

The message does sound like that, but I don't think that's the problem. The same message is shown when in the details about phrabricator.wikimedia.org (which has a green badge).


RobH added a comment.Jan 13 2015, 7:12 PM

There are two issues here that I can see:

  1. gerrit.wikimedia.org certificate is sha1
  1. gerrit.wikimedia.org is rapidssl certificate, a vendor we have migrated away from, so any reissues or revocations of a certificate means we just move it to the new vendor.

As such, I've created an RT ticket (9130) to get approval to order a new certificate for this. Since our procurement process has not migrated quite yet (it will soon, but will be last RT item to migrate), I'll just keep this assigned to me until after the order is completed and implemented.

mark added a comment.Jan 14 2015, 4:27 PM
In T76562#964609, @RobH wrote:

So this won't work behind misc-web-lb until after the gerrit dependency on using sshd on the same interface is happening (since misc-web-lb redirects all backend traffic to port 8080.)
The sshd port could be configured to a second interface to work around this. I'm not sure if it is worth a lot more cycles of work, but it would be nice to have all these misc services behind a centrally configured ssl termination cluster.

I don't understand what this means. Can you elaborate?

RobH added a comment.Jan 14 2015, 9:03 PM

Well, first Faidon advised this wouldn't work, and that I shouldn't waste time on it. I wanted to confirm why it wouldn't work, since I imagined I'd be asked in a ticket at some point.

Ytterbium is the current gerrit host, and it communicates over port 8080 for some of its functions. Additionally, it has to use the typical 443 or port 80 for web requests. My understanding of the varnish backend config is you can only route a single port (default 80). The role/cache:misc shows port 80 for default, but previous attempts to make gerrit work listing the port 8080 requirement. Since we cannot do forwards for both 8080 and 80 (replacing 443 once we used misc-web), it seems this wouldn't work. Again, this is my understanding of it, and if I'm wrong, please let me know! (The ideal is this goes behind misc-web if only to simplify our ssl certificate requirements.)

Previous testing discussion with both Daniel and Chad seem to back this up. While there may be a work around, I don't quite know how to do so.

So if a work around is known, we can attempt it, or we can simply purchase a new gerrit certificate. (RT#9130)

RobH added a comment.Jan 14 2015, 9:39 PM

Ok, I was wrong about some of the above, Chad volunteered a bit more time so we could properly detail this.

Gerrit runs and listens to 8080 for jetty on ytterbium. It also listens on 80 for redirection to 443, and then 443 in apache redirects to 8080. (This isn't confusing at all!) If we moved this behind misc-web, we would drop all the apache config, and simply have misc-web direct traffic to 8080, and configure jetty to listen on the IP address. However, this doesn't address the actual issue; ssh pulls for git/gerrit use port 29418.

So the issue is we need to use port 8080 (gerrit web) and port 29418 (git/gerrit ssh).

Daniel suggested that we could use an iptables rule on the misc-web systems to redirect all traffic for ytterbium port 29418 to bypass the varnish directly and go to the system, but we aren't entirely certain if it would work properly, and it is a non-standard solution to the issue. So I'm not sure if we should adopt it, test it, or simply purchase the SSL certificate for gerrit to use on its own.

Also note we'll have this issue again when gerrit is replaced by phabricator. We'll either have to disable ssl pulls (and use https only, which is what Chad and I discussed but we aren't 100% certain ssl pull is needed), or do a similar hack as the above suggestion, since we want/need phabricator to stay cached, just buying a certificate won't solve that issue.

Dzahn added a comment.Jan 14 2015, 9:50 PM

for the related issue that we want to make Gerrit listen on 22, and not on 29418, which we'd like anyways, besides the cert issue being influence by it

https://phabricator.wikimedia.org/T84713
https://gerrit.wikimedia.org/r/#/c/174015/
https://gerrit.wikimedia.org/r/#/c/172313/
https://phabricator.wikimedia.org/T37611

Dzahn added a comment.Jan 14 2015, 9:51 PM

the suggestion to use an iptables rule to forward 22 to 29418 came from Faidon recently on IRC

RobH removed RobH as the assignee of this task.Jan 15 2015, 10:12 PM
RobH raised the priority of this task from High to Needs Triage.
RobH added a project: ops-core.
mark assigned this task to RobH.Jan 20 2015, 12:01 AM

Alright, if it's complicated to do with misc-web, let's buy a one-off ticket for it then.

Dzahn triaged this task as High priority.Jan 28 2015, 5:57 PM
Dzahn removed a project: ops-core.
RobH added a comment.Feb 3 2015, 6:12 PM

cert has been purchased, the patchset to merge for this is:

https://gerrit.wikimedia.org/r/#/c/188396/

When this happens, someone has to rm the chained file for it to be properly recreated. I've put in both Daniel and Chad to possibly review my patch (I rather not be the only set of eyes)

RobH closed this task as Resolved.Feb 3 2015, 9:00 PM

gerrit.wikimedia.org now has a new globalsign cert that is sha256, resolving.

Restricted Application added a project: Traffic. · View Herald TranscriptAug 5 2016, 11:40 AM