Page MenuHomePhabricator
Create Task
Maniphest T82319

Use SSLCertificateChainFile in Gerrit Apache configuration
Closed, ResolvedPublic
Actions

Description

Author: hashar

Description:

We have a bunch of virtual hosts explicitly declaring the SSL certificate to
use. Per Faidon review, we should instead use: SSLCACertificatePath
/etc/ssl/certs Then apache will find out which one to use. (note I have closed
the bug report, lets track this in RT).

Bugzilla Ticket: 46325 => T48325: SSLCACertificatePath /etc/ssl/certs for all our virtual hosts

Referred To By:
{T82359}

Details

Reference
rt4823
SubjectRepoBranchLines +/-
ganglia: use SSLCertificateChainFileoperations/puppetproduction+2 -0
Customize query in gerrit

Related Objects

Event Timeline

• rtimport raised the priority of this task from to Medium.Dec 18 2014, 1:33 AM
• rtimport added projects: acl*sre-team, ops-requests.
• rtimport set Reference to rt4823.
• rtimport created this task.Mar 26 2013, 12:13 PM
Dzahn added a comment.Mar 26 2013, 5:57 PM

blog.wikimedia.org: SSLEngine on
blog.wikimedia.org: SSLCertificateFile /etc/ssl/certs/star.wikimedia.org.pem
blog.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
blog.wikimedia.org: SSLEngine on
blog.wikimedia.org: SSLCertificateFile /etc/ssl/certs/star.wikimedia.org.pem
blog.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
bugzilla.wikimedia.org: SSLEngine On
bugzilla.wikimedia.org: SSLCertificateFile
/etc/ssl/certs/star.wikimedia.org.pem
bugzilla.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
bugzilla.wikimedia.org: SSLEngine On
bugzilla.wikimedia.org: SSLCertificateFile
/etc/ssl/certs/star.wikimedia.org.pem
bugzilla.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
bugzilla.wikimedia.org: SSLEngine On
bugzilla.wikimedia.org: SSLCertificateFile
/etc/ssl/certs/star.wikimedia.org.pem
bugzilla.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
contacts.wikimedia.org: SSLEngine On
contacts.wikimedia.org: SSLCertificateFile
/etc/ssl/certs/star.wikimedia.org.pem
contacts.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
controller.wikimedialabs.org: SSLEngine on
controller.wikimedialabs.org: SSLCertificateFile
/etc/ssl/certs/star.wikimedia.org.pem
controller.wikimedialabs.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
doc.wikimedia.org: SSLEngine on
doc.wikimedia.org: SSLCertificateFile /etc/ssl/certs/star.mediawiki.org.pem
doc.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.mediawiki.org.key
doc.wikimedia.org: SSLCACertificateFile /etc/ssl/certs/RapidSSL_CA.pem
doc.wikimedia.org: SSLEngine on
doc.wikimedia.org: SSLCertificateFile /etc/ssl/certs/star.wikimedia.org.pem
doc.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
doc.wikimedia.org: SSLCACertificateFile /etc/ssl/certs/RapidSSL_CA.pem
noc.wikimedia.org: SSLEngine on
noc.wikimedia.org: SSLCertificateFile /etc/ssl/certs/star.wikimedia.org.pem
noc.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
ocs.wikimania2009.wikimedia.org: SSLEngine on
ocs.wikimania2009.wikimedia.org: SSLCertificateFile
/etc/ssl/certs/star.wikimedia.org.pem
ocs.wikimania2009.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
outreachcivi.wikimedia.org: SSLEngine On
outreachcivi.wikimedia.org: SSLCertificateFile
/etc/ssl/certs/star.wikimedia.org.pem
outreachcivi.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
planet.wikimedia.org:# make all planet requests go to http instead of ssl
planet.wikimedia.org: SSLEngine on
planet.wikimedia.org: SSLCertificateFile /etc/ssl/certs/star.wikimedia.org.pem
planet.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
secure.wikimedia.org: SSLEngine on
secure.wikimedia.org: SSLCertificateFile /etc/ssl/certs/star.wikimedia.org.pem
secure.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
secure.wikimedia.org: # old URLs, redirect them to proper SSL
svn.wikimedia.org: SSLEngine on
svn.wikimedia.org: SSLCertificateFile /etc/ssl/certs/star.wikimedia.org.pem
svn.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key

• rtimport added a comment.Mar 26 2013, 5:57 PM

Status changed from 'new' to 'open' by RT_System

Dzahn added a comment.Mar 26 2013, 5:58 PM

etherpad_lite.wikimedia.org.erb: SSLEngine on
etherpad_lite.wikimedia.org.erb: SSLCertificateFile <%= etherpad_ssl_cert %>
etherpad_lite.wikimedia.org.erb: SSLCertificateKeyFile <%= etherpad_ssl_key %>
ganglia.wikimedia.org.erb: SSLEngine on
ganglia.wikimedia.org.erb: SSLCertificateFile <%= ganglia_ssl_cert %>
ganglia.wikimedia.org.erb: SSLCertificateKeyFile <%= ganglia_ssl_key %>
gerrit.wikimedia.org.erb: SSLEngine on
gerrit.wikimedia.org.erb: SSLCertificateFile /etc/ssl/certs/<%= ssl_cert %>.pem
gerrit.wikimedia.org.erb: SSLCertificateKeyFile /etc/ssl/private/<%=
ssl_cert_key %>.key
gerrit.wikimedia.org.erb: SSLCACertificateFile /etc/ssl/certs/<%= ssl_ca %>.pem
graphite.wikimedia.org: SSLEngine on
graphite.wikimedia.org: SSLCertificateFile
/etc/ssl/certs/star.wikimedia.org.pem
graphite.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
graphite.wikimedia.org: SSLEngine on
graphite.wikimedia.org: SSLCertificateFile
/etc/ssl/certs/star.wikimedia.org.pem
graphite.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
graphite.wikimedia.org: SSLEngine on
graphite.wikimedia.org: SSLCertificateFile
/etc/ssl/certs/star.wikimedia.org.pem
graphite.wikimedia.org: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
icinga.wikimedia.org.erb: SSLEngine On
icinga.wikimedia.org.erb: SSLCertificateFile
/etc/ssl/private/star.wikimedia.org.pem
icinga.wikimedia.org.erb: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
icinga.wikimedia.org.erb: SSLCACertificateFile /etc/ssl/certs/RapidSSL_CA.pem
icinga.wikimedia.org.erb: SSLEngine On
icinga.wikimedia.org.erb: SSLCertificateFile
/etc/ssl/private/star.wikimedia.org.pem
icinga.wikimedia.org.erb: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
icinga.wikimedia.org.erb: SSLCACertificateFile /etc/ssl/certs/RapidSSL_CA.pem
icinga.wikimedia.org.erb: SSLRequireSSL
icinga.wikimedia.org.erb: SSLEngine On
icinga.wikimedia.org.erb: SSLCertificateFile
/etc/ssl/private/star.wikimedia.org.pem
icinga.wikimedia.org.erb: SSLCertificateKeyFile
/etc/ssl/private/star.wikimedia.org.key
icinga.wikimedia.org.erb: SSLCACertificateFile /etc/ssl/certs/RapidSSL_CA.pem
icinga.wikimedia.org.erb: SSLRequireSSL
planet.erb: SSLEngine on
planet.erb: SSLCertificateFile /etc/ssl/certs/star.planet.<%=
planet_domain_name %>.pem
planet.erb: SSLCertificateKeyFile /etc/ssl/private/star.planet.<%=
planet_domain_name %>.key
planet-language.erb: SSLEngine on
planet-language.erb: SSLCertificateFile /etc/ssl/certs/star.planet.<%=
planet_domain_name %>.pem
planet-language.erb: SSLCertificateKeyFile /etc/ssl/private/star.planet.<%=
planet_domain_name %>.key
racktables.wikimedia.org.erb: SSLEngine on
racktables.wikimedia.org.erb: SSLCertificateFile <%= racktables_ssl_cert %>
racktables.wikimedia.org.erb: SSLCertificateKeyFile <%= racktables_ssl_key %>
wikistats.erb: SSLEngine on
wikistats.erb: SSLCertificateFile <%= wikistats_ssl_cert %>
wikistats.erb: SSLCertificateKeyFile <%= wikistats_ssl_key %>
wikitech.wikimedia.org.erb: SSLEngine on
wikitech.wikimedia.org.erb: SSLCertificateFile /etc/ssl/certs/<%= certificate
%>.pem
wikitech.wikimedia.org.erb: SSLCertificateKeyFile /etc/ssl/private/<%=
certificate %>.key

Dzahn added a comment.Mar 26 2013, 7:05 PM

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608719

jeremyb added a comment.Mar 26 2013, 7:27 PM

On 2013-03-26 12:13:22, hashar wrote:

We have a bunch of virtual hosts explicitly declaring the SSL
certificate to
use. Per Faidon review, we should instead use: SSLCACertificatePath
/etc/ssl/certs Then apache will find out which one to use. (note I
have closed
the bug report, lets track this in RT).

[citation needed]
The docs for that directive don't mention anything relevant. That
directive seems to be used only when using client certs.
SSLCACertificatePath<https://httpd.apache.org/docs/2.2/mod/mod_ssl.html#sslcacertificatepath>
says:

This directive sets the directory where you keep the Certificates of
Certification Authorities (CAs) whose clients you deal with. These are

used

to verify the client certificate on Client Authentication.

jeremyb added a comment.Mar 26 2013, 7:31 PM

AdminCc jeremyb added by jeremyb

faidon added a comment.Mar 26 2013, 7:32 PM

On Tue, Mar 26, 2013 at 07:27:25PM +0000, Jeremy Baron via RT wrote:

On 2013-03-26 12:13:22, hashar wrote:

We have a bunch of virtual hosts explicitly declaring the SSL
certificate to
use. Per Faidon review, we should instead use: SSLCACertificatePath
/etc/ssl/certs Then apache will find out which one to use. (note I
have closed
the bug report, lets track this in RT).

[citation needed]

The docs for that directive don't mention anything relevant. That
directive seems to be used only when using client certs.

SSLCACertificatePathhttps://httpd.apache.org/docs/2.2/mod/mod_ssl.html#sslcacertificatepath
says:

This directive sets the directory where you keep the Certificates of
Certification Authorities (CAs) whose clients you deal with. These are used
to verify the client certificate on Client Authentication.

The same page says:
SSLCertificateChainFile
This should be used alternatively and/or additionally to
SSLCACertificatePath for explicitly constructing the server
certificate chain which is sent to the browser in addition to the
server certificate. It is especially useful to avoid conflicts with CA
certificates when using client authentication. Because although
placing a CA certificate of the server certificate chain into
SSLCACertificatePath has the same effect for the certificate chain
construction, it has the side-effect that client certificates issued
by this same CA certificate are also accepted on client
authentication.
Trust me, it works.
Regards,
Faidon

faidon added a comment.Mar 26 2013, 7:32 PM

On Tue, Mar 26, 2013 at 07:05:38PM +0000, Daniel Zahn via RT wrote:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608719

This is unrelated to this, but indeed it's a minor security issue that
affects us, since we put our certificates in /etc/ssl/certs. I pointed
that out to Ryan the other day and we had a discussion but I think we
both forgot about it. It's minor, I can't think of a real attack
scenario in our case (our certificates do not have CA:true).

jeremyb added a comment.Mar 26 2013, 7:37 PM

On 2013-03-26 19:32:00, faidon wrote:

The same page says:

[...]

Trust me, it works.

Ok, great.

Dzahn added a comment.Apr 11 2013, 6:34 PM

Reference by ticket #4912 added by dzahn

Dzahn added a comment.Sep 18 2013, 9:41 PM

Should this stay open? done besides the unrelated issue?

Dzahn added a comment.Sep 18 2013, 9:45 PM

Issue taken by dzahn

Dzahn added a comment.Sep 18 2013, 9:46 PM

do the opposite. remove that from these, per Ryan.
gerrit.wikimedia.org.erb: SSLCACertificatePath /etc/ssl/certs/
git.wikimedia.org.erb: SSLCACertificatePath /etc/ssl/certs/
metrics.wikimedia.org.erb: SSLCACertificatePath /etc/ssl/certs/
planet.erb: SSLCACertificatePath /etc/ssl/certs/
planet-language.erb: SSLCACertificatePath /etc/ssl/certs/
then reject

Dzahn added a comment.Sep 18 2013, 11:29 PM

https://gerrit.wikimedia.org/r/#/c/84901/

Dzahn added a comment.Sep 24 2013, 11:57 AM

On Wed Sep 18 23:29:41 2013, dzahn wrote:

https://gerrit.wikimedia.org/r/#/c/84901/

new patch set, "
replace SSLCACertificatePath with SSLCertificateChainFile
better?

Dzahn added a comment.Sep 27 2013, 2:21 PM

Subject changed from 'Use SSLCACertificatePath in Apache configuration' to 'Use SSLCertificateChainFile in Apache configuration' by dzahn

Dzahn added a comment.Sep 27 2013, 2:22 PM

please review https://gerrit.wikimedia.org/r/#/c/84901/

Dzahn added a comment.Oct 1 2013, 2:36 PM

Untaken by dzahn

Dzahn added a comment.Oct 22 2013, 3:02 AM

we're now doing this per ticket title on metrics.wm and planet.wm:
https://gerrit.wikimedia.org/r/#/c/84901/
just reverted it on gerrit, because of issue with puppet-merge if you don't
specify SSLCACertificatePath (Apache restarts fine)
https://gerrit.wikimedia.org/r/#/c/90676/

Dzahn added a comment.Dec 20 2013, 1:09 AM

Subject changed from 'Use SSLCertificateChainFile in Apache configuration' to 'Use SSLCertificateChainFile in Gerrit Apache configuration' by dzahn

Dzahn added a comment.Dec 20 2013, 1:11 AM

i renamed the ticket from being a global request to just being a Gerrit issue
now. the other services i believe we have all fixed.
why it had to be reverted for gerrit: https://gerrit.wikimedia.org/r/#/c/90676/
so gerrit still does not do this and the ticket is open but it is totally done
on other services now
this is to clarify and get the ticket unstuck

Dzahn added a comment.Dec 20 2013, 1:12 AM

the problem is that if you do this for gerrit and think you do the obvious fix
and Apache still works, then you just broke puppet-merge without expecting it
at all!!
error: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt
CRLfile: none while accessing https://gerrit.wikimedia.org/r/p/operations/pusfatal: HTTP request failed
that is the real problem here, so maybe it should rather be a Bugzilla ticket for Chad now.

ArielGlenn added a comment.Apr 16 2014, 9:03 AM

On Fri Dec 20 01:12:59 2013, dzahn wrote:

the problem is that if you do this for gerrit and think you do the
obvious fix
and Apache still works, then you just broke puppet-merge without
expecting it
at all!!

error: server certificate verification failed. CAfile:
/etc/ssl/certs/ca-certificates.crt
CRLfile: none while accessing
https://gerrit.wikimedia.org/r/p/operations/pusfatal: HTTP request
failed

that is the real problem here, so maybe it should rather be a Bugzilla
ticket for Chad now.

The whine is from git (git fetch, I suppose) and not anything fancy about
puppet-merge. The standard openssl verfication on the chained pem works fine,
so I don't know why giit wouldn't like it. Chad, do you want to look at the
gerrit/apache end of this a bit?

• demon added a comment.Apr 16 2014, 2:22 PM

On Wed Apr 16 09:03:45 2014, ariel wrote:

On Fri Dec 20 01:12:59 2013, dzahn wrote:

the problem is that if you do this for gerrit and think you do the
obvious fix
and Apache still works, then you just broke puppet-merge without
expecting it
at all!!

error: server certificate verification failed. CAfile:
/etc/ssl/certs/ca-certificates.crt
CRLfile: none while accessing
https://gerrit.wikimedia.org/r/p/operations/pusfatal: HTTP request
failed

that is the real problem here, so maybe it should rather be a

Bugzilla

ticket for Chad now.

The whine is from git (git fetch, I suppose) and not anything fancy
about
puppet-merge. The standard openssl verfication on the chained pem
works fine,
so I don't know why giit wouldn't like it. Chad, do you want to look
at the
gerrit/apache end of this a bit?

Huh? I have no clue what's going on here. Way over my head.
-Chad

ArielGlenn added a comment.Apr 16 2014, 3:00 PM

After irc conversation with Chad:
he doesn't know about how git does cert verification so the buck is rightly
passed back to us.

Dzahn removed a project: ops-requests.Feb 3 2015, 8:04 PM
Dzahn set Security to None.
Dzahn changed the visibility from "WMF-NDA (Project)" to "Public (No Login Required)".May 29 2015, 8:04 PM
Dzahn changed the edit policy from "WMF-NDA (Project)" to "All Users".
gerritbot subscribed.Jun 3 2015, 6:14 PM

Change 215508 had a related patch set uploaded (by Dzahn):
ganglia: use SSLCertificateChainFile

https://gerrit.wikimedia.org/r/215508

gerritbot added a project: Patch-For-Review.Jun 3 2015, 6:14 PM
Dzahn added a project: Release-Engineering-Team.Jun 3 2015, 7:39 PM
hashar subscribed.Jun 3 2015, 8:06 PM

Does Release-Engineering-Team has anything to do there? Seems like some infrastructure tech debt.

gerritbot added a comment.Jun 4 2015, 5:08 PM

Change 215508 merged by Dzahn:
ganglia: use SSLCertificateChainFile

https://gerrit.wikimedia.org/r/215508

Dzahn mentioned this in rOPUPba200381a8ed: ganglia: use SSLCertificateChainFile.Jun 4 2015, 5:08 PM
Dzahn subscribed.Jun 5 2015, 12:47 AM
In T82319#1335241, @hashar wrote:

Does Release-Engineering-Team has anything to do there? Seems like some infrastructure tech debt.

I added since we just added the entire releng team to gerrit admins.

Dzahn added a comment.Jun 5 2015, 1:08 AM
In T82319#1335241, @hashar wrote:

Does Release-Engineering-Team has anything to do there? Seems like some infrastructure tech debt.

Actually, you are the author the bug :)

Dzahn closed this task as Resolved.Jun 5 2015, 1:10 AM
Dzahn claimed this task.

we use "SSLCACertificatePath /etc/ssl/certs/" in the Gerrit config (meanwhile) and that is ok too

Dzahn added a comment.Jun 5 2015, 1:13 AM

https://www.ssllabs.com/ssltest/analyze.html?d=gerrit.wikimedia.org

the "-" in "A-" is because we are not supporting PFS which is because the Apache is 2.2.x which is because ytterbium is precise

hashar added a comment.Jun 5 2015, 9:39 AM

Thanks @Dzahn :-)

greg moved this task from INBOX to Done on the Release-Engineering-Team board.Jun 5 2015, 3:44 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL