Page MenuHomePhabricator

SSL-config of the OTRS is outdated
Closed, ResolvedPublic

Description

The SSL-config of the OTRS (ticket.wikimedia.org) is outdated. No PFS, weak key-options and the MAC of the certificate still use SHA1 (the later is not that urgent and can be fixed during the next cert-renewing).
Because the OTRS contains sensitive data, a SSL-setup that is at least as secure as the SSL-config for Wikipedia is needed.
Maybe it would be a good idea to config DNSSEC for that domain too (and if you REALLY in the mood: DANE too).

Event Timeline

DaBPunkt created this task.Mar 4 2015, 2:25 PM
DaBPunkt raised the priority of this task from to High.
DaBPunkt updated the task description. (Show Details)
DaBPunkt added a subscriber: DaBPunkt.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 4 2015, 2:25 PM
Steinsplitter updated the task description. (Show Details)Mar 4 2015, 3:19 PM
Steinsplitter set Security to None.
Krenair added a subscriber: Krenair.

Change 198714 had a related patch set uploaded (by Matanya):
otrs: Enable strict transport security

https://gerrit.wikimedia.org/r/198714

SSL config supports PFS, and the keys are ok. I pushed a patch for hsts, and the SHA1 will probably taker care of later.

SSL config supports PFS

Sure it does, but the webserver for our OTRS doesn’t use it. HSTS is a nice idea, yes

Change 198714 abandoned by Matanya:
otrs: Enable strict transport security

Reason:
dup

https://gerrit.wikimedia.org/r/198714

MC8 updated the task description. (Show Details)Mar 23 2015, 5:45 PM
MC8 removed a project: Patch-For-Review.
MC8 added a subscriber: MC8.
Dzahn added a subscriber: Dzahn.Mar 25 2015, 5:33 AM

manifests/role/otrs.pp already uses:

$ssl_settings = ssl_ciphersuite('apache-2.2', 'compat', '365')

either this is not actually used or the same problems apply to a couple services using the same settings and could be solved globally in ssl_ciphersuite. HSTS should be enabled as well (the '365' part of it). The " MAC of the certificate still use SHA1" should be part of T73156 and is a bit different because it requires new certs, not just changing config.

Dzahn added a comment.Mar 25 2015, 5:42 AM
'compat' => 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECD    HE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA256    :ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-R    SA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AE    S256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128:AES256:HIGH:!aNULL:!eNULL:    !EXPORT:!DES:!MD5:!PSK:!DH',

Sure it does, but the webserver for our OTRS doesn’t use it. HSTS is a nice idea, yes

either this is not actually used or the same problems apply to a couple services using the same settings and could be solved globally in ssl_ciphersuite. HSTS should be enabled as well (the '365' part of it). The " MAC of the certificate still use SHA1" should be part of T73156 and is a bit different because it requires new certs, not just changing config.

Are there any problems in the settings? PFS is not supported here since Apache 2.2 doesn't support ECDHE cipher suites. I recall @Dzahn said that OTRS would be behind misc-web. If so, then both the cipher suite and certificate issues will be gone.

Dzahn added a subscriber: Jgreen.Mar 31 2015, 6:16 PM

recall @Dzahn said that OTRS would be behind misc-web. If so, then both the cipher suite and certificate issues will be gone.

No, it's not behind misc-web. OTRS is standalone on iodine. Maybe @Jgreen can comment if it's feasible to move it.

No, it's not behind misc-web. OTRS is standalone on iodine. Maybe @Jgreen can comment if it's feasible to move it.

I'm not that familiar with misc-web, but moving OTRS behind a reverse proxy would take some testing. I know it does session tracking using both client IP and cookies, this is already problematic with clients that change IP or IPv4/IPv6 dynamically. So we'll need to make sure OTRS is sane about XFF and similar headers.

Dzahn added a comment.Jun 4 2015, 8:33 PM

could this ticket be split into more specific ToDos?

No PFS, weak key-options and the MAC of the certificate still use SHA1
and
config DNSSEC for that domain too (and if you REALLY in the mood: DANE too).

are all kind of separate

Dzahn added a subscriber: RobH.Jun 4 2015, 8:58 PM

@RobH unlike all other replaced certs i don't see this one in Gerrit yet. Could you get that too?

Dzahn assigned this task to RobH.Jun 4 2015, 8:59 PM
FDMS awarded a token.Jun 15 2015, 2:40 PM
Keegan added a subscriber: Keegan.Jun 18 2015, 7:37 PM

@RobH any updates on this?

Change 221161 had a related patch set uploaded (by RobH):
updating ticket.wikimedia.org cert with sha256

https://gerrit.wikimedia.org/r/221161

RobH added a comment.Jul 2 2015, 9:03 PM

So I think I should be able to simply roll the above certificate replacement into place with minimal downtime. I'll run through said plan on this task for peer review.

OTRS is defined in manifests/role/otrs.pp and templates/apache/sites/ticket.wikimedia.org.erb. The update should be as simple as pushing in said change, and running puppet on iodine. The puppet run should replace the certificate file, (possibly) regenerate the chain file, and refresh/restart apache.

If the chain file isn't recreated automatically, it can then be manually deleted and puppet re-run to create the file.

Unfortunately, this process could result in certificate errors during the window. Since OTRS is high visibility to a significant volunteer base, this should be scheduled via the deployment calendar and email notification sent to the OTRS folks.

I'll create a task for the roadmap planning of next week's maintenance/deployment calendar.

RobH lowered the priority of this task from High to Normal.Jul 2 2015, 9:21 PM

Change 221161 merged by RobH:
updating ticket.wikimedia.org cert with sha256

https://gerrit.wikimedia.org/r/221161

RobH removed RobH as the assignee of this task.Jul 7 2015, 5:14 PM

I've completed the SHA256 update.

I'm not entirely certain what remains, but seems to be:

  • No PFS

-config DNSSEC for that domain too (and if you REALLY in the mood: DANE too).

BBlack added a subscriber: BBlack.Jul 7 2015, 5:17 PM

DNSSEC and DANE are not things we currently do.

BBlack added a comment.Jul 7 2015, 5:19 PM

And the server is currently configured to do PFS with reasonably-modern clients. It can't support PFS with some older/crappier clients until the machine is upgraded to Jessie, basically.

BBlack added a comment.Jul 7 2015, 5:23 PM

Actually that's not an accurate statement at all. The Apache2.2 on precise can't really do PFS at all in our configs :/ Still, upgrade to Jessie!

DNSSEC and DANE are not things we currently do.

The question is: Why? A non-public-system like the OTRS is the ideal enviroment to test such (for the WMF) new technologies. The OTRS has its own sub-domain under wikimedia.org, so a DNSSEC-entry is a peace of cake and can not interrupt Wikipedia and co. Just run “zonesigner” on the zone-file and add the DS-hash at your domain-hoster – that’s not 1h of work, even if you never did it before. I’m willing to provide assistance.

BBlack added a comment.Jul 8 2015, 1:23 PM

DNSSEC and DANE are not things we currently do.

The question is: Why? A non-public-system like the OTRS is the ideal enviroment to test such (for the WMF) new technologies. The OTRS has its own sub-domain under wikimedia.org, so a DNSSEC-entry is a peace of cake and can not interrupt Wikipedia and co. Just run “zonesigner” on the zone-file and add the DS-hash at your domain-hoster – that’s not 1h of work, even if you never did it before. I’m willing to provide assistance.

We don't have a "domain hoster", we do this all ourselves in the Operations team. My reluctance isn't for lack of understanding how DNSSEC works, there are just a lot of other complex issues to deal with before we can look at it here (not the least of which is the reflection problems and other general design issues), and integrating DNSSEC support into the DNS server software we're using (which, full disclosure, is one I wrote before I started working here: https://github.com/gdnsd/gdnsd ) is non-trivial as well, which we value for its other properties related to geographic loadbalancing and such. There's a separate and somewhat-stale ticket about it here: https://phabricator.wikimedia.org/T26413 . Pragmatically speaking, I don't expect much movement on this until at least "sometime later this year".

We don't have a "domain hoster", we do this all ourselves in the Operations team.

I spoke of the provider where you manage the org-domain. But maybe you have an even more direct access (which would simple the DNSSEC also more…)

Pragmatically speaking, I don't expect much movement on this until at least "sometime later this year".

It is not THAT urgent, if it happens in 2015 that’s more than early enough.

Krenair added a comment.EditedSep 18 2015, 5:44 PM

This depends on T105125: upgrade iodine to jessie
Which should be done as part of T105554: move OTRS to a VM
Which depends on T111532: EQIAD: 1 VM request for OTRS (actually, it looks to me like this was already done, will comment there in a sec)
Which mentions an OTRS upgrade as one of ops' goals (T74109: Upgrade OTRS to a more recent stable release - the previous goal in that objective was completed today) for Q1 2015-2016, so it sounds to me like this has a pretty good chance of getting ops attention this year or maybe next?

Seems everything from this ticket except DNSSEC and DANE are fixed. Does otrs does its own SMTP?

Dzahn moved this task from Backlog to Services to fix on the HTTPS board.Dec 3 2015, 7:34 PM
Restricted Application added a subscriber: Matthewrbowker. · View Herald TranscriptDec 3 2015, 7:34 PM
Dzahn added a comment.Feb 4 2016, 12:15 AM

we should check one more time since today OTRS moved to mendelevium. i think we already fixed everything except the DNSSEC and DANE part before though, as Jan Zerebecki said above. and those parts are probably won't fix.

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptFeb 4 2016, 12:15 AM
Dzahn added a comment.Feb 4 2016, 12:25 AM

https://www.ssllabs.com/ssltest/analyze.html?d=ticket.wikimedia.org grade A

@DaBPunkt i would like to claim this is resolved. All the main things you listed when this was created have been addressed meanwhile.

  • does not use SHA1, uses SHA256
  • Forward Secrecy is: Yes (with most browsers) ROBUST
  • EC 256 bits

and DNSSEC and DANE was listed as a "if you are in the mood" and is not going to happen as a special case for just OTRS. if it is it would be a separate and long-term ticket for multiple services.

Dzahn closed this task as Resolved.Feb 4 2016, 12:25 AM
Dzahn claimed this task.

https://www.ssllabs.com/ssltest/analyze.html?d=ticket.wikimedia.org grade A
@DaBPunkt i would like to claim this is resolved. All the main things you listed when this was created have been addressed meanwhile.

ok.