Page MenuHomePhabricator

TLS certificates for Analytics origin servers
Closed, ResolvedPublic13 Story Points

Description

As briefly discussed on irc with @elukey and documented in T210411, we need origin servers to be accessible via TLS. When it comes to Analytics, this should mean that analytics-tool100[1-3] (and possibly others that I might have forgotten?) need to be accessible via https. Certificates should include the following in the X509v3 Subject Alternative Name field:

  • hue.wikimedia.org
  • yarn.wikimedia.org
  • pivot.wikimedia.org
  • turnilo.wikimedia.org
  • superset.wikimedia.org
  • analytics.wikimedia.org
  • stats.wikimedia.org
  • piwik.wikimedia.org

Event Timeline

ema created this task.Jul 12 2019, 9:48 AM
Restricted Application added a project: Operations. · View Herald TranscriptJul 12 2019, 9:48 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ema triaged this task as Normal priority.Jul 12 2019, 9:48 AM
ema moved this task from Triage to Caching on the Traffic board.Jul 12 2019, 9:53 AM
elukey added a subscriber: Ottomata.EditedJul 16 2019, 1:40 PM

The idea that I have is to re-use what done for the appservers, namely put nginx in front of httpd to terminate TLS. We'll re-use the current standards and most recent puppet code for nginx rather than having to configure httpd.

In theory we could:

  • generate one or more certificates as indicated by Ema via cergen, and then add them to private/public puppet (the .crt should go in ssl/files in puppet public, the private key in the private repo IIUC).
  • use tlsproxy::localssl in all the profiles with httpd

@Ottomata thoughts?

elukey claimed this task.Jul 16 2019, 1:41 PM
elukey updated the task description. (Show Details)
elukey updated the task description. (Show Details)
elukey added a project: Analytics-Kanban.

Hm, all for it! Although, do you think it would be worth exploring the built in TLS support in the services where they support it? I'm pretty sure Hue does. Maybe not though, as then each service would need custom configuration to set that up. Using tlsproxy::localssl would unify the config.

Hm, all for it! Although, do you think it would be worth exploring the built in TLS support in the services where they support it? I'm pretty sure Hue does. Maybe not though, as then each service would need custom configuration to set that up. Using tlsproxy::localssl would unify the config.

Yes exactly, we'd have to configure all the services and make sure that they support the standards provided by nginx..

elukey moved this task from Backlog to In Progress on the User-Elukey board.Jul 16 2019, 2:40 PM

Change 524177 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] tlsproxy::instance: allow overriding ssl compatibility mode

https://gerrit.wikimedia.org/r/524177

elukey updated the task description. (Show Details)Jul 18 2019, 10:29 AM

Change 524184 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add TLS .crt file for Analytics UIs backend services

https://gerrit.wikimedia.org/r/524184

Change 524184 merged by Elukey:
[operations/puppet@production] Add TLS .crt file for Analytics UIs backend services

https://gerrit.wikimedia.org/r/524184

elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.Jul 18 2019, 11:15 AM

Change 524177 merged by Ema:
[operations/puppet@production] tlsproxy::instance: allow overriding ssl compatibility mode

https://gerrit.wikimedia.org/r/524177

Change 524227 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::hadoop::ui: add TLS proxy

https://gerrit.wikimedia.org/r/524227

Change 524227 merged by Elukey:
[operations/puppet@production] role::analytics_cluster::hadoop::ui: add TLS proxy

https://gerrit.wikimedia.org/r/524227

Change 524255 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::superset: add TLS proxy

https://gerrit.wikimedia.org/r/524255

Change 524258 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::webserver: add TLS proxy

https://gerrit.wikimedia.org/r/524258

Change 524259 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::turnilo: add TLS proxy

https://gerrit.wikimedia.org/r/524259

Change 524482 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: use TLS to connect to analytics-tool hosts

https://gerrit.wikimedia.org/r/524482

Change 524488 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::ui: add health check for Yarn and Hue

https://gerrit.wikimedia.org/r/524488

Change 524488 merged by Elukey:
[operations/puppet@production] role::analytics_cluster::ui: add health check for Yarn and Hue

https://gerrit.wikimedia.org/r/524488

Change 524516 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::tlsproxy::service: allow to modify the Nagios contact_group

https://gerrit.wikimedia.org/r/524516

Change 524516 merged by Elukey:
[operations/puppet@production] profile::tlsproxy::service: allow to modify the Nagios contact_group

https://gerrit.wikimedia.org/r/524516

Change 524524 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Update yarn.wikimedia.org's crt file

https://gerrit.wikimedia.org/r/524524

Change 524524 merged by Elukey:
[operations/puppet@production] Update yarn.wikimedia.org's crt file

https://gerrit.wikimedia.org/r/524524

Change 524531 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] superset: move httpd proxy config to a profile

https://gerrit.wikimedia.org/r/524531

Change 524531 merged by Elukey:
[operations/puppet@production] superset: move httpd proxy config to a profile

https://gerrit.wikimedia.org/r/524531

Change 524788 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::superset: add TLS proxy

https://gerrit.wikimedia.org/r/524788

Change 524788 merged by Elukey:
[operations/puppet@production] role::analytics_cluster::superset: add TLS proxy

https://gerrit.wikimedia.org/r/524788

Change 525039 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::tlsproxy::service: add more granularity in monitoring

https://gerrit.wikimedia.org/r/525039

elukey changed the task status from Open to Stalled.Jul 23 2019, 8:01 AM

Change 525039 merged by Elukey:
[operations/puppet@production] profile::tlsproxy::service: add more granularity in monitoring

https://gerrit.wikimedia.org/r/525039

elukey changed the task status from Stalled to Open.Jul 30 2019, 1:33 PM
elukey updated the task description. (Show Details)

Change 526428 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::druid::turnilo: add TLS proxy

https://gerrit.wikimedia.org/r/526428

Change 526428 merged by Elukey:
[operations/puppet@production] role::druid::turnilo: add TLS proxy

https://gerrit.wikimedia.org/r/526428

Change 526438 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::webserver: add TLS proxy

https://gerrit.wikimedia.org/r/526438

Change 526438 merged by Elukey:
[operations/puppet@production] role::analytics_cluster::webserver: add TLS proxy

https://gerrit.wikimedia.org/r/526438

elukey updated the task description. (Show Details)Jul 30 2019, 3:15 PM

Change 526602 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::druid::turnilo::proxy: add health check

https://gerrit.wikimedia.org/r/526602

Change 526602 merged by Elukey:
[operations/puppet@production] profile::druid::turnilo::proxy: add health check

https://gerrit.wikimedia.org/r/526602

Change 524259 abandoned by Elukey:
role::analytics_cluster::turnilo: add TLS proxy

https://gerrit.wikimedia.org/r/524259

Change 524258 abandoned by Elukey:
role::analytics_cluster::webserver: add TLS proxy

https://gerrit.wikimedia.org/r/524258

Change 524255 abandoned by Elukey:
role::analytics_cluster::superset: add TLS proxy

https://gerrit.wikimedia.org/r/524255

Change 528490 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::piwik: add TLS proxy

https://gerrit.wikimedia.org/r/528490

Change 528490 merged by Elukey:
[operations/puppet@production] role::piwik: add TLS proxy

https://gerrit.wikimedia.org/r/528490

elukey updated the task description. (Show Details)Aug 6 2019, 3:05 PM
elukey set the point value for this task to 13.
elukey moved this task from In Progress to Done on the Analytics-Kanban board.

Change 524482 merged by Ema:
[operations/puppet@production] ATS: use TLS to connect to analytics hosts

https://gerrit.wikimedia.org/r/524482

ema closed this task as Resolved.Aug 7 2019, 7:54 AM

Thank you so much @elukey! ATS is now using TLS only for connections to Analytics origins.

Change 528704 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: use TLS to connect to matomo

https://gerrit.wikimedia.org/r/528704

Change 528704 merged by Ema:
[operations/puppet@production] ATS: use TLS to connect to matomo

https://gerrit.wikimedia.org/r/528704