Page MenuHomePhabricator

Create Druid public cluster such AQS can query druid public data
Closed, ResolvedPublic21 Story Points

Description

Authentication from aqs to druid

  • Rename druid configs in puppet to 'druid-analytics'.
  • Create LVS endpoint for druid-analytics-broker - @Ottomata - Will be postponed for the moment, details in the task T177511
  • Remove druid100[456] from druid-analytics cluster (druid decom)
  • Wipe druid data from druid100[456].
  • Puppetize new druid-public cluster for druid100[456]
  • Ferm rule to allow AQS and LVS (health checks) to reach druid100[456]'s broker ports - already done since druid100[456]:8082 are reachable by DOMAIN_NETWORK
  • Create LVS endpoint for druid-public-broker

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 19 2017, 3:54 PM

Change 378956 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add LVS service for druid-broker

https://gerrit.wikimedia.org/r/378956

Ottomata updated the task description. (Show Details)Sep 19 2017, 4:47 PM

Change 378967 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/dns@master] Add druid LVS svc name

https://gerrit.wikimedia.org/r/378967

Change 379513 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add druid.svc.eqiad.wmnet crt file

https://gerrit.wikimedia.org/r/379513

Change 379513 merged by Elukey:
[operations/puppet@production] Add druid.svc.eqiad.wmnet crt file

https://gerrit.wikimedia.org/r/379513

Change 379533 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::druid::worker: introduce tlsproxy for druid

https://gerrit.wikimedia.org/r/379533

Change 379538 had a related patch set uploaded (by Elukey; owner: Elukey):
[labs/private@master] Add tlsproxy fake credentials for Druid

https://gerrit.wikimedia.org/r/379538

Change 379538 merged by Elukey:
[labs/private@master] Add tlsproxy fake credentials for Druid

https://gerrit.wikimedia.org/r/379538

Change 379540 had a related patch set uploaded (by Elukey; owner: Elukey):
[labs/private@master] Rename druid worker hieradata config

https://gerrit.wikimedia.org/r/379540

Change 379540 merged by Elukey:
[labs/private@master] Rename druid worker hieradata config

https://gerrit.wikimedia.org/r/379540

Change 379543 had a related patch set uploaded (by Elukey; owner: Elukey):
[labs/private@master] Add fake TLS private key for druid.svc.eqiad.wmnet

https://gerrit.wikimedia.org/r/379543

Change 379543 merged by Elukey:
[labs/private@master] Add fake TLS private key for druid.svc.eqiad.wmnet

https://gerrit.wikimedia.org/r/379543

Change 379544 had a related patch set uploaded (by Elukey; owner: Elukey):
[labs/private@master] Rename the druid fake SSL key

https://gerrit.wikimedia.org/r/379544

Change 379544 merged by Elukey:
[labs/private@master] Rename the druid fake SSL key

https://gerrit.wikimedia.org/r/379544

Change 379559 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] network::constants: add aqs hosts

https://gerrit.wikimedia.org/r/379559

Change 379559 merged by Elukey:
[operations/puppet@production] network::constants: add aqs hosts

https://gerrit.wikimedia.org/r/379559

Change 380449 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Introduce profile::druid::worker

https://gerrit.wikimedia.org/r/380449

Change 380449 merged by Elukey:
[operations/puppet@production] Introduce profile::druid::worker

https://gerrit.wikimedia.org/r/380449

Change 380800 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Improvements to druid profiles, move druid role out of analytics_cluster

https://gerrit.wikimedia.org/r/380800

Change 380804 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] [WIP] Set up separate druid public-eqiad cluster.

https://gerrit.wikimedia.org/r/380804

Change 380800 merged by Elukey:
[operations/puppet@production] Improvements to druid profiles, move druid role out of analytics_cluster

https://gerrit.wikimedia.org/r/380800

Change 381221 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::druid::analytics::worker: split properties into public/private

https://gerrit.wikimedia.org/r/381221

Change 381221 merged by Elukey:
[operations/puppet@production] role::druid::analytics::worker: split properties into public/private

https://gerrit.wikimedia.org/r/381221

Change 381230 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] hieradata: add sites for kafka_jumbo in ganglia_clusters

https://gerrit.wikimedia.org/r/381230

Change 381230 merged by Elukey:
[operations/puppet@production] hieradata: add sites for kafka_jumbo in ganglia_clusters

https://gerrit.wikimedia.org/r/381230

Change 381241 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::druid::*: include profile::druid::common

https://gerrit.wikimedia.org/r/381241

Change 381241 merged by Elukey:
[operations/puppet@production] profile::druid::*: include profile::druid::common

https://gerrit.wikimedia.org/r/381241

Nuria renamed this task from Create Druid HTTP basic auth proxy and LVS endpoints for druid, open this to AQS in prod network to Create Druid public cluster such AQS can query druid public data.Oct 2 2017, 3:42 PM
Nuria updated the task description. (Show Details)

Change 378967 merged by Ottomata:
[operations/dns@master] Add druid-analytics LVS svc name

https://gerrit.wikimedia.org/r/378967

Ottomata updated the task description. (Show Details)Oct 4 2017, 5:28 PM
Ottomata changed the point value for this task from 8 to 21.

Change 378956 merged by Ottomata:
[operations/puppet@production] Add LVS service for druid-analytics-broker

https://gerrit.wikimedia.org/r/378956

OOF.

We can't add LVS to stuff in the Analytics VLAN. At least, not so easily. To be discussed tomorrow post standup.

Ottomata updated the task description. (Show Details)Oct 5 2017, 4:08 PM

Change 383154 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove druid100[456] from druid analytics cluster.

https://gerrit.wikimedia.org/r/383154

Change 383154 merged by Ottomata:
[operations/puppet@production] Remove druid100[456] from druid analytics cluster.

https://gerrit.wikimedia.org/r/383154

Mentioned in SAL (#wikimedia-operations) [2017-10-09T16:34:44Z] <ottomata> beginning decom and reinstall process for druid1004-1006 -- T176223

Today I set druid100[456] as spare::systemm, and stopped druid services there. Depending on the outcome of T177511, they are ready to be repurposed as a new druid cluster.

elukey updated the task description. (Show Details)
elukey updated the task description. (Show Details)
elukey updated the task description. (Show Details)Oct 11 2017, 1:32 PM
elukey updated the task description. (Show Details)Oct 11 2017, 1:37 PM

Change 383568 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/dns@master] Rename 10.2.2.38 to druid-public

https://gerrit.wikimedia.org/r/383568

Change 383568 merged by Ottomata:
[operations/dns@master] Rename 10.2.2.38 to druid-public

https://gerrit.wikimedia.org/r/383568

Change 379533 abandoned by Elukey:
role::analytics_cluster::druid::worker: introduce tlsproxy for druid

https://gerrit.wikimedia.org/r/379533

Change 380804 merged by Elukey:
[operations/puppet@production] Set up separate druid public-eqiad cluster.

https://gerrit.wikimedia.org/r/380804

Change 383833 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::an_cluster::database::meta: allow the druid pulic cluster to use mysql

https://gerrit.wikimedia.org/r/383833

Change 383833 merged by Elukey:
[operations/puppet@production] role::an_cluster::database::meta: allow the druid pulic cluster to use mysql

https://gerrit.wikimedia.org/r/383833

elukey updated the task description. (Show Details)Oct 12 2017, 1:48 PM

Mentioned in SAL (#wikimedia-operations) [2017-10-12T13:48:37Z] <elukey> deployed the new Analytics Public Druid cluster - T176223

Change 383880 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] LVS for druid-public broker and overlord

https://gerrit.wikimedia.org/r/383880

Change 383998 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Rename druid-public to druid-public-broker

https://gerrit.wikimedia.org/r/383998

Change 383998 merged by Elukey:
[operations/dns@master] Rename druid-public to druid-public-broker

https://gerrit.wikimedia.org/r/383998

Mentioned in SAL (#wikimedia-operations) [2017-10-16T16:56:23Z] <ottomata> deploying LVS for druid-public-broker https://phabricator.wikimedia.org/T176223

Change 383880 merged by Ottomata:
[operations/puppet@production] Add the LVS config for the druid-public-broker service

https://gerrit.wikimedia.org/r/383880

Ottomata updated the task description. (Show Details)Oct 16 2017, 5:04 PM

Main thing to remember for the "Create LVS endpoing for druid-public-overlord (for oozie job indexing)" task: usually the target port of a LVS service is meant to be accessed by domain networks, not limited to a subset of hosts. In this case the Indexing service domain should only be accessible by the analytics network, so probably some comments in puppet explaining our use case are needed.

Change 386426 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/dns@master] Add LVS IP for druid-public-overlord

https://gerrit.wikimedia.org/r/386426

Change 386426 merged by Ottomata:
[operations/dns@master] Add LVS IP for druid-public-overlord

https://gerrit.wikimedia.org/r/386426

Change 386427 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add LVS for druid-public-overlord indexing service

https://gerrit.wikimedia.org/r/386427

We gotta block on T179027 before we can add ferm rules to allow LVS health checks for druid-public-overlord LVS.

Ottomata moved this task from In Progress to Paused on the Analytics-Kanban board.Oct 25 2017, 6:31 PM
Ottomata updated the task description. (Show Details)Nov 20 2017, 4:55 PM

Moved 'Create LVS endpoing for druid-public-overlord (for oozie job indexing)' to its own task so we can close this one. T180971

Ottomata moved this task from Paused to Done on the Analytics-Kanban board.Nov 20 2017, 4:56 PM
Nuria closed this task as Resolved.Nov 27 2017, 9:28 PM

Change 386427 abandoned by Ottomata:
Add LVS for druid-public-overlord indexing service

https://gerrit.wikimedia.org/r/386427