Page MenuHomePhabricator
Create Task
Maniphest T176223

Create Druid public cluster such AQS can query druid public data
Closed, ResolvedPublic21 Estimated Story Points
Actions

Description

Authentication from aqs to druid

  • Rename druid configs in puppet to 'druid-analytics'.
  • Create LVS endpoint for druid-analytics-broker - @Ottomata - Will be postponed for the moment, details in the task T177511
  • Remove druid100[456] from druid-analytics cluster (druid decom)
  • Wipe druid data from druid100[456].
  • Puppetize new druid-public cluster for druid100[456]
  • Ferm rule to allow AQS and LVS (health checks) to reach druid100[456]'s broker ports - already done since druid100[456]:8082 are reachable by DOMAIN_NETWORK
  • Create LVS endpoint for druid-public-broker

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+37 -5Add LVS for druid-public-overlord indexing service
operations/dnsmaster+3 -1Add LVS IP for druid-public-overlord
operations/puppetproduction+44 -0Add the LVS config for the druid-public-broker service
operations/dnsmaster+2 -2Rename druid-public to druid-public-broker
operations/puppetproduction+205 -31Set up separate druid public-eqiad cluster.
operations/puppetproduction+5 -2role::an_cluster::database::meta: allow the druid pulic cluster to use mysql
operations/puppetproduction+100 -0role::analytics_cluster::druid::worker: introduce tlsproxy for druid
operations/dnsmaster+2 -2Rename 10.2.2.38 to druid-public
operations/puppetproduction+12 -1Remove druid100[456] from druid analytics cluster.
operations/puppetproduction+40 -0Add LVS service for druid-analytics-broker
operations/dnsmaster+3 -0Add druid-analytics LVS svc name
operations/puppetproduction+395 -279Improvements to druid profiles, move druid role out of analytics_cluster
operations/puppetproduction+13 -1profile::druid::*: include profile::druid::common
operations/puppetproduction+2 -0hieradata: add sites for kafka_jumbo in ganglia_clusters
operations/puppetproduction+5 -5role::druid::analytics::worker: split properties into public/private
operations/puppetproduction+133 -69Introduce profile::druid::worker
operations/puppetproduction+14 -0network::constants: add aqs hosts
labs/privatemaster+0 -0Rename the druid fake SSL key
labs/privatemaster+3 -0Add fake TLS private key for druid.svc.eqiad.wmnet
labs/privatemaster+0 -0Rename druid worker hieradata config
labs/privatemaster+4 -0Add tlsproxy fake credentials for Druid
operations/puppetproduction+26 -0Add druid.svc.eqiad.wmnet crt file
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 19 2017, 3:54 PM
gerritbot added a subscriber: gerritbot.Sep 19 2017, 4:41 PM

Change 378956 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add LVS service for druid-broker

https://gerrit.wikimedia.org/r/378956

gerritbot added a project: Patch-For-Review.Sep 19 2017, 4:41 PM
Ottomata updated the task description. (Show Details)Sep 19 2017, 4:47 PM
gerritbot added a comment.Sep 19 2017, 5:18 PM

Change 378967 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/dns@master] Add druid LVS svc name

https://gerrit.wikimedia.org/r/378967

Ottomata updated the task description. (Show Details)Sep 19 2017, 5:19 PM
Ottomata merged a task: T176234: Add basic authentication abilities to restbase endpoint to be able to connect to druid authentication proxy.
Ottomata added subscribers: Nuria, • bd808, Zppix and 8 others.
gerritbot added a comment.Sep 21 2017, 10:59 AM

Change 379513 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add druid.svc.eqiad.wmnet crt file

https://gerrit.wikimedia.org/r/379513

gerritbot added a comment.Sep 21 2017, 10:59 AM

Change 379513 merged by Elukey:
[operations/puppet@production] Add druid.svc.eqiad.wmnet crt file

https://gerrit.wikimedia.org/r/379513

gerritbot added a comment.Sep 21 2017, 1:54 PM

Change 379533 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_cluster::druid::worker: introduce tlsproxy for druid

https://gerrit.wikimedia.org/r/379533

gerritbot added a comment.Sep 21 2017, 2:11 PM

Change 379538 had a related patch set uploaded (by Elukey; owner: Elukey):
[labs/private@master] Add tlsproxy fake credentials for Druid

https://gerrit.wikimedia.org/r/379538

gerritbot added a comment.Sep 21 2017, 2:11 PM

Change 379538 merged by Elukey:
[labs/private@master] Add tlsproxy fake credentials for Druid

https://gerrit.wikimedia.org/r/379538

gerritbot added a comment.Sep 21 2017, 2:19 PM

Change 379540 had a related patch set uploaded (by Elukey; owner: Elukey):
[labs/private@master] Rename druid worker hieradata config

https://gerrit.wikimedia.org/r/379540

gerritbot added a comment.Sep 21 2017, 2:19 PM

Change 379540 merged by Elukey:
[labs/private@master] Rename druid worker hieradata config

https://gerrit.wikimedia.org/r/379540

gerritbot added a comment.Sep 21 2017, 2:35 PM

Change 379543 had a related patch set uploaded (by Elukey; owner: Elukey):
[labs/private@master] Add fake TLS private key for druid.svc.eqiad.wmnet

https://gerrit.wikimedia.org/r/379543

gerritbot added a comment.Sep 21 2017, 2:35 PM

Change 379543 merged by Elukey:
[labs/private@master] Add fake TLS private key for druid.svc.eqiad.wmnet

https://gerrit.wikimedia.org/r/379543

gerritbot added a comment.Sep 21 2017, 2:39 PM

Change 379544 had a related patch set uploaded (by Elukey; owner: Elukey):
[labs/private@master] Rename the druid fake SSL key

https://gerrit.wikimedia.org/r/379544

gerritbot added a comment.Sep 21 2017, 2:40 PM

Change 379544 merged by Elukey:
[labs/private@master] Rename the druid fake SSL key

https://gerrit.wikimedia.org/r/379544

gerritbot added a comment.Sep 21 2017, 4:17 PM

Change 379559 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] network::constants: add aqs hosts

https://gerrit.wikimedia.org/r/379559

gerritbot added a comment.Sep 22 2017, 8:47 AM

Change 379559 merged by Elukey:
[operations/puppet@production] network::constants: add aqs hosts

https://gerrit.wikimedia.org/r/379559

gerritbot added a comment.Sep 25 2017, 8:59 AM

Change 380449 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Introduce profile::druid::worker

https://gerrit.wikimedia.org/r/380449

gerritbot added a comment.Sep 26 2017, 2:19 PM

Change 380449 merged by Elukey:
[operations/puppet@production] Introduce profile::druid::worker

https://gerrit.wikimedia.org/r/380449

gerritbot added a comment.Sep 26 2017, 6:03 PM

Change 380800 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Improvements to druid profiles, move druid role out of analytics_cluster

https://gerrit.wikimedia.org/r/380800

gerritbot added a comment.Sep 26 2017, 6:10 PM

Change 380804 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] [WIP] Set up separate druid public-eqiad cluster.

https://gerrit.wikimedia.org/r/380804

gerritbot added a comment.Sep 28 2017, 1:04 PM

Change 380800 merged by Elukey:
[operations/puppet@production] Improvements to druid profiles, move druid role out of analytics_cluster

https://gerrit.wikimedia.org/r/380800

gerritbot added a comment.Sep 28 2017, 2:24 PM

Change 381221 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::druid::analytics::worker: split properties into public/private

https://gerrit.wikimedia.org/r/381221

gerritbot added a comment.Sep 28 2017, 2:46 PM

Change 381221 merged by Elukey:
[operations/puppet@production] role::druid::analytics::worker: split properties into public/private

https://gerrit.wikimedia.org/r/381221

gerritbot added a comment.Sep 28 2017, 3:14 PM

Change 381230 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] hieradata: add sites for kafka_jumbo in ganglia_clusters

https://gerrit.wikimedia.org/r/381230

gerritbot added a comment.Sep 28 2017, 3:16 PM

Change 381230 merged by Elukey:
[operations/puppet@production] hieradata: add sites for kafka_jumbo in ganglia_clusters

https://gerrit.wikimedia.org/r/381230

gerritbot added a comment.Sep 28 2017, 4:34 PM

Change 381241 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::druid::*: include profile::druid::common

https://gerrit.wikimedia.org/r/381241

gerritbot added a comment.Sep 28 2017, 4:37 PM

Change 381241 merged by Elukey:
[operations/puppet@production] profile::druid::*: include profile::druid::common

https://gerrit.wikimedia.org/r/381241

Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.Sep 29 2017, 7:49 PM
Nuria renamed this task from Create Druid HTTP basic auth proxy and LVS endpoints for druid, open this to AQS in prod network to Create Druid public cluster such AQS can query druid public data.Oct 2 2017, 3:42 PM
Nuria updated the task description. (Show Details)
gerritbot added a comment.Oct 4 2017, 4:55 PM

Change 378967 merged by Ottomata:
[operations/dns@master] Add druid-analytics LVS svc name

https://gerrit.wikimedia.org/r/378967

Ottomata updated the task description. (Show Details)Oct 4 2017, 5:28 PM
Ottomata changed the point value for this task from 8 to 21.
gerritbot added a comment.Oct 4 2017, 5:30 PM

Change 378956 merged by Ottomata:
[operations/puppet@production] Add LVS service for druid-analytics-broker

https://gerrit.wikimedia.org/r/378956

Ottomata added a comment.Oct 4 2017, 7:09 PM

OOF.

We can't add LVS to stuff in the Analytics VLAN. At least, not so easily. To be discussed tomorrow post standup.

Ottomata mentioned this in T177511: LVS for Druid.Oct 5 2017, 4:04 PM
Ottomata created subtask T177511: LVS for Druid.
Ottomata updated the task description. (Show Details)Oct 5 2017, 4:08 PM
gerritbot added a comment.Oct 9 2017, 4:13 PM

Change 383154 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove druid100[456] from druid analytics cluster.

https://gerrit.wikimedia.org/r/383154

gerritbot added a comment.Oct 9 2017, 4:30 PM

Change 383154 merged by Ottomata:
[operations/puppet@production] Remove druid100[456] from druid analytics cluster.

https://gerrit.wikimedia.org/r/383154

Stashbot added a subscriber: Stashbot.Oct 9 2017, 4:34 PM

Mentioned in SAL (#wikimedia-operations) [2017-10-09T16:34:44Z] <ottomata> beginning decom and reinstall process for druid1004-1006 -- T176223

Ottomata added a comment.Oct 9 2017, 6:07 PM

Today I set druid100[456] as spare::systemm, and stopped druid services there. Depending on the outcome of T177511, they are ready to be repurposed as a new druid cluster.

elukey closed subtask T177511: LVS for Druid as Resolved.Oct 11 2017, 9:36 AM
elukey updated the task description. (Show Details)
elukey updated the task description. (Show Details)
elukey updated the task description. (Show Details)Oct 11 2017, 1:32 PM
elukey updated the task description. (Show Details)Oct 11 2017, 1:37 PM
gerritbot added a comment.Oct 11 2017, 1:58 PM

Change 383568 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/dns@master] Rename 10.2.2.38 to druid-public

https://gerrit.wikimedia.org/r/383568

gerritbot added a comment.Oct 11 2017, 1:59 PM

Change 383568 merged by Ottomata:
[operations/dns@master] Rename 10.2.2.38 to druid-public

https://gerrit.wikimedia.org/r/383568

gerritbot added a comment.Oct 12 2017, 8:04 AM

Change 379533 abandoned by Elukey:
role::analytics_cluster::druid::worker: introduce tlsproxy for druid

https://gerrit.wikimedia.org/r/379533

gerritbot added a comment.Oct 12 2017, 12:07 PM

Change 380804 merged by Elukey:
[operations/puppet@production] Set up separate druid public-eqiad cluster.

https://gerrit.wikimedia.org/r/380804

gerritbot added a comment.Oct 12 2017, 1:29 PM

Change 383833 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::an_cluster::database::meta: allow the druid pulic cluster to use mysql

https://gerrit.wikimedia.org/r/383833

gerritbot added a comment.Oct 12 2017, 1:32 PM

Change 383833 merged by Elukey:
[operations/puppet@production] role::an_cluster::database::meta: allow the druid pulic cluster to use mysql

https://gerrit.wikimedia.org/r/383833

elukey updated the task description. (Show Details)Oct 12 2017, 1:48 PM
Stashbot added a comment.Oct 12 2017, 1:48 PM

Mentioned in SAL (#wikimedia-operations) [2017-10-12T13:48:37Z] <elukey> deployed the new Analytics Public Druid cluster - T176223

gerritbot added a comment.Oct 12 2017, 4:51 PM

Change 383880 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] LVS for druid-public broker and overlord

https://gerrit.wikimedia.org/r/383880

gerritbot added a comment.Oct 13 2017, 9:37 AM

Change 383998 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Rename druid-public to druid-public-broker

https://gerrit.wikimedia.org/r/383998

gerritbot added a comment.Oct 13 2017, 9:49 AM

Change 383998 merged by Elukey:
[operations/dns@master] Rename druid-public to druid-public-broker

https://gerrit.wikimedia.org/r/383998

JAllemandou reassigned this task from JAllemandou to elukey.Oct 13 2017, 10:41 AM
Stashbot added a comment.Oct 16 2017, 4:56 PM

Mentioned in SAL (#wikimedia-operations) [2017-10-16T16:56:23Z] <ottomata> deploying LVS for druid-public-broker https://phabricator.wikimedia.org/T176223

gerritbot added a comment.Oct 16 2017, 4:56 PM

Change 383880 merged by Ottomata:
[operations/puppet@production] Add the LVS config for the druid-public-broker service

https://gerrit.wikimedia.org/r/383880

Ottomata added a comment.Oct 16 2017, 5:03 PM

Alright!

curl http://druid-public-broker.svc.eqiad.wmnet:8082/status
https://config-master.wikimedia.org/pybal/eqiad/druid-public-broker

Ottomata updated the task description. (Show Details)Oct 16 2017, 5:04 PM
elukey added a comment.Oct 18 2017, 1:02 PM

Main thing to remember for the "Create LVS endpoing for druid-public-overlord (for oozie job indexing)" task: usually the target port of a LVS service is meant to be accessed by domain networks, not limited to a subset of hosts. In this case the Indexing service domain should only be accessible by the analytics network, so probably some comments in puppet explaining our use case are needed.

gerritbot added a comment.Oct 25 2017, 5:34 PM

Change 386426 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/dns@master] Add LVS IP for druid-public-overlord

https://gerrit.wikimedia.org/r/386426

gerritbot added a comment.Oct 25 2017, 5:37 PM

Change 386426 merged by Ottomata:
[operations/dns@master] Add LVS IP for druid-public-overlord

https://gerrit.wikimedia.org/r/386426

gerritbot added a comment.Oct 25 2017, 5:38 PM

Change 386427 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add LVS for druid-public-overlord indexing service

https://gerrit.wikimedia.org/r/386427

Ottomata added a parent task: T179027: Puppetize LVS interface IP sets per-DC for easy use in ferm rules.Oct 25 2017, 6:29 PM

We gotta block on T179027 before we can add ferm rules to allow LVS health checks for druid-public-overlord LVS.

Ottomata moved this task from In Progress to Paused on the Analytics-Kanban board.Oct 25 2017, 6:31 PM
Ottomata updated the task description. (Show Details)Nov 20 2017, 4:55 PM

Moved 'Create LVS endpoing for druid-public-overlord (for oozie job indexing)' to its own task so we can close this one. T180971

Ottomata moved this task from Paused to Done on the Analytics-Kanban board.Nov 20 2017, 4:56 PM
Nuria closed this task as Resolved.Nov 27 2017, 9:28 PM
Liuxinyu970226 removed a subscriber: Liuxinyu970226.Nov 30 2017, 11:16 AM
gerritbot added a comment.Jun 10 2019, 7:09 PM

Change 386427 abandoned by Ottomata:
Add LVS for druid-public-overlord indexing service

https://gerrit.wikimedia.org/r/386427

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL