Page MenuHomePhabricator

Port Kafka clients to new jumbo cluster
Closed, ResolvedPublic0 Estimate Story Points

Description

  • webrequest logs (camus, frack): T185136
  • eventlogging analytics: T183297 (processors, consumers, camus, webperf)
  • Discovery: T188408
  • eventlogging eventbus (camus): T189713
  • EventStreams: T185225
  • MediaWiki (avro): T188136

Related Objects

StatusSubtypeAssignedTask
Declinedelukey
ResolvedOttomata
StalledNone
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
DeclinedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
DeclinedNone
Resolveddcausse
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
DuplicateOttomata

Event Timeline

Nuria created this task.Sep 10 2017, 1:40 AM

FYI, kafka-jumbo is now in deployment-prep: deployment-kafka-jumbo-[12].

Nuria moved this task from Next Up to Parent Tasks on the Analytics-Kanban board.Sep 13 2017, 3:11 PM
Nuria moved this task from Parent Tasks to Next Up on the Analytics-Kanban board.

Change 398468 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::cache::misc: add new vk instance to test TLS traffic

https://gerrit.wikimedia.org/r/398468

Change 398468 abandoned by Elukey:
role::cache::misc: add new vk instance to test TLS traffic

https://gerrit.wikimedia.org/r/398468

Ottomata renamed this task from Port Clients to new jumbo cluster to Port Kafka clients to new jumbo cluster.Jan 8 2018, 9:20 PM
Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)

Change 403059 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Create profile::cache::kafka::certificate class to DRY require of varnishkafka TLS cert

https://gerrit.wikimedia.org/r/403059

Change 403059 merged by Ottomata:
[operations/puppet@production] Create profile::cache::kafka::certificate to DRY require of cert

https://gerrit.wikimedia.org/r/403059

Change 404737 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] [WIP] Produce webrequests from varnishkafka to jumbo Kafka cluster via TLS

https://gerrit.wikimedia.org/r/404737

Change 404789 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] No-op for refinery job camus to ease future analytics -> jumbo kafka

https://gerrit.wikimedia.org/r/404789

Change 404789 merged by Ottomata:
[operations/puppet@production] No-op for refinery job camus to ease future analytics -> jumbo kafka

https://gerrit.wikimedia.org/r/404789

Yeehaw, FYI, all Kafka clients have been ported from analytics to jumbo in deployment-prep in Cloud VPS. EventLogging was a breeze there.

There will be more steps in production (more consumers), but the outlines we have so far should work.

Ottomata added a comment.EditedJan 17 2018, 8:55 PM

If all is still well tomorrow, I will delete the analytics instances in deployment-prep, which will let me close T174742 and T184235

Change 404737 merged by Ottomata:
[operations/puppet@production] Parameterize webrequests from varnishkafka to jumbo Kafka cluster via TLS

https://gerrit.wikimedia.org/r/404737

Ottomata updated the task description. (Show Details)Feb 23 2018, 6:35 PM
Ottomata updated the task description. (Show Details)Feb 23 2018, 8:18 PM
Ottomata moved this task from Incoming to Kafka Work on the Analytics board.
Ottomata updated the task description. (Show Details)Mar 8 2018, 3:43 PM
Ottomata updated the task description. (Show Details)Mar 14 2018, 3:14 PM
Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)Mar 14 2018, 3:17 PM
Ottomata updated the task description. (Show Details)Mar 14 2018, 5:53 PM
Ottomata updated the task description. (Show Details)Mar 14 2018, 5:56 PM
Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)Apr 16 2018, 8:10 PM
mforns raised the priority of this task from Medium to Needs Triage.Apr 23 2018, 4:19 PM
mforns triaged this task as Medium priority.
Ottomata updated the task description. (Show Details)Jun 20 2018, 7:41 PM

Change 441255 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Disable main-eqiad -> analytics MirrorMaker

https://gerrit.wikimedia.org/r/441255

Change 441255 merged by Ottomata:
[operations/puppet@production] Disable main-eqiad -> analytics MirrorMaker

https://gerrit.wikimedia.org/r/441255

Change 441256 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove Kafka MirrorMaker jmxtrans puppetization

https://gerrit.wikimedia.org/r/441256

Change 441256 merged by Ottomata:
[operations/puppet@production] Remove Kafka MirrorMaker jmxtrans puppetization

https://gerrit.wikimedia.org/r/441256

Change 441259 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove unused MirrorMaker old consumer parameters

https://gerrit.wikimedia.org/r/441259

Change 441259 merged by Ottomata:
[operations/puppet@production] Remove unused MirrorMaker old consumer parameters

https://gerrit.wikimedia.org/r/441259

Change 445078 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::cache::canary: move eventlogging vk instance to Jumbo

https://gerrit.wikimedia.org/r/445078

Change 445078 merged by Elukey:
[operations/puppet@production] role::cache::canary: move eventlogging vk instance to Jumbo

https://gerrit.wikimedia.org/r/445078

mforns raised the priority of this task from Medium to High.Mar 11 2019, 3:43 PM
mforns lowered the priority of this task from High to Medium.Mar 11 2019, 3:45 PM