Page MenuHomePhabricator

Port Kafka clients to new jumbo cluster
Closed, ResolvedPublic0 Estimated Story Points

Description

  • webrequest logs (camus, frack): T185136
  • eventlogging analytics: T183297 (processors, consumers, camus, webperf)
  • Discovery: T188408
  • eventlogging eventbus (camus): T189713
  • EventStreams: T185225
  • MediaWiki (avro): T188136

Related Objects

StatusSubtypeAssignedTask
Declinedelukey
ResolvedOttomata
OpenLucas_Werkmeister_WMDE
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
DeclinedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
DeclinedNone
Resolveddcausse
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
DuplicateOttomata

Event Timeline

Nuria created this task.Sep 10 2017, 1:40 AM

FYI, kafka-jumbo is now in deployment-prep: deployment-kafka-jumbo-[12].

Nuria moved this task from Next Up to Parent Tasks on the Analytics-Kanban board.Sep 13 2017, 3:11 PM
Nuria moved this task from Parent Tasks to Next Up on the Analytics-Kanban board.

Change 398468 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::cache::misc: add new vk instance to test TLS traffic

https://gerrit.wikimedia.org/r/398468

Change 398468 abandoned by Elukey:
role::cache::misc: add new vk instance to test TLS traffic

https://gerrit.wikimedia.org/r/398468

Ottomata renamed this task from Port Clients to new jumbo cluster to Port Kafka clients to new jumbo cluster.Jan 8 2018, 9:20 PM
Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)

Change 403059 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Create profile::cache::kafka::certificate class to DRY require of varnishkafka TLS cert

https://gerrit.wikimedia.org/r/403059

Change 403059 merged by Ottomata:
[operations/puppet@production] Create profile::cache::kafka::certificate to DRY require of cert

https://gerrit.wikimedia.org/r/403059

Change 404737 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] [WIP] Produce webrequests from varnishkafka to jumbo Kafka cluster via TLS

https://gerrit.wikimedia.org/r/404737

Change 404789 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] No-op for refinery job camus to ease future analytics -> jumbo kafka

https://gerrit.wikimedia.org/r/404789

Change 404789 merged by Ottomata:
[operations/puppet@production] No-op for refinery job camus to ease future analytics -> jumbo kafka

https://gerrit.wikimedia.org/r/404789

Yeehaw, FYI, all Kafka clients have been ported from analytics to jumbo in deployment-prep in Cloud VPS. EventLogging was a breeze there.

There will be more steps in production (more consumers), but the outlines we have so far should work.

Ottomata added a comment.EditedJan 17 2018, 8:55 PM

If all is still well tomorrow, I will delete the analytics instances in deployment-prep, which will let me close T174742 and T184235

Change 404737 merged by Ottomata:
[operations/puppet@production] Parameterize webrequests from varnishkafka to jumbo Kafka cluster via TLS

https://gerrit.wikimedia.org/r/404737

Ottomata updated the task description. (Show Details)Feb 23 2018, 6:35 PM
Ottomata updated the task description. (Show Details)Feb 23 2018, 8:18 PM
Ottomata moved this task from Incoming to Kafka Work on the Analytics board.
Ottomata updated the task description. (Show Details)Mar 8 2018, 3:43 PM
Ottomata updated the task description. (Show Details)Mar 14 2018, 3:14 PM
Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)Mar 14 2018, 3:17 PM
Ottomata updated the task description. (Show Details)Mar 14 2018, 5:53 PM
Ottomata updated the task description. (Show Details)Mar 14 2018, 5:56 PM
Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)Apr 16 2018, 8:10 PM
mforns raised the priority of this task from Medium to Needs Triage.Apr 23 2018, 4:19 PM
mforns triaged this task as Medium priority.
Ottomata updated the task description. (Show Details)Jun 20 2018, 7:41 PM

Change 441255 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Disable main-eqiad -> analytics MirrorMaker

https://gerrit.wikimedia.org/r/441255

Change 441255 merged by Ottomata:
[operations/puppet@production] Disable main-eqiad -> analytics MirrorMaker

https://gerrit.wikimedia.org/r/441255

Change 441256 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove Kafka MirrorMaker jmxtrans puppetization

https://gerrit.wikimedia.org/r/441256

Change 441256 merged by Ottomata:
[operations/puppet@production] Remove Kafka MirrorMaker jmxtrans puppetization

https://gerrit.wikimedia.org/r/441256

Change 441259 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove unused MirrorMaker old consumer parameters

https://gerrit.wikimedia.org/r/441259

Change 441259 merged by Ottomata:
[operations/puppet@production] Remove unused MirrorMaker old consumer parameters

https://gerrit.wikimedia.org/r/441259

Change 445078 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::cache::canary: move eventlogging vk instance to Jumbo

https://gerrit.wikimedia.org/r/445078

Change 445078 merged by Elukey:
[operations/puppet@production] role::cache::canary: move eventlogging vk instance to Jumbo

https://gerrit.wikimedia.org/r/445078

mforns raised the priority of this task from Medium to High.Mar 11 2019, 3:43 PM
mforns lowered the priority of this task from High to Medium.Mar 11 2019, 3:45 PM
Aklapper removed a project: Analytics.Jul 4 2020, 7:59 AM