Page MenuHomePhabricator

Port Kafka clients to new jumbo cluster
Closed, ResolvedPublic0 Estimated Story Points

Description

  • webrequest logs (camus, frack): T185136
  • eventlogging analytics: T183297 (processors, consumers, camus, webperf)
  • Discovery: T188408
  • eventlogging eventbus (camus): T189713
  • EventStreams: T185225
  • MediaWiki (avro): T188136

Related Objects

StatusSubtypeAssignedTask
Declinedelukey
ResolvedOttomata
ResolvedLucas_Werkmeister_WMDE
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
DeclinedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
DeclinedNone
Resolved dcausse
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
ResolvedOttomata
DuplicateOttomata

Event Timeline

FYI, kafka-jumbo is now in deployment-prep: deployment-kafka-jumbo-[12].

Change 398468 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::cache::misc: add new vk instance to test TLS traffic

https://gerrit.wikimedia.org/r/398468

Change 398468 abandoned by Elukey:
role::cache::misc: add new vk instance to test TLS traffic

https://gerrit.wikimedia.org/r/398468

Ottomata renamed this task from Port Clients to new jumbo cluster to Port Kafka clients to new jumbo cluster.Jan 8 2018, 9:20 PM
Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)

Change 403059 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Create profile::cache::kafka::certificate class to DRY require of varnishkafka TLS cert

https://gerrit.wikimedia.org/r/403059

Change 403059 merged by Ottomata:
[operations/puppet@production] Create profile::cache::kafka::certificate to DRY require of cert

https://gerrit.wikimedia.org/r/403059

Change 404737 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] [WIP] Produce webrequests from varnishkafka to jumbo Kafka cluster via TLS

https://gerrit.wikimedia.org/r/404737

Change 404789 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] No-op for refinery job camus to ease future analytics -> jumbo kafka

https://gerrit.wikimedia.org/r/404789

Change 404789 merged by Ottomata:
[operations/puppet@production] No-op for refinery job camus to ease future analytics -> jumbo kafka

https://gerrit.wikimedia.org/r/404789

Yeehaw, FYI, all Kafka clients have been ported from analytics to jumbo in deployment-prep in Cloud VPS. EventLogging was a breeze there.

There will be more steps in production (more consumers), but the outlines we have so far should work.

If all is still well tomorrow, I will delete the analytics instances in deployment-prep, which will let me close T174742 and T184235

Change 404737 merged by Ottomata:
[operations/puppet@production] Parameterize webrequests from varnishkafka to jumbo Kafka cluster via TLS

https://gerrit.wikimedia.org/r/404737

Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)
mforns raised the priority of this task from Medium to Needs Triage.Apr 23 2018, 4:19 PM
mforns triaged this task as Medium priority.

Change 441255 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Disable main-eqiad -> analytics MirrorMaker

https://gerrit.wikimedia.org/r/441255

Change 441255 merged by Ottomata:
[operations/puppet@production] Disable main-eqiad -> analytics MirrorMaker

https://gerrit.wikimedia.org/r/441255

Change 441256 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove Kafka MirrorMaker jmxtrans puppetization

https://gerrit.wikimedia.org/r/441256

Change 441256 merged by Ottomata:
[operations/puppet@production] Remove Kafka MirrorMaker jmxtrans puppetization

https://gerrit.wikimedia.org/r/441256

Change 441259 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove unused MirrorMaker old consumer parameters

https://gerrit.wikimedia.org/r/441259

Change 441259 merged by Ottomata:
[operations/puppet@production] Remove unused MirrorMaker old consumer parameters

https://gerrit.wikimedia.org/r/441259

Change 445078 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::cache::canary: move eventlogging vk instance to Jumbo

https://gerrit.wikimedia.org/r/445078

Change 445078 merged by Elukey:
[operations/puppet@production] role::cache::canary: move eventlogging vk instance to Jumbo

https://gerrit.wikimedia.org/r/445078

mforns raised the priority of this task from Medium to High.Mar 11 2019, 3:43 PM
mforns lowered the priority of this task from High to Medium.Mar 11 2019, 3:45 PM