Page MenuHomePhabricator
Create Task
Maniphest T175461

Port Kafka clients to new jumbo cluster
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

  • webrequest logs (camus, frack): T185136
  • eventlogging analytics: T183297 (processors, consumers, camus, webperf)
  • Discovery: T188408
  • eventlogging eventbus (camus): T189713
  • EventStreams: T185225
  • MediaWiki (avro): T188136

Details

SubjectRepoBranchLines +/-
role::cache::canary: move eventlogging vk instance to Jumbooperations/puppetproduction+2 -1
Remove unused MirrorMaker old consumer parametersoperations/puppetproduction+2 -54
Remove Kafka MirrorMaker jmxtrans puppetizationoperations/puppetproduction+0 -379
Disable main-eqiad -> analytics MirrorMakeroperations/puppetproduction+15 -78
Parameterize webrequests from varnishkafka to jumbo Kafka cluster via TLSoperations/puppetproduction+65 -20
No-op for refinery job camus to ease future analytics -> jumbo kafkaoperations/puppetproduction+24 -11
Create profile::cache::kafka::certificate to DRY require of certoperations/puppetproduction+62 -52
role::cache::misc: add new vk instance to test TLS trafficoperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 DeclinedelukeyT166833 Produce webrequests from varnishkafka to Kafka with Kafka message timestamp set to configurable content field
 ResolvedOttomataT152015 Provision new Kafka cluster(s) with security features
 ResolvedLucas_Werkmeister_WMDET145712 Use RDF statement counts from entity data, not page props ( wikibase:identifiers, wikibase:statements and wikibase:sitelinks )
 ResolvedOttomataT161731 Create reliable change stream for specific wiki
 ResolvedOttomataT183303 Decomission old analytics kafka cluster
 ResolvedOttomataT175461 Port Kafka clients to new jumbo cluster
 DeclinedOttomataT176352 Port statsv to kafka-jumbo
 ResolvedOttomataT179093 Support multi DC statsv
 ResolvedOttomataT183297 Move EventLogging analytics processes to Kafka jumbo-eqiad cluster
 ResolvedOttomataT185136 Move webrequest varnishkafka and consumers to Kafka jumbo cluster.
Restricted Task
 ResolvedOttomataT187890 Refactor kafkatee module to support multi instance
 ResolvedOttomataT185225 Move EventStreams to main Kafka clusters
 ResolvedOttomataT196553 Support connection/rate limiting in EventStreams
 ResolvedOttomataT188136 Modern Event Platform: Stream Intake Service: Migrate Mediawiki monolog Kafka uses to eventgate-analytics
 DeclinedNoneT126494 Send Mediawiki Kafka logs to Kafka jumbo cluster with TLS encryption
 ResolveddcausseT188408 Migrate mjolnir Kafka clients to use Kafka jumbo
 ResolvedOttomataT189464 Fix Mirror Maker erratic behavior when replicating from main-eqiad to jumbo
 ResolvedOttomataT189611 Alert for Kafka MirrorMaker lag
 ResolvedOttomataT190049 Spike: Consider alternatives to MirrorMaker: uReplicator, Confluent Replicator
 ResolvedOttomataT189713 Migrate eventbus camus to Kafka jumbo
 DuplicateOttomataT189716 Migrate EventStreams to Kafka Jumbo

Event Timeline

• Nuria created this task.Sep 10 2017, 1:40 AM
Ottomata added a comment.Sep 12 2017, 6:57 PM

FYI, kafka-jumbo is now in deployment-prep: deployment-kafka-jumbo-[12].

• Nuria moved this task from Next Up to Parent Tasks on the Analytics-Kanban board.Sep 13 2017, 3:11 PM
• Nuria moved this task from Parent Tasks to Next Up on the Analytics-Kanban board.
Ottomata created subtask T176352: Port statsv to kafka-jumbo.Sep 20 2017, 8:07 PM
Ottomata added a subtask: T179093: Support multi DC statsv.Oct 26 2017, 5:18 PM
Ottomata closed subtask T176352: Port statsv to kafka-jumbo as Declined.
Ottomata added a parent task: T161731: Create reliable change stream for specific wiki.Dec 4 2017, 2:19 PM
gerritbot subscribed.Dec 15 2017, 1:53 PM

Change 398468 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::cache::misc: add new vk instance to test TLS traffic

https://gerrit.wikimedia.org/r/398468

gerritbot added a project: Patch-For-Review.Dec 15 2017, 1:53 PM
gerritbot added a comment.Dec 15 2017, 3:17 PM

Change 398468 abandoned by Elukey:
role::cache::misc: add new vk instance to test TLS traffic

https://gerrit.wikimedia.org/r/398468

Ottomata moved this task from Next Up to Parent Tasks on the Analytics-Kanban board.Dec 19 2017, 9:33 PM
Ottomata renamed this task from Port Clients to new jumbo cluster to Port Kafka clients to new jumbo cluster.Jan 8 2018, 9:20 PM
Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)
gerritbot added a comment.Jan 8 2018, 9:41 PM

Change 403059 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Create profile::cache::kafka::certificate class to DRY require of varnishkafka TLS cert

https://gerrit.wikimedia.org/r/403059

gerritbot added a comment.Jan 9 2018, 2:55 PM

Change 403059 merged by Ottomata:
[operations/puppet@production] Create profile::cache::kafka::certificate to DRY require of cert

https://gerrit.wikimedia.org/r/403059

gerritbot added a comment.Jan 17 2018, 5:43 PM

Change 404737 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] [WIP] Produce webrequests from varnishkafka to jumbo Kafka cluster via TLS

https://gerrit.wikimedia.org/r/404737

gerritbot added a comment.Jan 17 2018, 8:14 PM

Change 404789 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] No-op for refinery job camus to ease future analytics -> jumbo kafka

https://gerrit.wikimedia.org/r/404789

gerritbot added a comment.Jan 17 2018, 8:21 PM

Change 404789 merged by Ottomata:
[operations/puppet@production] No-op for refinery job camus to ease future analytics -> jumbo kafka

https://gerrit.wikimedia.org/r/404789

Ottomata added a comment.Jan 17 2018, 8:55 PM

Yeehaw, FYI, all Kafka clients have been ported from analytics to jumbo in deployment-prep in Cloud VPS. EventLogging was a breeze there.

There will be more steps in production (more consumers), but the outlines we have so far should work.

Ottomata added a comment.EditedJan 17 2018, 8:55 PM

If all is still well tomorrow, I will delete the analytics instances in deployment-prep, which will let me close T174742 and T184235

Ottomata added a subtask: T126494: Send Mediawiki Kafka logs to Kafka jumbo cluster with TLS encryption.Feb 5 2018, 6:16 PM
• Nuria closed subtask T179093: Support multi DC statsv as Resolved.Feb 11 2018, 10:37 PM
gerritbot added a comment.Feb 20 2018, 9:31 PM

Change 404737 merged by Ottomata:
[operations/puppet@production] Parameterize webrequests from varnishkafka to jumbo Kafka cluster via TLS

https://gerrit.wikimedia.org/r/404737

Ottomata updated the task description. (Show Details)Feb 23 2018, 6:35 PM
Ottomata updated the task description. (Show Details)Feb 23 2018, 8:18 PM
Ottomata added a project: Analytics.Mar 1 2018, 6:48 PM
Ottomata moved this task from Incoming to Kafka Work on the Analytics board.
Ottomata removed a project: Analytics-Kanban.Mar 1 2018, 6:54 PM
Ottomata updated the task description. (Show Details)Mar 8 2018, 3:43 PM
Ottomata updated the task description. (Show Details)Mar 14 2018, 3:14 PM
Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)Mar 14 2018, 3:17 PM
dcausse closed subtask T188408: Migrate mjolnir Kafka clients to use Kafka jumbo as Resolved.Mar 14 2018, 4:25 PM
Ottomata updated the task description. (Show Details)Mar 14 2018, 5:53 PM
Ottomata updated the task description. (Show Details)Mar 14 2018, 5:56 PM
Ottomata updated the task description. (Show Details)
• Nuria closed subtask T185136: Move webrequest varnishkafka and consumers to Kafka jumbo cluster. as Resolved.Mar 26 2018, 9:49 PM
• Nuria closed subtask T183297: Move EventLogging analytics processes to Kafka jumbo-eqiad cluster as Resolved.Mar 28 2018, 3:57 PM
Ottomata updated the task description. (Show Details)Apr 16 2018, 8:10 PM
mforns raised the priority of this task from Medium to Needs Triage.Apr 23 2018, 4:19 PM
mforns triaged this task as Medium priority.
Ottomata added a parent task: T183303: Decomission old analytics kafka cluster.May 15 2018, 2:24 PM
• Nuria closed subtask T189713: Migrate eventbus camus to Kafka jumbo as Resolved.Jun 11 2018, 10:38 PM
• Nuria closed subtask T189464: Fix Mirror Maker erratic behavior when replicating from main-eqiad to jumbo as Resolved.Jun 11 2018, 11:04 PM
Ottomata updated the task description. (Show Details)Jun 20 2018, 7:41 PM
gerritbot added a comment.Jun 20 2018, 7:44 PM

Change 441255 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Disable main-eqiad -> analytics MirrorMaker

https://gerrit.wikimedia.org/r/441255

gerritbot added a comment.Jun 20 2018, 7:47 PM

Change 441255 merged by Ottomata:
[operations/puppet@production] Disable main-eqiad -> analytics MirrorMaker

https://gerrit.wikimedia.org/r/441255

gerritbot added a comment.Jun 20 2018, 7:58 PM

Change 441256 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove Kafka MirrorMaker jmxtrans puppetization

https://gerrit.wikimedia.org/r/441256

gerritbot added a comment.Jun 20 2018, 8:08 PM

Change 441256 merged by Ottomata:
[operations/puppet@production] Remove Kafka MirrorMaker jmxtrans puppetization

https://gerrit.wikimedia.org/r/441256

gerritbot added a comment.Jun 20 2018, 8:11 PM

Change 441259 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove unused MirrorMaker old consumer parameters

https://gerrit.wikimedia.org/r/441259

gerritbot added a comment.Jun 20 2018, 8:20 PM

Change 441259 merged by Ottomata:
[operations/puppet@production] Remove unused MirrorMaker old consumer parameters

https://gerrit.wikimedia.org/r/441259

• Nuria closed subtask T185225: Move EventStreams to main Kafka clusters as Resolved.Jun 25 2018, 10:15 PM
gerritbot added a comment.Jul 11 2018, 6:36 AM

Change 445078 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::cache::canary: move eventlogging vk instance to Jumbo

https://gerrit.wikimedia.org/r/445078

gerritbot added a comment.Jul 11 2018, 6:38 AM

Change 445078 merged by Elukey:
[operations/puppet@production] role::cache::canary: move eventlogging vk instance to Jumbo

https://gerrit.wikimedia.org/r/445078

fgiunchedi subscribed.Sep 5 2018, 10:40 AM
mforns raised the priority of this task from Medium to High.Mar 11 2019, 3:43 PM
mforns lowered the priority of this task from High to Medium.Mar 11 2019, 3:45 PM
Milimetric moved this task from Kafka Work to Event Platform on the Analytics board.May 6 2019, 4:01 PM
Ottomata closed subtask T126494: Send Mediawiki Kafka logs to Kafka jumbo cluster with TLS encryption as Declined.May 29 2019, 9:49 AM
Ottomata closed this task as Resolved.Jun 24 2019, 7:31 PM
Ottomata claimed this task.
Ottomata closed subtask T188136: Modern Event Platform: Stream Intake Service: Migrate Mediawiki monolog Kafka uses to eventgate-analytics as Resolved.
Ottomata updated the task description. (Show Details)

Also WOW

Aklapper removed a project: Analytics.Jul 4 2020, 7:59 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits