Page MenuHomePhabricator

Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events
Closed, ResolvedPublic

Description

We need a way to identify active streams for ingestion into Hadoop. We also need a way to monitor them to A. be sure that desired data is being ingested properly and B. that all data for a given hour has been imported.

We currently use a complicated regex topic include list with Camus and Refine to do ingestion into Hive. This task is about replacing that list with something smarter, and using it to build both automated ingestion and monitoring.

This is relevant to both T249261: Vertical: Migrate SearchSatisfaction EventLogging event stream to Event Platform and T241241: Produce an instrumentation event stream using new EPC and EventGate from client side browsers, and will also help improve our present system. This task is not about replacing Camus; that will be done in T238400: Evaluate possible replacements for Camus: Gobblin, Marmaray, Kafka Connect HDFS, etc.

I'll describe the current state of ingestion and its problems here in the description, and then work out ideas in the comments.


How things work now

Camus: Kafka -> HDFS raw JSON data

We have 3 different Camus jobs for event data. Each job selects which topics to import via topic include and exclude regexes.

https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/manifests/analytics/refinery/job/camus.pp

Camus Job 'eventlogging'

Topics like 'eventlogging_SearchSatisfaction' are ingested into /wmf/data/raw/eventlogging.

The per topic imported directories in end up looking like
/wmf/data/raw/eventlogging/eventlogging_SearchSatisfaction/hourly/2020/04/10/10

Camus Job 'mediawiki_events'

Topics like 'eqiad.mediawiki.revision-create' are ingested into /wmf/data/raw/event.

The per topic imported directories in end up looking like
/wmf/data/raw/event/eqiad_mediawiki_revision-create/hourly/2020/04/10/10

Camus Job 'mediawiki_analytics_events'

High volume topics like 'eqiad.mediawiki.api-request' are ingested into /wmf/data/raw/event by a separate Camus job. High volume topics take longer to import, so we run a separate job to keep them from starving out the lower volume ones.

The per topic camus imported directories in end up looking like
/wmf/data/raw/event/eqiad_mediawiki_api-request/hourly/2020/04/10/10

Camus Job 'mediawiki_job'

Topics like 'eqiad.mediawiki.job.refreshLinks' are ingested into /wmf/data/raw/mediawiki_job.

The per topic imported directories in end up looking like
/wmf/data/raw/mediawiki_job/eqiad_mediawiki_job_refreshLinks/hourly/2020/04/10/10

Refine: HDFS raw JSON data -> Hive (parquet) tables (augmented by TransformFunctions)

These jobs import raw JSON data from Camus imported HDFS directories into parquet backed Hive tables. There are separate jobs so we can specify different base camus directories (e.g /wmf/data/raw/eventlogging vs /wmf/data/raw/event) and different Hive partition capture group regexes (e.g. eventlogging_(.+) vs (eqiad|codfw)_(.+)).

https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/manifests/analytics/refinery/job/refine.pp

Refine job 'eventlogging_analytics'

Uses schemas from meta.wikimedia.org and refines from
/wmf/data/raw/eventlogging -> /wmf/data/event into the Hive event database with some tables like 'Edit' excluded.
Camus per topic directories in /wmf/data/raw/eventlogging look like 'eventlogging_SearchSatisfaction', and table names are extracted via eventlogging_(.+).

Refine job 'event'

Uses schemas from schema.discovery.wmnet and refines from
/wmf/data/raw/event -> /wmf/data/event into the Hive event database with some tables like 'mediawiki_recentchange' excluded.
Camus per topic directories in /wmf/data/raw/event look like 'eqiad_mediawiki_revision-create',
and datacenter and table names are extracted via (eqiad|codfw)_(.+)

Refine job 'mediawiki_job_events'

Infers schemas from JSON data and refines from
/wmf/data/raw/mediawiki_job -> /wmf/data/event into the Hive event database with some tables like 'EchoNotificationJob' excluded.
Camus per topic directories in /wmf/data/raw/mediawiki_job look like 'eqiad_mediawiki_job_refreshLinks'
and datacenter and table names are extracted via (eqiad|codfw)_(.+).

Refine job 'eventlogging_legacy'

This job is soon TBD, but I thought I'd include it here to illustrate what we need for the migration process. We'll need to add a new Refine job for T238230: Decommission EventLogging backend components by migrating to MEP that uses schema.discovery.wmnet to lookup migrated legacy EventLogging schemas instead of meta.wikimedia.org. This new Refine job will eventually replace 'eventlogging_analytics' once all EventLogging streams are migrated to MEP.


Problems:

  • Legacy EventLogging topics are not prefixed with the datacenter name. Their Hive tables do not have a datacenter partition. This fact makes their ingestion jobs harder to to unify with the other event ingestion jobs.
  • There is no consistent way to identify which topics should be imported. This is mostly a Camus problem, but could also be one for Refine too. Refine can be configured to consider all data it finds in a base directory path (with some hardcoded exclude lists). But The data in HDFS that Refine uses is put there by Camus, and Camus sees all topics in Kafka as potential for ingesting to HDFS. We need a way to dynamically configure each Camus job's topic include list. I started this work in https://gerrit.wikimedia.org/r/c/analytics/refinery/+/593047, but on more thought I'm not sure that will be enough.

Outcome

  • EventStreamConfig is used to declare streams and whether or not canary events are enabled.
  • wikimedia-event-utilties, a Java library for discovering streams and configs was developed.
  • A ProduceCanaryEvents job was created in refinery-source to produce canary events to specific streams once per hour.
  • Camus was modified to integrate with EventStreamConfig via wikimedia-event-utilities to discover streams and topics to ingest.
  • CamusPartitionChecker was modified to integrate with EventStreamConfig to automatically check topics that have canary events enabled.
  • All 3 event service Camus jobs use EventStreamConfig for stream topic discovery.
  • CamusPartitionChecker for eventgate-analytics-external and eventgate-analytics (but not eventgate-main) is configured to only check stream topics that have canary events enabled.

All this together means that:

  • An event stream that is added to $wgEventStreams (AKA stream config) will automatically be ingested by a Camus job.
  • Any stream with canary_events_enabled: true will automatically have canary events produce to it once an hour, and will be monitored to be sure that some data is ingested each hour for those streams.

Details

SubjectRepoBranchLines +/-
operations/puppetproduction+1 -1
analytics/refinery/sourcemaster+63 -12
operations/puppetproduction+1 -1
analytics/refinerymaster+21 -25
analytics/refinerymaster+3 -23
operations/puppetproduction+1 -33
operations/puppetproduction+1 -4
operations/mediawiki-configmaster+4 -0
operations/puppetproduction+3 -1
operations/puppetproduction+24 -51
operations/puppetproduction+23 -26
operations/mediawiki-configmaster+5 -2
operations/puppetproduction+1 -1
operations/puppetproduction+1 -0
operations/puppetproduction+35 -3
operations/puppetproduction+3 -4
operations/puppetproduction+2 -0
operations/puppetproduction+4 -4
operations/puppetproduction+3 -1
operations/puppetproduction+63 -18
operations/puppetproduction+1 -1
operations/puppetproduction+5 -5
operations/puppetproduction+1 -0
operations/puppetproduction+1 -0
operations/puppetproduction+90 -34
analytics/refinerymaster+2 -1
analytics/refinery/sourcemaster+82 -27
analytics/camuswmf+40 -25
analytics/camuswmf+217 -23
wikimedia-event-utilitiesmaster+233 -5
operations/puppetproduction+1 -0
schemas/event/secondarymaster+74 -70
operations/mediawiki-configmaster+1 -0
operations/puppetproduction+223 -2
mediawiki/extensions/EventStreamConfigwmf/1.36.0-wmf.8+1 -38
mediawiki/extensions/EventStreamConfigwmf/1.36.0-wmf.8+38 -1
mediawiki/extensions/EventStreamConfigmaster+38 -1
mediawiki/extensions/EventStreamConfigmaster+4 -1
operations/mediawiki-configmaster+5 -11
operations/puppetproduction+46 -10
analytics/refinery/sourcemaster+295 -15
analytics/refinery/sourcemaster+15 -1 K
wikimedia-event-utilitiesmaster+643 -441
wikimedia-event-utilitiesmaster+434 -1
integration/configmaster+9 -0
wikimedia-event-utilitiesrefs/meta/config+3 -0
integration/configmaster+14 -0
operations/deployment-chartsmaster+210 -18
analytics/refinery/sourcemaster+4 K -25
mediawiki/extensions/EventStreamConfigmaster+99 -25
analytics/refinery/sourcemaster+280 -156
analytics/refinery/sourcemaster+273 -152
mediawiki/extensions/EventStreamConfigmaster+294 -5
operations/deployment-chartsmaster+12 -12
analytics/refinerymaster+1 -1
operations/puppetproduction+84 -7
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 625961 merged by Ottomata:
[mediawiki/extensions/EventStreamConfig@master] Make canary_events_enabled an 'internal' setting

https://gerrit.wikimedia.org/r/625961

Change 625962 merged by Ottomata:
[mediawiki/extensions/EventStreamConfig@master] Default to using API json formatversion=2

https://gerrit.wikimedia.org/r/625962

Change 626375 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/EventStreamConfig@wmf/1.36.0-wmf.8] Default to using API json formatversion=2

https://gerrit.wikimedia.org/r/626375

Change 626375 merged by jenkins-bot:
[mediawiki/extensions/EventStreamConfig@wmf/1.36.0-wmf.8] Default to using API json formatversion=2

https://gerrit.wikimedia.org/r/626375

Change 626433 had a related patch set uploaded (by Urbanecm; owner: Urbanecm):
[mediawiki/extensions/EventStreamConfig@wmf/1.36.0-wmf.8] Revert "Default to using API json formatversion=2"

https://gerrit.wikimedia.org/r/626433

Change 626433 merged by Urbanecm:
[mediawiki/extensions/EventStreamConfig@wmf/1.36.0-wmf.8] Revert "Default to using API json formatversion=2"

https://gerrit.wikimedia.org/r/626433

Mentioned in SAL (#wikimedia-operations) [2020-09-14T18:30:08Z] <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.8/extensions/EventStreamConfig/includes/: a4c86089371319ae5a3bb6053c4a9b3e83130286: Default to using API json formatversion=2 (T251609) (duration: 00m 57s)

Change 624168 merged by Ottomata:
[operations/puppet@production] Canary events refinery job

https://gerrit.wikimedia.org/r/624168

Change 627511 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] wgEventStreams - enable canary events for eventlogging_TemplateWizard

https://gerrit.wikimedia.org/r/627511

Change 627511 merged by jenkins-bot:
[operations/mediawiki-config@master] wgEventStreams - enable canary events for eventlogging_TemplateWizard

https://gerrit.wikimedia.org/r/627511

Mentioned in SAL (#wikimedia-operations) [2020-09-15T14:21:28Z] <otto@deploy1001> sync-file aborted: wgEventStreams: Set canary_events_enabled: true for eventlogging_TemplateWizard - T251609 (duration: 00m 06s)

Mentioned in SAL (#wikimedia-operations) [2020-09-15T14:22:31Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: wgEventStreams: Set canary_events_enabled: true for eventlogging_TemplateWizard - T251609 (duration: 00m 56s)

Canary events are now being produced for all test streams, as well as eventlogging_TemplateWizard.

Change 627584 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[schemas/event/secondary@master] Add legacy "schema" field to eventlogging legacy examples

https://gerrit.wikimedia.org/r/627584

Change 627584 merged by Ottomata:
[schemas/event/secondary@master] Add legacy "schema" field to eventlogging legacy examples

https://gerrit.wikimedia.org/r/627584

Change 627876 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] refine.pp - use eventlogging_legacy job to refine Test schema events

https://gerrit.wikimedia.org/r/627876

Change 627876 merged by Ottomata:
[operations/puppet@production] refine.pp - use eventlogging_legacy job to refine Test schema events

https://gerrit.wikimedia.org/r/627876

Change 628349 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery@master] eventstreamconfig.py - remove custom logic for computing topic lists

https://gerrit.wikimedia.org/r/628349

Change 628416 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[wikimedia-event-utilities@master] Add toRegex static function to EventStreamConfig

https://gerrit.wikimedia.org/r/628416

Change 628447 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/camus@wmf] [WIP] Add option to use Wikimedia EventStreamConfig to get kafka topics to ingest

https://gerrit.wikimedia.org/r/628447

Change 629377 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] [WIP] Use EventStreamConfig in CamusPartitionChecker

https://gerrit.wikimedia.org/r/629377

Change 628416 merged by Ottomata:
[wikimedia-event-utilities@master] Add functions to EventStreamFactory and EventStreamConfig to match streams by settings

https://gerrit.wikimedia.org/r/628416

Change 628447 merged by Ottomata:
[analytics/camus@wmf] Add option to use Wikimedia EventStreamConfig to get kafka topics to ingest

https://gerrit.wikimedia.org/r/628447

Change 633800 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery@master] Add camus-wmf-0.1.0-wmf11.jar with EventStreamConfig support

https://gerrit.wikimedia.org/r/633800

Change 634067 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/camus@wmf] Safeguards when using EventStreamConfig in EtlInputFormat

https://gerrit.wikimedia.org/r/634067

Change 634067 merged by Ottomata:
[analytics/camus@wmf] Safeguards when using EventStreamConfig in EtlInputFormat

https://gerrit.wikimedia.org/r/634067

Change 629377 merged by jenkins-bot:
[analytics/refinery/source@master] Use camus + EventStreamConfig integration in CamusPartitionChecker

https://gerrit.wikimedia.org/r/629377

Change 634266 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus::job - replace check_whitelist_topics with check_java_opts

https://gerrit.wikimedia.org/r/634266

Change 633800 merged by Ottomata:
[analytics/refinery@master] Add camus-wmf-0.1.0-wmf12.jar with EventStreamConfig support

https://gerrit.wikimedia.org/r/633800

Change 634266 merged by Ottomata:
[operations/puppet@production] camus::job - replace check_whitelist_topics with check_java_opts

https://gerrit.wikimedia.org/r/634266

Change 635545 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus::job - don't fail if dynamic_stream_configs is set without kafka.whitelist.topics

https://gerrit.wikimedia.org/r/635545

Change 635545 merged by Ottomata:
[operations/puppet@production] camus::job - don't fail if dynamic_stream_configs is set

https://gerrit.wikimedia.org/r/635545

Change 635549 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus::job - eg-analytics-external: add missing check_java_opts for kafka.whitelist.topics

https://gerrit.wikimedia.org/r/635549

Change 635549 merged by Ottomata:
[operations/puppet@production] camus::job - eg-analytics-external: add missing check_java_opts

https://gerrit.wikimedia.org/r/635549

Change 635553 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus::job - use double quotes around java opts value

https://gerrit.wikimedia.org/r/635553

Change 635553 merged by Ottomata:
[operations/puppet@production] camus::job - use double quotes around java opts value

https://gerrit.wikimedia.org/r/635553

Change 635561 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus - Bump check_jar version to refinery 0.0.137

https://gerrit.wikimedia.org/r/635561

Change 635561 merged by Ottomata:
[operations/puppet@production] camus - Bump check_jar version to refinery 0.0.137

https://gerrit.wikimedia.org/r/635561

Change 635572 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus - Refactor http proxy envs

https://gerrit.wikimedia.org/r/635572

Change 635572 merged by Ottomata:
[operations/puppet@production] camus - Refactor http proxy envs

https://gerrit.wikimedia.org/r/635572

Change 635618 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus::job - use \s for spaces in systemd unit Environment var

https://gerrit.wikimedia.org/r/635618

Change 635618 merged by Ottomata:
[operations/puppet@production] camus::job - use \s for spaces in systemd unit Environment var

https://gerrit.wikimedia.org/r/635618

Change 635619 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus::job - only use \s in systemd Environment, regular CLI needs regular spaces

https://gerrit.wikimedia.org/r/635619

Change 635619 merged by Ottomata:
[operations/puppet@production] camus::job - only use \s in systemd Environment, regular CLI needs spaces

https://gerrit.wikimedia.org/r/635619

Change 635620 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Include profile::analytics::refinery::event_service_config in camus test

https://gerrit.wikimedia.org/r/635620

Change 635620 merged by Ottomata:
[operations/puppet@production] Include profile::analytics::refinery::event_service_config in camus test

https://gerrit.wikimedia.org/r/635620

Change 635626 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus test - use proper eventstreamconfig.stream_names property

https://gerrit.wikimedia.org/r/635626

Change 635626 merged by Ottomata:
[operations/puppet@production] camus test - use proper eventstreamconfig.stream_names property

https://gerrit.wikimedia.org/r/635626

Change 635632 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus - use eventstreamconfig for eventgate-analytics-external streams

https://gerrit.wikimedia.org/r/635632

Change 635632 merged by Ottomata:
[operations/puppet@production] camus - use eventstreamconfig for eventgate-analytics-external streams

https://gerrit.wikimedia.org/r/635632

Change 635817 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus - bump camus jar version for eventstreamconfig event jobs to wmf12

https://gerrit.wikimedia.org/r/635817

Change 635817 merged by Ottomata:
[operations/puppet@production] camus - bump camus jar version for eventstreamconfig event jobs to wmf12

https://gerrit.wikimedia.org/r/635817

Change 635820 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus - take 2 bump camus jar version for eventstreamconfig event jobs to wmf12

https://gerrit.wikimedia.org/r/635820

Change 635820 merged by Ottomata:
[operations/puppet@production] camus - take 2 bump camus jar version for eventstreamconfig event jobs to wmf12

https://gerrit.wikimedia.org/r/635820

Change 635822 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable canary events for 3 eventgate-analytics bound streams

https://gerrit.wikimedia.org/r/635822

Change 635822 merged by Ottomata:
[operations/mediawiki-config@master] Enable canary events for 3 eventgate-analytics bound streams

https://gerrit.wikimedia.org/r/635822

Mentioned in SAL (#wikimedia-operations) [2020-10-22T12:24:20Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: wgEventStreams: Enable canary events for 3 eventgate-analytics bound streams - T251609 (duration: 01m 05s)

Change 635824 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus - use eventstreamconfig for eventgate-analytics streams

https://gerrit.wikimedia.org/r/635824

Change 635824 merged by Ottomata:
[operations/puppet@production] camus - use eventstreamconfig for eventgate-analytics streams

https://gerrit.wikimedia.org/r/635824

Mentioned in SAL (#wikimedia-analytics) [2020-10-22T13:04:27Z] <ottomata> camus-eventgate-analytics_events now uses EventStreamConfig to discovery topics to ingest and canary topics to monitor - T251609

Change 635832 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable canary events for all eventgate-analytics-external bound streams

https://gerrit.wikimedia.org/r/635832

Change 635836 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus - use eventstreamconfig for eventgate-main streams

https://gerrit.wikimedia.org/r/635836

Change 635836 merged by Ottomata:
[operations/puppet@production] camus - use eventstreamconfig for eventgate-main streams

https://gerrit.wikimedia.org/r/635836

Change 635837 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus - add extra backslash escapes for regex in systemd unit

https://gerrit.wikimedia.org/r/635837

Change 635837 merged by Ottomata:
[operations/puppet@production] camus - add extra backslash escapes for regex in systemd unit

https://gerrit.wikimedia.org/r/635837

Mentioned in SAL (#wikimedia-analytics) [2020-10-22T13:56:12Z] <ottomata> camus-eventgate-main_events now uses EventStreamConfig to discover topics to ingest, but still uses regex to find topics to monitor - T251609

Change 635832 merged by Ottomata:
[operations/mediawiki-config@master] Enable canary events for all eventgate-analytics-external bound streams

https://gerrit.wikimedia.org/r/635832

Mentioned in SAL (#wikimedia-operations) [2020-10-22T14:00:45Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: wgEventStreams: Enable canary events for all eventgate-analytics-external bound streams - T251609 (duration: 01m 02s)

Change 635841 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus - bump to camus jar version wmf12 for all camus jobs

https://gerrit.wikimedia.org/r/635841

Change 635841 merged by Ottomata:
[operations/puppet@production] camus - bump to camus jar version wmf12 for all camus jobs

https://gerrit.wikimedia.org/r/635841

Mentioned in SAL (#wikimedia-operations) [2020-10-22T14:05:27Z] <ottomata> bump camus version to wmf12 for all camus jobs. should be no-op now. - T251609

Mentioned in SAL (#wikimedia-analytics) [2020-10-22T14:05:44Z] <ottomata> bump camus version to wmf12 for all camus jobs. should be no-op now. - T251609

Status:

Canary events are enabled for all eventgate-analytics-external stream and most eventgate-analytics streams. Stream topics for ingestion are discovered via EventStreamConfig for all camus jobs, and stream topics to monitor the ones with canary events enabled for eventgate-analytics-external and eventgate-analytics. eventgate-main uses EventStreamConfig for stream topic discovery, but not for monitoring (as we don't (yet) emit canary events into eventgate-main streams, other than test.event).

Change 635846 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery@master] bin/camus - Remove support for 'dynamic-stream-config'

https://gerrit.wikimedia.org/r/635846

Change 635847 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus::job - Remove support for 'dynamic-stream-config'

https://gerrit.wikimedia.org/r/635847

Change 635847 merged by Ottomata:
[operations/puppet@production] camus::job - Remove support for 'dynamic-stream-config'

https://gerrit.wikimedia.org/r/635847

Change 635846 merged by Ottomata:
[analytics/refinery@master] bin/camus - Remove support for 'dynamic-stream-config'

https://gerrit.wikimedia.org/r/635846

Change 628349 merged by Ottomata:
[analytics/refinery@master] eventstreamconfig.py - remove custom logic for computing topic lists

https://gerrit.wikimedia.org/r/628349

Change 635877 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus - run job 5 minutes earlier to not conflict with timing of canary events

https://gerrit.wikimedia.org/r/635877

Change 635877 merged by Ottomata:
[operations/puppet@production] camus - run job 5 minutes earlier to not conflict with timing of canary events

https://gerrit.wikimedia.org/r/635877

Change 651542 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] Add Refine TransformFunction to remove canary events

https://gerrit.wikimedia.org/r/651542

Change 651542 merged by Ottomata:
[analytics/refinery/source@master] Add Refine TransformFunction to remove canary events

https://gerrit.wikimedia.org/r/651542

Change 654308 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Bump refine jar version to refinery-job 0.0.143

https://gerrit.wikimedia.org/r/654308

Change 654308 merged by Razzi:
[operations/puppet@production] Bump refine jar version to refinery-job 0.0.143

https://gerrit.wikimedia.org/r/654308