Page MenuHomePhabricator
Create Task
Maniphest T251935

All EventGate instances should use EventStreamConfig
Closed, ResolvedPublic
Actions

Description

In https://phabricator.wikimedia.org/T251609#6100233 we are realizing that we really need all streams, including those statically configured for 'production' EventGate instances, in a centralized place. eventgate-analytics-external already uses dynamic stream config via the EventStreamConfig MW API.

All other instances use a statically configured stream config to map stream names to allowed schema titles. We could just have them use dynamic stream config too, but I don't want to couple critical EventGate instances to MW API at runtime. Instead, we could build the static stream for EventGate at deploy time. We can either do this as part of the chart generation, or more simply with a helper script that can be manually called to render out stream configs into your local deployment-charts helmfile values.

Keeping static configs that are manually rendered from the dynamic centralized ones seems the safest way to accomplish this.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+2 -34camus - Remove now unused mediawiki_events job
operations/puppetproduction+5 -0camus - Exclude mediawiki.job streams from eventgate-main_events job
operations/puppetproduction+17 -4camus - replace mediawiki_events job with eventgate-main_events job
operations/deployment-chartsmaster+30 -88eventgate-main - Use MW EventStreamConfig API
operations/deployment-chartsmaster+12 -12eventgate-* - bump to version 2020-08-12-144657-production
eventgate-wikimediamaster+2 -2Bump eventgate dependency to e7cfa6d to pick up bug fix for test events
operations/deployment-chartsmaster+15 -44eventgate-main - use MW EventStreamConfig in staging
operations/mediawiki-configmaster+105 -1EventStreamConfig - add streams for eventgate-main
operations/deployment-chartsmaster+30 -26eventgate-analytics - Use remote EventStreamConfig in eqiad and codfw
operations/puppetproduction+29 -51camus - include mediawiki.api-request in eventgate-analytics topics to check
operations/puppetproduction+15 -30camus - replace mediawiki_analytics_events with eventgate-analytics_events job
operations/mediawiki-configmaster+0 -5EventStreamConfig - Remove extraneous mediawiki.api-request stream
operations/puppetproduction+2 -2camus::job - Fix typo in stream_configs_constraints_opt
operations/puppetproduction+56 -5camus - declare event service specific camus::jobs.
operations/puppetproduction+32 -20camus::job - pass stream_configs_constraints through to refinery camus wrapper
analytics/refinerymaster+22 -7Add constraints parameter when using bin/camus with EventStreamConfig API
operations/mediawiki-configmaster+35 -1Add eventgate-analytics stream config
operations/deployment-chartsmaster+15 -13eventgate-analytics - Use MW EventStreamConfig API, only in staging for now
operations/deployment-chartsmaster+34 -14eventgate-logging-external: use remote stream config, eventgate-analytics-external: use constraints
eventgate-wikimediamaster+5 -2Apply stream_config_uri_options for static stream config URIs too
operations/deployment-chartsmaster+12 -12eventgate-* - precache /test/event/1.0.0 schema
operations/mediawiki-configmaster+2 -2wgEventStreams - fix another typo in eventgate stream config
operations/mediawiki-configmaster+2 -2wgEventStreams - fix typo in eventgate stream config
operations/deployment-chartsmaster+52 -33eventgate - use /v1/_test/events route for readinessProbe
operations/mediawiki-configmaster+86 -39Add eventgate service specific test.event streams
eventgate-wikimediamaster+24 -7Bump eventgate version and to support /v1/_test/events route
operations/deployment-chartsmaster+15 -5eventgate-logging-external - Use MW EventStreamConfig API to get static stream configs
operations/mediawiki-configmaster+56 -8Add eventgate-logging-external streams, and add destination_event_service to all stream configs
eventgate-wikimediamaster+7 -2Split :streams param on , or |
mediawiki/extensions/EventStreamConfigmaster+227 -12Add constraints API parameter to filter by settings constraints
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Ottomata added a comment.EditedMay 5 2020, 6:05 PM

Hm, but then again, right now static stream config is manually declared for each EventGate instance. eventgate-main only accepts defined streams, eventgate-logging only defined streams, etc. If we make all EventGate instances use the same centralized stream config, e.g. eventgate-logging will accept mediawiki.revision-create events. In practice this won't likely be a problem as producers will be configured correctly. But if there is some bug that say causes EventBus to produce to eventgate-analytics, events will be accepted and produced to Kafka jumbo instead of Kafka main.

EventGate could ensure that the stream config event_service setting matches its service name. E.g. if stream mediawiki.revision-create has event_service: eventgate-main and service-runner has its service name as 'eventgate-main', the event will be allowed.

This is all feeling very brittle.

Ottomata added a comment.May 5 2020, 6:19 PM

Let's keep discussion of this on T251609. Moving some of ^ there.

Ottomata mentioned this in T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events.May 5 2020, 6:19 PM
Milimetric triaged this task as High priority.May 7 2020, 3:35 PM
Milimetric moved this task from Incoming to Event Platform on the Analytics board.
Ottomata closed this task as Declined.May 7 2020, 4:01 PM

After discussion the Event Platform Engineering sync yesterday, we all agreed that this is hard and we don't have a good solution. I'm going to decline this task for now. If we come up with something amazing we'll reopen.

Ottomata reopened this task as Open.EditedMay 27 2020, 6:26 PM
Ottomata added a subscriber: mpopov.

Had a discussion with @mpopov about this in IRC today, and then discussed it more with the Analytics team. I'd like to reopen this and reconsider.

We declined this for 2 reasons:
A. Each eventgate-wikimedia instance needs to restrict which streams can be produced, if we centralize the stream config, how do we do that?
B. We support regex based stream configs. Even if we centralize all stream configs in wgEventStreams we still don't get a complete list of all active streams anywhere.

In T251609: Automate ingestion and refinement into Hive of event data from Kafka using stream configs and canary/heartbeat events we were going to work around B. by starting with the list of all Kafka topics in jumbo-eqiad and then querying each eventgate-wikimedia instance for stream config. @mpopov came into this discussion because he wanted to have a complete list of streams for his own purposes, and I think that will be something that is useful outside of just analytics cluster use cases too. So, I'd like to revisit the idea of centralizing the stream configs in wgEventStreams for more general usage.

I don't think we can do much about problem B. The regex stream names are a bit of a pain, but they are only really used for mediawiki.job.* streams. We don't need to ingest those in Analytics, so perhaps we can ignore them for now, and just accept that streams that use regexes in config won't be as automate-able as the other ones are.

For problem A. we had originally considered hacking together some other stream config that an eventgate-wikimedia instance could use to restrict which streams it is allowed to produce. I didn't like the idea of implicitly using the service name...but what if made this more explicit. Suggestion: add a stream_config_allowed_settings configuration to eventgate-wikimedia. E.g.

stream_config_constraints:
  destination_event_service: eventgate-main

Then, when eventgate-wikimedia is checking if an event is allowed in a stream, it would get stream config entry for that stream and then check that all of the stream_config_constraints are set for that stream config entry. E.g. in

$wgEventStreams = [
  [
     'mediawiki.revision-create' => [
         'schema_title' => 'mediawiki/revision/create',
         'destination_event_service' => 'eventgate-main'
     ],
     'analytics.external_link_hover' => [
         'schema_title' => 'analytics/link_hover',
         'destination_event_service' => 'eventgate-analytics-external'
     ],
     ...
  ]
]

eventgate-wikimedia would check that each stream config entry has the same properties as stream_config_constraints.

I think this would be slightly less brittle than implicitly mapping the service-runner name to some stream config setting.

We'd still need to figure out how to make eventgate-wikimedia generate static stream config from EventStreamConfig API at runtime. We'd have to have some script that runs during deployment and gets all stream configs that match stream_config_constraints and renders out a static stream-config.yaml file that is used as the stream_config_uri.

I think doing this will make implementing T251609 a little less crazy, and also enable for easier stream dataset discovery. We might even eventually put extra configs into stream config that will allow for even easier data discovery; e.g. adding a default remote absolute schema_url to make looking up the latest schema for a stream easier latest_schema_url: https://schema.wikimedia.org/repositories/secondary/jsonschema/analytics/link_hover/latest. Then at least we wouldn't have to infer some URL from a schema_title when trying to e.g. use the schema to create canary events, or to automatically construct a Spark or Flink or Hive schema for a given stream.

@Pchelolo any objections?

Ottomata added a comment.May 27 2020, 6:39 PM

Hm, an interesting extension of this idea would be to make the eventgate-wikimedia's schema_title checking more flexible. Instead of explicitly setting schema_title in a stream config entry, we'd set something like

schema_shape:
  title: mediawiki/revision/create
  // and any other extra properties we'd want to match here, e.g.
  required:
    - cool_required_field

This would make sure that any event in the stream had a schema that conformed to schema_shape, instead of just explicitly checking schema_title == schema.title.

Anyway, just parking this idea here, we don't have a use case for this ATM, just would be cool to generalize.

• Pchelolo added a comment.May 27 2020, 6:54 PM

We'd still need to figure out how to make eventgate-wikimedia generate static stream config from EventStreamConfig API at runtime. We'd have to have some script that runs during deployment and gets all stream configs that match stream_config_allowed_settings and renders out a static stream-config.yaml file that is used as the stream_config_uri.

You can add an endpoint in the streamconfig extension that would filter by destination_event_service and return a list. You can make it even more flexible by allowing to filter by any stream property.

As for using service name directly, or duplicating the same name under a new property - I donno what would be the benefit or yet-another-key for the same value?

As for destination_event_service - I guess if we have that info, we can preserve EventBus::getInstanceForStream.

Ottomata added a comment.May 27 2020, 7:23 PM

We'd still need to figure out how to make eventgate-wikimedia generate static stream config from EventStreamConfig API at runtime

AH! eventgate-wikimedia already supports this. All I have to do is

stream_config_url: https://api.wikimedia.org/w/api.php?action=streamconfigs...
stream_config_is_dynamic: false

And eventgate-wikimedia will request all stream configs when it starts up, rather than when it receives a POST event request.

I donno what would be the benefit or yet-another-key for the same value?

Ah, mostly I don't want to couple a stream config to a service-runner concept. I'd rather be explicit about what the constraints are in configuration, so that the service-runner name isn't needed by code to compare with something explicit custom value in stream config entries.

You can add an endpoint in the streamconfig extension that would filter by destination_event_service and return a list.

Cool idea! Will look into it for sure. It might be easier just to implement this in eventgate-wikimedia, but perhaps doing it in EventStreamConfig will be more usable for other use cases.

Ottomata added a comment.May 27 2020, 9:53 PM

Oh ho, if I do implement this as an API param in EventStreamConfig, then we don't need any extra eventgate configs; we can just set stream_config_uri to one that includes the settings constraints, e.g.

stream_config_uri: https://api.wikimedia.org/w/api.php?action=streamconfigs&constraints=destination_event_service=eventgate-main
gerritbot added a comment.May 27 2020, 9:53 PM

Change 599127 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/EventStreamConfig@master] Add constraints API parameter to filter by settings constraints

https://gerrit.wikimedia.org/r/599127

gerritbot added a project: Patch-For-Review.May 27 2020, 9:53 PM
Ottomata added a comment.May 28 2020, 2:03 PM

Hm, we should settle on a config key for identifying the eventgate instance. The obvious one to use would be 'event_service_name', although that doesn't quite clarify what is being specified. I had suggested 'destination_event_service' (or '..._name'). We also may one day need stream config to have an explicit event service URL, to allow for dynamic configuration of clients...although we don't need that now. If we just set a name (defined in ProductionServices.php), MW php clients can easily map that to the URL, but remote ones can't. Perhaps we will need different config keys for both of these?

We don't need the event service URL in config yet; currently the only remote client is EventLogging (and WikimediaEvents), and those are both configured using specific config variables (e.g wgEventLoggingServiceUri and wgWMEClientErrorIntakeURL), but we should try to pick something that doesn't conflict if we need to put these URIs in stream config eventually.

Perhaps destination_event_service_name is the specific and future proof enough?

gerritbot added a comment.May 28 2020, 4:30 PM

Change 599127 merged by jenkins-bot:
[mediawiki/extensions/EventStreamConfig@master] Add constraints API parameter to filter by settings constraints

https://gerrit.wikimedia.org/r/599127

Maintenance_bot removed a project: Patch-For-Review.May 28 2020, 5:11 PM
ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.35; 2020-06-02).May 28 2020, 7:00 PM
gerritbot added a comment.May 28 2020, 8:43 PM

Change 599435 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[eventgate-wikimedia@master] Split :streams param on , or |

https://gerrit.wikimedia.org/r/599435

gerritbot added a project: Patch-For-Review.May 28 2020, 8:43 PM
gerritbot added a comment.May 28 2020, 9:12 PM

Change 599435 merged by Ottomata:
[eventgate-wikimedia@master] Split :streams param on , or |

https://gerrit.wikimedia.org/r/599435

Maintenance_bot removed a project: Patch-For-Review.May 28 2020, 10:10 PM
mpopov added a comment.May 29 2020, 1:03 PM
In T251935#6172791, @Ottomata wrote:

Hm, we should settle on a config key for identifying the eventgate instance. The obvious one to use would be 'event_service_name', although that doesn't quite clarify what is being specified. I had suggested 'destination_event_service' (or '..._name'). We also may one day need stream config to have an explicit event service URL, to allow for dynamic configuration of clients...although we don't need that now. If we just set a name (defined in ProductionServices.php), MW php clients can easily map that to the URL, but remote ones can't. Perhaps we will need different config keys for both of these?

We don't need the event service URL in config yet; currently the only remote client is EventLogging (and WikimediaEvents), and those are both configured using specific config variables (e.g wgEventLoggingServiceUri and wgWMEClientErrorIntakeURL), but we should try to pick something that doesn't conflict if we need to put these URIs in stream config eventually.

Perhaps destination_event_service_name is the specific and future proof enough?

What about adding a config top-level field and setting the event service name in config.destination_name? (And potentially config.destination_uri for cases when we don't want to hard-code a name-uri map into the client.) For EPC streams we're planning on keeping sampling configuration in the stream's config.sampling – which initially will have rate and identifier but eventually finer targeting too. When we showed you that doc you made the great suggestion of nesting sampling within a config to allow for more per-stream configurations, and I think this ties into that nicely.

Ottomata added a comment.May 29 2020, 11:18 PM

Hm. I don't love the name config, which would mean: Stream Config config setting. Also, a nested config might be hard to work with. I've justt added a 'constraints' parameter to the EventStreamConfig API setting, which allow you specify in the GET URI params key=values that must match in the Stream Config. With a non-nested destination_event_service, we'll be able to do

/w/api\.php?action=streamconfigs&constraints=destination_event_service=eventgate-main

If we go nested, I'll have to figure out how to encode the nested object into the query params, OR we'll have to switch to a POST based API and handle a POST body.

(I don't mean to say that a nested entry in a stream config is bad, its mostly that for the use case here: querying for stream configs that matches some constraint, a nested object is going to be complicated to specify in a query.)

Jdforrester-WMF mentioned this in T254390: Argument 1 passed to MediaWiki\Extension\EventStreamConfig\ApiStreamConfigs::multiParamToAssocArray() must be of the type array, null given, called in /srv/mediawiki/php-1.35.0-wmf.35/extensions/EventStreamConfig/includes/ApiStreamConfigs.php on line 66.Jun 3 2020, 7:29 PM
gerritbot added a comment.Aug 4 2020, 8:57 PM

Change 618394 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Add eventgate-logging-external streams, and add destination_event_service to all stream configs

https://gerrit.wikimedia.org/r/618394

gerritbot added a project: Patch-For-Review.Aug 4 2020, 8:57 PM
gerritbot added a comment.Aug 4 2020, 9:05 PM

Change 618395 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-logging-external - Use MW EventStreamConfig API to get static stream configs

https://gerrit.wikimedia.org/r/618395

gerritbot added a comment.Aug 5 2020, 3:51 PM

Change 618394 merged by Ottomata:
[operations/mediawiki-config@master] Add eventgate-logging-external streams, and add destination_event_service to all stream configs

https://gerrit.wikimedia.org/r/618394

Stashbot added a comment.Aug 5 2020, 3:56 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-05T15:56:22Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - Add eventgate-logging-external streams and destination_event_service settings - T251935 (duration: 01m 05s)

gerritbot added a comment.Aug 5 2020, 4:04 PM

Change 618395 merged by Ottomata:
[operations/deployment-charts@master] eventgate-logging-external - Use MW EventStreamConfig API to get static stream configs

https://gerrit.wikimedia.org/r/618395

gerritbot added a comment.Aug 5 2020, 4:28 PM

Change 618550 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Add eventgate service specific test.event streams

https://gerrit.wikimedia.org/r/618550

gerritbot added a comment.Aug 5 2020, 8:20 PM

Change 618612 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[eventgate-wikimedia@master] Bump eventgate version and to support /v1/_test/events route

https://gerrit.wikimedia.org/r/618612

gerritbot added a comment.Aug 5 2020, 8:24 PM

Change 618612 merged by Ottomata:
[eventgate-wikimedia@master] Bump eventgate version and to support /v1/_test/events route

https://gerrit.wikimedia.org/r/618612

gerritbot added a comment.Aug 5 2020, 9:06 PM

Change 618624 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate - use /v1/_test/events route for readinessProbe

https://gerrit.wikimedia.org/r/618624

gerritbot added a comment.Aug 6 2020, 1:57 PM

Change 618550 merged by Ottomata:
[operations/mediawiki-config@master] Add eventgate service specific test.event streams

https://gerrit.wikimedia.org/r/618550

Stashbot added a comment.Aug 6 2020, 2:00 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-06T14:00:54Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - Add eventgate-* test.event streams - T251935 (duration: 01m 08s)

gerritbot added a comment.Aug 6 2020, 2:38 PM

Change 618624 merged by Ottomata:
[operations/deployment-charts@master] eventgate - use /v1/_test/events route for readinessProbe

https://gerrit.wikimedia.org/r/618624

gerritbot added a comment.Aug 6 2020, 7:35 PM

Change 618833 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] wgEventStreams - fix typo in eventgate stream config

https://gerrit.wikimedia.org/r/618833

gerritbot added a comment.Aug 6 2020, 7:38 PM

Change 618833 merged by Ottomata:
[operations/mediawiki-config@master] wgEventStreams - fix typo in eventgate stream config

https://gerrit.wikimedia.org/r/618833

Stashbot added a comment.Aug 6 2020, 7:40 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-06T19:40:14Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - wgEventStreams - fix typo in eventgate stream config - T251935 (duration: 00m 59s)

gerritbot added a comment.Aug 6 2020, 7:42 PM

Change 618837 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] wgEventStreams - fix another typo in eventgate stream config

https://gerrit.wikimedia.org/r/618837

gerritbot added a comment.Aug 6 2020, 7:44 PM

Change 618837 merged by Ottomata:
[operations/mediawiki-config@master] wgEventStreams - fix another typo in eventgate stream config

https://gerrit.wikimedia.org/r/618837

Stashbot added a comment.Aug 6 2020, 7:45 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-06T19:45:27Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - wgEventStreams - fix another typo in eventgate stream config - T251935 (duration: 00m 58s)

gerritbot added a comment.Aug 6 2020, 7:54 PM

Change 618838 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-* - precache /test/event/1.0.0 schema

https://gerrit.wikimedia.org/r/618838

gerritbot added a comment.Aug 6 2020, 7:56 PM

Change 618838 merged by Ottomata:
[operations/deployment-charts@master] eventgate-* - precache /test/event/1.0.0 schema

https://gerrit.wikimedia.org/r/618838

gerritbot added a comment.Aug 6 2020, 8:25 PM

Change 618846 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[eventgate-wikimedia@master] Apply stream_config_uri_options for static stream config URIs too

https://gerrit.wikimedia.org/r/618846

gerritbot added a comment.Aug 6 2020, 8:29 PM

Change 618846 merged by Ottomata:
[eventgate-wikimedia@master] Apply stream_config_uri_options for static stream config URIs too

https://gerrit.wikimedia.org/r/618846

gerritbot added a comment.Aug 6 2020, 8:47 PM

Change 618851 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-logging-external: use remote stream config, eventgate-analytics-external: use constraints

https://gerrit.wikimedia.org/r/618851

gerritbot added a comment.Aug 6 2020, 8:49 PM

Change 618851 merged by jenkins-bot:
[operations/deployment-charts@master] eventgate-logging-external: use remote stream config, eventgate-analytics-external: use constraints

https://gerrit.wikimedia.org/r/618851

gerritbot added a comment.Aug 10 2020, 5:45 PM

Change 619331 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Add eventgate-analytics stream config

https://gerrit.wikimedia.org/r/619331

gerritbot added a comment.Aug 10 2020, 5:47 PM

Change 619331 merged by Ottomata:
[operations/mediawiki-config@master] Add eventgate-analytics stream config

https://gerrit.wikimedia.org/r/619331

Stashbot added a comment.Aug 10 2020, 5:49 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-10T17:49:53Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - Add eventgate-analytics streams - T251935 (duration: 01m 02s)

gerritbot added a comment.Aug 10 2020, 5:52 PM

Change 619334 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-analytics - Use MW EventStreamConfig API, only in staging for now

https://gerrit.wikimedia.org/r/619334

gerritbot added a comment.Aug 10 2020, 5:55 PM

Change 619334 merged by Ottomata:
[operations/deployment-charts@master] eventgate-analytics - Use MW EventStreamConfig API, only in staging for now

https://gerrit.wikimedia.org/r/619334

gerritbot added a comment.Aug 11 2020, 1:54 PM

Change 619469 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery@master] Add constraints parameter when using bin/camus with EventStreamConfig API

https://gerrit.wikimedia.org/r/619469

gerritbot added a comment.Aug 11 2020, 2:00 PM

Change 619472 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus::job - pass stream_configs_constraints through to refinery camus wrapper

https://gerrit.wikimedia.org/r/619472

gerritbot added a comment.Aug 11 2020, 2:50 PM

Change 619469 merged by Ottomata:
[analytics/refinery@master] Add constraints parameter when using bin/camus with EventStreamConfig API

https://gerrit.wikimedia.org/r/619469

Stashbot added a comment.Aug 11 2020, 2:51 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-11T14:51:39Z] <otto@deploy1001> Started deploy [analytics/refinery@35c4430]: Deploying to an-launcher1002 to get camus wrapper script changes - T251935

Stashbot added a comment.Aug 11 2020, 2:52 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-11T14:52:54Z] <otto@deploy1001> Finished deploy [analytics/refinery@35c4430]: Deploying to an-launcher1002 to get camus wrapper script changes - T251935 (duration: 01m 14s)

gerritbot added a comment.Aug 11 2020, 3:07 PM

Change 619472 merged by Ottomata:
[operations/puppet@production] camus::job - pass stream_configs_constraints through to refinery camus wrapper

https://gerrit.wikimedia.org/r/619472

gerritbot added a comment.Aug 11 2020, 3:15 PM

Change 619487 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus - declare event service specific camus::jobs.

https://gerrit.wikimedia.org/r/619487

gerritbot added a comment.Aug 11 2020, 3:35 PM

Change 619487 merged by Ottomata:
[operations/puppet@production] camus - declare event service specific camus::jobs.

https://gerrit.wikimedia.org/r/619487

gerritbot added a comment.Aug 11 2020, 3:47 PM

Change 619491 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus::job - Fix typo in stream_configs_constraints_opt

https://gerrit.wikimedia.org/r/619491

gerritbot added a comment.Aug 11 2020, 3:48 PM

Change 619491 merged by Ottomata:
[operations/puppet@production] camus::job - Fix typo in stream_configs_constraints_opt

https://gerrit.wikimedia.org/r/619491

gerritbot added a comment.Aug 11 2020, 5:50 PM

Change 619523 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] EventStreamConfig - Remove extraneous mediawiki.api-request stream

https://gerrit.wikimedia.org/r/619523

gerritbot added a comment.Aug 11 2020, 5:53 PM

Change 619523 merged by Ottomata:
[operations/mediawiki-config@master] EventStreamConfig - Remove extraneous mediawiki.api-request stream

https://gerrit.wikimedia.org/r/619523

Stashbot added a comment.Aug 11 2020, 5:56 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-11T17:56:37Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - Remove extraneous mediawiki.api-request stream - T251935 (duration: 01m 01s)

gerritbot added a comment.Aug 11 2020, 6:04 PM

Change 619532 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus - replace mediawiki_analytics_events with eventgate-analytics_events job

https://gerrit.wikimedia.org/r/619532

gerritbot added a comment.Aug 11 2020, 6:09 PM

Change 619532 merged by Ottomata:
[operations/puppet@production] camus - replace mediawiki_analytics_events with eventgate-analytics_events job

https://gerrit.wikimedia.org/r/619532

gerritbot added a comment.Aug 11 2020, 6:22 PM

Change 619541 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus - include mediawiki.api-request in eventgate-analytics topics to check

https://gerrit.wikimedia.org/r/619541

gerritbot added a comment.Aug 11 2020, 6:36 PM

Change 619541 merged by Ottomata:
[operations/puppet@production] camus - include mediawiki.api-request in eventgate-analytics topics to check

https://gerrit.wikimedia.org/r/619541

gerritbot added a comment.Aug 11 2020, 6:46 PM

Change 619544 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-analytics - Use remote EventStreamConfig in eqiad and codfw

https://gerrit.wikimedia.org/r/619544

gerritbot added a comment.Aug 11 2020, 6:47 PM

Change 619544 merged by Ottomata:
[operations/deployment-charts@master] eventgate-analytics - Use remote EventStreamConfig in eqiad and codfw

https://gerrit.wikimedia.org/r/619544

gerritbot added a comment.Aug 11 2020, 7:11 PM

Change 619552 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] EventStreamConfig - add streams for eventgate-main

https://gerrit.wikimedia.org/r/619552

gerritbot added a comment.Aug 11 2020, 7:32 PM

Change 619552 merged by Ottomata:
[operations/mediawiki-config@master] EventStreamConfig - add streams for eventgate-main

https://gerrit.wikimedia.org/r/619552

Stashbot added a comment.Aug 11 2020, 7:35 PM

Mentioned in SAL (#wikimedia-operations) [2020-08-11T19:35:42Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: EventStreamConfig - Add streams for eventgate-main - T251935 (duration: 01m 04s)

gerritbot added a comment.Aug 11 2020, 7:37 PM

Change 619554 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-main - use MW EventStreamConfig in staging

https://gerrit.wikimedia.org/r/619554

gerritbot added a comment.Aug 11 2020, 7:38 PM

Change 619554 merged by Ottomata:
[operations/deployment-charts@master] eventgate-main - use MW EventStreamConfig in staging

https://gerrit.wikimedia.org/r/619554

gerritbot added a comment.Aug 12 2020, 2:44 PM

Change 619763 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[eventgate-wikimedia@master] Bump eventgate dependency to e7cfa6d to pick up bug fix for test events

https://gerrit.wikimedia.org/r/619763

gerritbot added a comment.Aug 12 2020, 2:46 PM

Change 619763 merged by Ottomata:
[eventgate-wikimedia@master] Bump eventgate dependency to e7cfa6d to pick up bug fix for test events

https://gerrit.wikimedia.org/r/619763

gerritbot added a comment.Aug 12 2020, 2:54 PM

Change 619767 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-* - bump to version 2020-08-12-144657-production

https://gerrit.wikimedia.org/r/619767

gerritbot added a comment.Aug 12 2020, 2:57 PM

Change 619767 merged by Ottomata:
[operations/deployment-charts@master] eventgate-* - bump to version 2020-08-12-144657-production

https://gerrit.wikimedia.org/r/619767

gerritbot added a comment.Aug 12 2020, 6:13 PM

Change 619813 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-main - Use MW EventStreamConfig API

https://gerrit.wikimedia.org/r/619813

gerritbot added a comment.Aug 12 2020, 6:20 PM

Change 619813 merged by Ottomata:
[operations/deployment-charts@master] eventgate-main - Use MW EventStreamConfig API

https://gerrit.wikimedia.org/r/619813

gerritbot added a comment.Aug 12 2020, 6:28 PM

Change 619816 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus - replace mediawiki_events job with eventgate-main_events job

https://gerrit.wikimedia.org/r/619816

gerritbot added a comment.Aug 12 2020, 6:34 PM

Change 619816 merged by Ottomata:
[operations/puppet@production] camus - replace mediawiki_events job with eventgate-main_events job

https://gerrit.wikimedia.org/r/619816

gerritbot added a comment.Aug 12 2020, 6:42 PM

Change 619820 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus - Exclude mediawiki.job streams from eventgate-main_events job

https://gerrit.wikimedia.org/r/619820

gerritbot added a comment.Aug 12 2020, 6:46 PM

Change 619820 merged by Ottomata:
[operations/puppet@production] camus - Exclude mediawiki.job streams from eventgate-main_events job

https://gerrit.wikimedia.org/r/619820

gerritbot added a comment.Aug 12 2020, 6:59 PM

Change 619830 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] camus - Remove now unused mediawiki_events job

https://gerrit.wikimedia.org/r/619830

gerritbot added a comment.Aug 12 2020, 7:03 PM

Change 619830 merged by Ottomata:
[operations/puppet@production] camus - Remove now unused mediawiki_events job

https://gerrit.wikimedia.org/r/619830

Ottomata added a comment.Aug 12 2020, 8:32 PM

WOOHOO I think we are done! I updated some docs:

https://wikitech.wikimedia.org/wiki/Event_Platform/Stream_Configuration
https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate/Administration#EventStreamConfig_change
https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate#Event_Stream_Config

Ottomata moved this task from Next Up to Done on the Analytics-Kanban board.Aug 12 2020, 8:32 PM
Ottomata set Final Story Points to 21.
mpopov mentioned this in T259917: Streams with empty configs should be rendered as {} in the JSON returned by StreamConfig API.Aug 19 2020, 3:11 PM
mpopov awarded a token.Aug 19 2020, 3:19 PM
Nuria closed this task as Resolved.Sep 9 2020, 3:09 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL