Semi-Related: T223602: Define variant Wikimedia production config in compiled, static files

Is there a location in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master where the schema can live?

It seems like since EventStreamConfig extension owns the $wgEventStreams config variable, perhaps the schema could live there? Hm, but...

There are only a few hardcoded EventStreamConfig settings, but most are pretty dynamic and only relevant to certain clients.

I'd be reluctant to add e.g. ExP settings to a schema in the generic EventStreamConfig extension. Hm.

Hm. I suppose...if the schema did indeed live in mediawiki-config and was validated by logic in MediaWiki config, then it would be already to add client specific settings to the schema.

Okay! Sounds good! :)

I suppose EventStreamConfig could have the logic for validating a provided schema, and the schema could live in mediawiki-config? EventStreamConfig could have a bare bones default schema that allowed additionalProperties, and validated its few hardcoded fields? And then additional schemas with which to validate could be provided? Hm!

In T405516#11211895, @Ottomata wrote:

I suppose EventStreamConfig could have the logic for validating a provided schema, and the schema could live in mediawiki-config? EventStreamConfig could have a bare bones default schema that allowed additionalProperties, and validated its few hardcoded fields? And then additional schemas with which to validate could be provided? Hm!

This sounds great, I think that's the right approach.

I drafted a schema: https://gitlab.wikimedia.org/bearloga/eventstreamconfig-schema-prototype/-/blob/main/eventstreamconfig/1.0.0.yml?ref_type=heads

and have tested it with a copy of the config returned by https://meta.wikimedia.org/w/api.php?action=streamconfigs&format=json

Feel free to download the repo and try it out yourself. If you want to check the live stream config (rather than the api-result.json copy in the repo), you need to comment out pytest.skip in test_remote_api_result in test_schema.py

However!

Learning 1: null is not a valid value for topic_prefixes and EventStreamConfig should be changed to render empty topic_prefixes as [] in the API result, e.g.

"eventlogging_PrefUpdate": {
  "topic_prefixes": [],
  "canary_events_enabled": true,

Learning 2: mediawiki.product_metrics.checkuser_ip_auto_reveal_interaction stream is currently subtly misconfigured

Learning 2: mediawiki.product_metrics.checkuser_ip_auto_reveal_interaction stream is currently subtly misconfigured

Patch to fix it: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1194733

COOL!

topic_prefixes

I wonder if we could just make this a 'private' config and never expose it in API response. It is used to generate auto generate topics if it is not explicitly set.

I wonder if we could just make this a 'private' config and never expose it in API response. It is used to generate auto generate topics if it is not explicitly set.

How would EventGate or Kafka or whatever obtain in (in cases where topics is not explicitly set) if it's omitted from the API response? Do they not query the API?

How would EventGate or Kafka or whatever obtain in (in cases where topics is not explicitly set) if it's omitted from the API response? Do they not query the API?

topics is automatically set in the API response if topic_prefixes is set (and topics is not explicitly set).