Page MenuHomePhabricator
Create Task
Maniphest T286344

Remove StreamConfig::INTERNAL_SETTINGS logic from EventStreamConfig and do it in EventLogging client instead
Closed, ResolvedPublic
Actions

Description

The EventLogging MediaWiki extension requests stream configs for many streams on every page load. To keep the payload size down, we excluded extraneous stream config settings by default via the StreamConfig::INTERNAL_SETTINGS const. This is a bit hacky, as it hardcodes some settings in the extension code itself.

Stream Config is beginning to be used by more clients, and more settings are being added, e.g. https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/668124.

Since the reason for automatically excluding certain settings is really just for the EventLogging on every page load use case, we should make the EventLogging + EventStreamConfig + ResourceLoader bridge do the filtering of settings for EventLogging. That way, EventLogging can explictly set what settings should be exported.

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added subscribers: Masumrezarock100, Aklapper. · View Herald TranscriptJul 8 2021, 2:22 PM
• odimitrijevic triaged this task as High priority.Jul 26 2021, 3:53 PM
• odimitrijevic moved this task from Incoming to Event Platform on the Analytics board.
• odimitrijevic removed a project: MediaWiki-extensions-EventLogging.Nov 29 2021, 10:56 PM
Ottomata mentioned this in T303602: Generate $wgEventLoggingStreamNames from $wgEventStreams.Mar 14 2022, 11:51 AM
Maintenance_bot added a project: Data-Engineering.Mar 14 2022, 12:46 PM
• EChetty moved this task from Incoming (new tickets) to Apache Iceberg Migration on the Data-Engineering board.Mar 24 2022, 3:18 PM
• EChetty added a project: Data-Engineering-Kanban.Mar 24 2022, 5:06 PM
Ottomata added a subscriber: phuedx.May 11 2022, 3:18 PM
Krinkle awarded a token.May 11 2022, 5:52 PM
Krinkle unsubscribed.
JArguello-WMF removed a project: Analytics.Jun 23 2022, 9:46 PM
JArguello-WMF edited projects, added Data-Engineering-Planning; removed Data-Engineering.Jun 30 2022, 3:14 PM
• EChetty moved this task from Backlog to Event Platform on the Data-Engineering-Planning board.Aug 15 2022, 8:45 AM
phuedx added a comment.EditedAug 15 2022, 11:00 AM

I was about to write a task to request that consumer is also removed from the stream configs that are included in the ext.eventLogging RL module as its extraneous and increases payload size.

I'd also suggest that we shift to allowlisting stream config settings sent as part of the payload. An allowlist would be easier to maintain and would be more resilient to changes in the shape of the underlying stream configs, however infrequent.

Ottomata added a comment.Aug 15 2022, 12:50 PM

I'd also suggest that we shift to allowlisting stream config settings sent as part of the payload

+1

phuedx added a parent task: T309313: Minimise bytes transferred in Metrics Platform client.Sep 9 2022, 11:52 AM
gerritbot added a comment.Sep 9 2022, 12:36 PM

Change 831084 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/EventLogging@master] Hooks: Filter stream config settings

https://gerrit.wikimedia.org/r/831084

gerritbot added a project: Patch-For-Review.Sep 9 2022, 12:36 PM
gerritbot added a comment.Sep 9 2022, 6:15 PM

Change 831129 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/EventStreamConfig@master] StreamConfig: Soft deprecate parameter

https://gerrit.wikimedia.org/r/831129

gerritbot added a comment.Sep 9 2022, 6:15 PM

Change 831130 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/EventStreamConfig@master] Api: Do not pass deprecated parameter

https://gerrit.wikimedia.org/r/831130

gerritbot added a comment.Sep 9 2022, 6:15 PM

Change 831131 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/EventStreamConfig@master] StreamConfigs: Hard deprecate parameter

https://gerrit.wikimedia.org/r/831131

gerritbot added a comment.Sep 9 2022, 6:15 PM

Change 831132 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/EventBus@master] EventBusFactory: Do not pass deprecated parameter

https://gerrit.wikimedia.org/r/831132

gerritbot added a comment.Sep 10 2022, 1:44 AM

Change 831130 abandoned by Phuedx:

[mediawiki/extensions/EventStreamConfig@master] Api: Do not pass deprecated parameter

Reason:

See I83429bc3238.

https://gerrit.wikimedia.org/r/831130

gerritbot added a comment.Sep 10 2022, 1:45 AM

Change 831129 abandoned by Phuedx:

[mediawiki/extensions/EventStreamConfig@master] StreamConfig: Soft deprecate parameter

Reason:

See I83429bc3238.

https://gerrit.wikimedia.org/r/831129

gerritbot added a comment.Sep 10 2022, 10:29 AM

Change 831129 restored by Phuedx:

[mediawiki/extensions/EventStreamConfig@master] StreamConfig: Soft deprecate parameter

https://gerrit.wikimedia.org/r/831129

gerritbot added a comment.Sep 11 2022, 7:21 PM

Change 831236 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/EventLogging@master] EventLogging: Do not pass deprecated parameter

https://gerrit.wikimedia.org/r/831236

Ottomata added a comment.Sep 12 2022, 6:15 AM

Great stuff! TY Sam!

I think there are action API usages out there that set all_settings=true, but IIUC that param will just be ignored now, and all settings will always be returned, right?

phuedx added a comment.Sep 12 2022, 9:34 AM
In T286344#8227782, @Ottomata wrote:

I think there are action API usages out there that set all_settings=true, but IIUC that param will just be ignored now, and all settings will always be returned, right?

Here's a list of those action API usages: https://codesearch.wmcloud.org/search/?q=all_settings%5C%3D(1%7Ctrue). There don't appear to be any usages where all_settings is falsy: https://codesearch.wmcloud.org/search/?q=all_settings%5C%3D(0%7Cfalse).

I would recommend updating those usages before merging StreamConfigs: Hard deprecate parameter, which is the last patch in the relation chain.

phuedx added a project: Metrics Platform.Sep 17 2022, 5:21 AM
phuedx moved this task from Backlog to QA/Review on the Metrics Platform board.
Ottomata added a comment.Sep 20 2022, 2:55 PM

+1

phuedx added a comment.Sep 23 2022, 6:50 AM

Status update:

@Ottomata has signed off on (+1'd) all of the patches in the relation chain. Metrics Platform is responsible for getting them deployed. We'll do so in two stages to ease monitoring each deployment:

Stage 1: Deploy the patch that filters stream config settings in EventLogging
Stage 2: Deploy the patches that deprecate the $includeAllSettings parameter for StreamConfigs::get()

phuedx claimed this task.Sep 23 2022, 6:50 AM
• EChetty moved this task from QA/Review to Metrics Platform Kanban on the Metrics Platform board.Oct 1 2022, 4:28 PM
• EChetty edited projects, added Metrics Platform (Metrics Platform Kanban); removed Metrics Platform.
• EChetty moved this task from Ready to In Review on the Metrics Platform (Metrics Platform Kanban) board.
phuedx removed phuedx as the assignee of this task.Oct 3 2022, 10:25 AM
• EChetty moved this task from In Review to Done on the Metrics Platform (Metrics Platform Kanban) board.Oct 4 2022, 3:25 PM
• EChetty moved this task from Done to In Review on the Metrics Platform (Metrics Platform Kanban) board.
• EChetty moved this task from In Review to Blocked/Paused on the Metrics Platform (Metrics Platform Kanban) board.
• EChetty edited projects, added Data Pipelines (Sprint 02); removed Data-Engineering-Planning, Data-Engineering-Kanban, Product-Data-Infrastructure.Oct 4 2022, 3:28 PM
• EChetty removed subscribers: • Mholloway, • jlinehan.
Restricted Application added a project: Data-Engineering. · View Herald TranscriptOct 4 2022, 3:28 PM
• EChetty edited projects, added Data-Engineering-Planning; removed Data-Engineering, Data Pipelines (Sprint 02).Oct 4 2022, 3:28 PM
phuedx moved this task from Blocked/Paused to Next Up on the Metrics Platform (Metrics Platform Kanban) board.Nov 2 2022, 9:16 AM
• lbowmaker moved this task from Backlog to Investigate on the Event-Platform board.Nov 16 2022, 4:43 PM
phuedx moved this task from Next Up to Blocked/Paused on the Metrics Platform (Metrics Platform Kanban) board.Nov 30 2022, 4:03 PM
• EChetty moved this task from Metrics Platform Kanban to Tracking on the Metrics Platform board.Jan 10 2023, 4:09 PM
• EChetty edited projects, added Metrics Platform; removed Metrics Platform (Metrics Platform Kanban).
JArguello-WMF subscribed.Jan 17 2023, 2:31 PM
gerritbot added a comment.Feb 14 2023, 8:37 AM

Change 831084 merged by jenkins-bot:

[mediawiki/extensions/EventLogging@master] Hooks: Filter stream config settings

https://gerrit.wikimedia.org/r/831084

ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.24; 2023-02-20).Feb 14 2023, 9:00 AM
JArguello-WMF moved this task from Investigate to Radar on the Event-Platform board.Mar 6 2023, 4:04 PM
gerritbot added a comment.Mar 20 2023, 3:36 PM

Change 831129 merged by jenkins-bot:

[mediawiki/extensions/EventStreamConfig@master] StreamConfig: Soft deprecate parameter

https://gerrit.wikimedia.org/r/831129

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.1; 2023-03-20).Mar 20 2023, 4:00 PM
gerritbot added a comment.Mar 21 2023, 2:24 PM

Change 831236 merged by jenkins-bot:

[mediawiki/extensions/EventLogging@master] EventLogging: Do not pass deprecated parameter

https://gerrit.wikimedia.org/r/831236

gerritbot added a comment.Mar 21 2023, 2:24 PM

Change 831132 merged by jenkins-bot:

[mediawiki/extensions/EventBus@master] EventBusFactory: Do not pass deprecated parameter

https://gerrit.wikimedia.org/r/831132

ReleaseTaggerBot edited projects, added MW-1.41-notes (1.41.0-wmf.2; 2023-03-27); removed MW-1.41-notes (1.41.0-wmf.1; 2023-03-20).Mar 21 2023, 3:00 PM
phuedx moved this task from Tracking to Metrics Platform Kanban on the Metrics Platform board.Mar 22 2023, 10:31 AM
phuedx edited projects, added Metrics Platform (Metrics Platform Kanban); removed Metrics Platform.
phuedx moved this task from Blocked/Paused to In Review on the Metrics Platform (Metrics Platform Kanban) board.
gerritbot added a comment.Mar 22 2023, 12:25 PM

Change 902075 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/EventBus@master] EventBusFactory: Do not pass deprecated parameter

https://gerrit.wikimedia.org/r/902075

gerritbot added a comment.Mar 22 2023, 1:41 PM

Change 902075 merged by jenkins-bot:

[mediawiki/extensions/EventBus@master] EventBusFactory: Do not pass deprecated parameter

https://gerrit.wikimedia.org/r/902075

gerritbot added a comment.Mar 22 2023, 4:16 PM

Change 902118 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/libs/metrics-platform@master] [Java][Swift] Remove all_settings param from API call

https://gerrit.wikimedia.org/r/902118

gerritbot added a comment.Mar 22 2023, 4:54 PM

Change 902118 merged by jenkins-bot:

[mediawiki/libs/metrics-platform@master] [Java][Swift] Remove all_settings param from API call

https://gerrit.wikimedia.org/r/902118

gerritbot added a comment.Mar 22 2023, 5:43 PM

Change 831131 merged by jenkins-bot:

[mediawiki/extensions/EventStreamConfig@master] StreamConfigs: Hard deprecate parameter

https://gerrit.wikimedia.org/r/831131

phuedx added a comment.Mar 23 2023, 9:58 AM

🎉

We'll monitor logstash as the hard deprecation rolls out next week.

phuedx moved this task from In Review to Ready to Deploy on the Metrics Platform (Metrics Platform Kanban) board.Mar 23 2023, 9:58 AM
phuedx added a comment.Apr 3 2023, 11:46 AM

https://logstash.wikimedia.org/goto/b5e0407128dc6718930d0528dfc03f85

phuedx added a comment.Apr 3 2023, 11:58 AM

I'll check back in on the dashboard above on Monday 17th.

phuedx moved this task from Ready to Deploy to Blocked/Paused on the Metrics Platform (Metrics Platform Kanban) board.Apr 3 2023, 11:58 AM
gerritbot added a comment.Apr 12 2023, 2:47 PM

Change 908251 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] eventgate - remove deprecated all_settings stream config param

https://gerrit.wikimedia.org/r/908251

gerritbot added a comment.Apr 12 2023, 2:48 PM

Change 908251 merged by Ottomata:

[operations/deployment-charts@master] eventgate - remove deprecated all_settings stream config param

https://gerrit.wikimedia.org/r/908251

gerritbot added a comment.Apr 19 2023, 4:21 PM

Change 910046 had a related patch set uploaded (by Phuedx; author: Phuedx):

[analytics/refinery@master] Remove deprecated all_settings streamconfigs param

https://gerrit.wikimedia.org/r/910046

gerritbot added a comment.Apr 19 2023, 4:48 PM

Change 910046 merged by Ottomata:

[analytics/refinery@master] Remove deprecated all_settings streamconfigs param

https://gerrit.wikimedia.org/r/910046

phuedx added a comment.Apr 19 2023, 7:36 PM

@Ottomata: Thanks for merging that patch. There are a couple of mentions remaining, which look like they need to be updated. Is that right?

Ottomata added a comment.Apr 19 2023, 8:28 PM

Looks right! All of those can be safely changed.

gerritbot added a comment.Apr 20 2023, 12:44 PM

Change 910470 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/services/eventstreams@master] Remove deprecated all_settings streamconfigs param

https://gerrit.wikimedia.org/r/910470

gerritbot added a comment.Apr 20 2023, 12:48 PM

Change 910471 had a related patch set uploaded (by Phuedx; author: Phuedx):

[operations/deployment-charts@master] Remove deprecated all_settings streamconfigs param

https://gerrit.wikimedia.org/r/910471

gerritbot added a comment.Apr 20 2023, 12:49 PM

Change 910470 merged by jenkins-bot:

[mediawiki/services/eventstreams@master] Remove deprecated all_settings streamconfigs param

https://gerrit.wikimedia.org/r/910470

phuedx added a comment.Apr 20 2023, 1:12 PM

@Ottomata: I don't appear to have permissions to edit the Puppet config for deployment-eventstreams-2.deployment-prep on Horizon. Do you?

phuedx moved this task from Blocked/Paused to In Review on the Metrics Platform (Metrics Platform Kanban) board.Apr 20 2023, 2:39 PM
Ottomata added a comment.Apr 20 2023, 2:39 PM

Yes, and done!

phuedx moved this task from In Review to Sign Off on the Metrics Platform (Metrics Platform Kanban) board.Apr 24 2023, 12:08 PM
phuedx moved this task from Sign Off to Done on the Metrics Platform (Metrics Platform Kanban) board.Apr 24 2023, 12:36 PM
Ottomata added a comment.Jun 6 2023, 1:20 AM

Can we close this?

JArguello-WMF removed a project: Data-Engineering-Planning.Jun 29 2023, 9:49 PM
Restricted Application added a project: Data-Engineering. · View Herald TranscriptJun 29 2023, 9:49 PM
JArguello-WMF moved this task from Apache Iceberg Migration to Event Platform Backlog on the Data-Engineering board.Jun 29 2023, 11:33 PM
JArguello-WMF added a project: Data Engineering and Event Platform Team.Jun 30 2023, 4:26 PM
JArguello-WMF moved this task from Data Eng Backlog to To be Estimated/To be discussed on the Data Engineering and Event Platform Team board.Jun 30 2023, 4:27 PM
JArguello-WMF moved this task from To be Estimated/To be discussed to Radar (External Teams) on the Data Engineering and Event Platform Team board.
JArguello-WMF closed this task as Resolved.Jun 30 2023, 11:04 PM
gerritbot added a comment.Nov 6 2023, 2:47 PM

Change 910471 abandoned by Ottomata:

[operations/deployment-charts@master] Remove deprecated all_settings streamconfigs param

Reason:

Done in a different patch

https://gerrit.wikimedia.org/r/910471

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits