To support {T410940}, Data Engineering will be deploying 2 pyflink streaming applications. These will result in 2 new event stream data products:
- mediawiki.page_html_content_change
- mediawiki.page_html_feature_counts_change
As of April 20 2026, we are ready to move from development phase to release candidate, and eventually to a final .v1 of these event streams.
This task is a checklist / container task to track the remaining work needed to reach v1 release for these streams.
Please update this task description with details (new subtasks) and additional work.
---
=== For release candidate
[x] Finalize schemas
-- {T415158}
-- Move rendering_content_change and rendering_feature_counts_change out of [[ https://schema.wikimedia.org/#!//primary/jsonschema/development | development namespace ]].
-- -> [[ https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-primary/-/merge_requests/41 | MR41 ]]
[x] Create mediawiki.page_html_content_change.rc0 and mediawiki.page_html_feature_counts_change.rc0 streams and use them with finalized schema URIs in stream jobs.
-- -> [EventStreamConfig - add rc0 streams for html and feature count change (1276397)](https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1276397)
[x] Refactor edit type helmfiles to use `_*_common_*/` values file symlinks pattern, rather than duplicating common settings in different files.
[x] Settle all helmfiles to their final values. (Stop using --set overrides)
[x] Create mw-page-html-feature-counts-change-enrich (edit type) helmfile for production deployment
[x] Rename mw-page-edit-type-enrich-next (staging) (edit-type) helmfile to mw-page-html-feature-counts-change-enrich-next
[x] Use `process_async_enabled_default=false` for edit types feature-change enrichment.
[x] Set up alerting for stream jobs (should be able to re-use prior work) {T423996} {T424224}
[x] ensure error DLQ streams are working properly
[x] Resolve any other outstanding TODOs in code and helmfiles
[x] Drop rc0 and dev tables, delete data from hdfs
=== For v1 release
[x] SLOs? - (Note: SLOs are not defined, but alerts are and v1 was released some time ago.)
[x] Update https://wikitech.wikimedia.org/wiki/MediaWiki_Event_Enrichment docs
-- Especially note which kind of events are and are not in these streams (e.g. content_model=wikitext, no enrichment on visibility change, etc.)
-- Update datahub docs for the streams as well?
[x] Create mediawiki.page_html_content_change.v1 and mediawiki.page_html_feature_counts_change.v1 (name TBD) streams and use them in stream jobs.
-- [EventStreamConfig - Declare .v1 streams for html content and feature counts (1278476)](https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1278476)
[x] Remove .rc0 streams and no longer any remaining used development/ schemas
[x] Add relevant event table(s) to [[ https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/+/refs/heads/master/static_data/sanitization/event_sanitized_main_allowlist.yaml | event_santized_main_allowlist.yaml ]] so that we keep data longer than 90 days. (Feature counts was added, HTML could have PII)
-- [x] mediawiki_html_feature_counts_change for sure
-- Do we want to also keep HTML? Would be nice but: PII issues? Size issues?
=== Additional tasks
Not blocking v1 release.
[x] {T418996}
[x] {T422928}
[] {T409464} and use it to simplify enrichment conditional logic.
[] Update [[ https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas/Guidelines | Event_Platform/Schemas/Guidelines ]] with docs about new `delta` state modeling convention. - @ottomata TODO
[] Expose mediawiki_html_feature_counts in EventStreams?
-- Actually, to do this we'd need these events in kafka main (or other work). But, in hindsight, perhaps we should produce these to kafka main either way?
[x] Delete development schemas
[x] Delete unused Kafka topics: https://phabricator.wikimedia.org/T427951