Incremental MediaWiki History Phase II
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	xcollazo
	Jun 5 2026, 3:27 PM

Project Tags

Referenced Files

None

Subscribers

Aklapper

nshahquinn-wmf

xcollazo

Description

On T424350: Incremental MediaWiki History Phase I, we delivered a v1 of wmf_content. mediawiki_history_incremental_v1.

Lots of compromises were made to deliver on time.

On this follow up work, we should focus on resource usage efficiency (ex. use merge-on-read when appropriate), data quality (ex. how can we track events when they switch from source=events to source=snapshot? How much drift do we have?), and we should also make space to deliver a v2 table (tables?) with better ergonomics for downstream users based on feedback from T425573: mediawiki_history_incremental_v1: schema specification for stakeholder review.

Related Objects
Search...

Status	Assigned	Task
Open	None	T428273 Incremental MediaWiki History Phase II
Open	None	T428208 Evaluate write sort-order and event_entity partitioning for mediawiki_history_incremental_v1
Open	xcollazo	T427592 Adopt MoR for MWHistoryDeltaWriter MERGE 2 and MERGE 6 (back-patches
Open	APizzata-WMF	T426801 Iceberg 1.6.1 vectorized Parquet reader crashes Spark 3.5 SELECTs
Open	None	T426510 Migrate MediawikiHistoryRunner to Spark 3.5 + AQE
Open	None	T426331 MWHistorySnapshotMerger: skip COW rewrites for unchanged rows via hash-based MERGE condition
Open	APizzata-WMF	T424358 Investigate oversized driver logs for `mediawiki_history_denormalize`
Open	None	T426369 analytics-refinery-source repo fails post-merge builds, releases succeed
Open	Ottomata	T423935 Incremental MWH - MediaWiki event data source improvements
Open	Ottomata	T423952 Create mediawiki.user_change event stream
Open	Ottomata	T424685 Emit comprehensive mediawiki user block change information in an event stream
Declined	Ottomata	T425360 EventBus - user entity schema should differentiate between explicit and implicit user groups
Resolved	Ottomata	T425801 CheckUser extension should use EventBus.UserEntitySerializer service instead of constructor
Resolved	Ottomata	T426198 Event schemas - mediawiki user entity should be wiki aware
Resolved	Ottomata	T424767 EventBus - consider schema versions when serializing entities
Open	None	T427817 MediaWiki DomainEvents - Create new User related DomainEvents
Open	Ottomata	T424887 MediaWiki state change streams- add access_method & platform details
Open	Ahoelzl	T429211 Track mobile app and api account registration separately
Open	tchin	T425029 mediawiki.page_change.v1 - add revision.editor.first_edit_dt field
Open	Ottomata	T423583 mediawiki.page_change.v1 event - Add revision revert details
Declined	Ottomata	T426998 mediawiki.page_change.v1 event - Add user first_registration_dt field
Open	None	T425996 Add log_id to mediawiki page_change and user_change events
Open	None	T427815 MediaWiki DomainEvents - Include LogEntry
Open	Ottomata	T428176 mediawiki.page_change.v1 events - add wikidata id for pages
Open	Ottomata	T430967 MediaWiki state change events - emit global edit count
Open	None	T432050 EventBus - move event producers to a different extension
Open	APizzata-WMF	T427441 Delete `wmf_raw.mediawiki_content` table after a few months
Open	None	T427862 Convert `control_map MAP<STRING,STRING>` to a typed struct in `mediawiki_history_incremental_v1`
Open	None	T429453 Revisit only changing `source = 'event'` when merging
Open	Ahoelzl	T428018 Add user status details for cross-wiki users in MWH
Open	None	T429753 Consistent NULL semantics for Inc-MWH
Open	None	T264791 Rework how mediawiki-history differentiates fake page-create from real ones
In Progress	APizzata-WMF	T425734 Quality verification for mediawiki_history_incremental_v1 using Iceberg time travel
Open	None	T431598 44305 revisions in June with timestamp < current_date -90D