Summary
Propagate source_log_id from PageState/UserState through MediawikiEvent.fromPageState/fromUserState into the final denormalized wmf.mediawiki_history table as a new nullable BIGINT column event_log_id.
log_id is the natural key for all page and user events in MediaWiki — every move, delete, restore, rename, block, and account-creation corresponds to exactly one row in the logging table identified by (wiki_db, log_id). It flows into the intermediate page_history/ and user_history/ Parquet files (as source_log_id on PageState/UserState) but was stripped during denormalization. This task closes that gap.
event_log_id will be NULL for revision rows, which use revision_id as their identity column.
Changes
1. analytics/refinery/source — propagate through denormalization
- Add eventLogId: Option[Long] to MediawikiEvent (position 1 in schema, after wiki_db)
- Populate from state.sourceLogId in fromPageState / fromUserState
- Explicit None in both fromRevisionRow constructors
2. analytics/refinery — DDL update
- Add event_log_id BIGINT column to create_mediawiki_history_table.hql
Validation and getting changes to prod
- First run it myself and write to xcollazo database. Compare output to prod data at wmf, make sure it looks good.
- If everything looks ok, then merge the changes, release refinery and refinery source.
- Since table is EXTERNAL, then just DROP TABLLE and then CREATE again with new schema.
- Rerun DAG for this month.
Relationship to incremental pipeline (T424350)
Once event_log_id is available in wmf.mediawiki_history, the monthly SnapshotMerger can use it as the MERGE key for page and user entity rows:
ON t.wiki_db = s.wiki_db AND ( (s.revision_id IS NOT NULL AND t.revision_id = s.revision_id) OR (s.event_log_id IS NOT NULL AND t.event_log_id = s.event_log_id) )
Full page/user entity support in the daily DeltaWriter also requires log_id in the page_change_v1 event stream — tracked on T425996: Add log_id to mediawiki page_change and user_change events.