This is as far as I can get this before my leave. Docs may need to be updated with any changes that land before we to go prod with v1.

Associated brief diagram of the pipeline now at https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/1301381.

A first iteration of docs now available at https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Edits/Mediawiki_history_incremental_v1.

(https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/1300876 is merged but still needs to be released and put in prod via refinery source bump on Airflow DAGs)

(I'm now boldly closing this phase I task. There will be opportunities to make this all more ergonomic on T428273: Incremental MediaWiki History Phase II, and indeed the feedback provided here will be the guiding principles for that work.)

Current schema for mediawiki_history_incremental_v1 (full reference)

Schema update — what changed since the original spec

I suspect this one is zh_min_nanwiki bad data.

CC @AKhatun_WMF, as FYI, and to acknowledge that the whole row_update_dt dance from T428503 could definitely be better. Please chime in whether you think this approach would better address your use case?

While drafting this ticket, I realized that an implementation of this is significantly more complicated that I originally though.

In T427862#12011654, @Ottomata wrote:

FWIW, it looks like the issue encountered in the linked task would be handled by suggestion above:

I suggested to maybe use a map with TIMESTAMP values, and in cases where you want something other than a timestamp, use a different control (map?) field.

Opening this as part of Phase I for consideration, but I think it would be totally fine if we move this for Phase II.

Opening this as part of Phase I for consideration, but I think it would be totally fine if we move this for Phase II.

For seeing this work thru, I am boldly assigning it to @APizzata-WMF, and setting @JAllemandou as co-assignee.

Re-opening and attaching to Phase II, as we continue to trip on the unstructured map. see T428503#12011591.

In T428503#12008157, @AKhatun_WMF wrote:

There seems to be one missed date time format thing in control map: https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/refinery-job-35/src/main/scala/org/wikimedia/analytics/refinery/job/incremental/mediawikihistory/MWHistoryDeltaWriter.scala#752

Stale comment about 90 days? https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/refinery-job-35/src/main/scala/org/wikimedia/analytics/refinery/job/incremental/mediawikihistory/MWHistoryDeltaWriter.scala#133

WAD in production. Love the new branching pattern, we should do similarly for MWCH, and for any new Iceberg tables that have multi-step data ingest!

Q: Are the daily back-patch MERGEs in MWHistoryDeltaWriter meant to guard against out-of-order re-runs, or is the monthly reconcile the only consistency backstop?
...
Mainly trying to understand whether downstream consumers can safely re-read/backfill a window between monthly reconciles.

Both DAGs WAD in production at https://airflow.wikimedia.org/home?tags=mediawiki_incremental_history.

CC @BTullis, for visibility.

In T425996#12010812, @xcollazo wrote:

In T425996#12010236, @xcollazo wrote:

@Ottomata Did you intent to move this one to Phase II as well?

In T425996#12010438, @Ottomata wrote:

I did! Should I not have?

Whoops, my bad, I thought it was still under Phase I. Please ignore.

In T425996#12010236, @xcollazo wrote:

@Ottomata Did you intent to move this one to Phase II as well?

Note that TABLESAMPLE is not deterministic, so we may want to do our own way of sampling here so that reruns are repeatable?

@Ottomata did you intent to move this one to Phase II?

@Ottomata Did you intent to move this one to Phase II as well?

We can wait the beginning of July and close the task?

In T424355#11975313, @APizzata-WMF wrote:

Yesterday's sqoop process terminated ingesting all the necessary tables for MWH at 2026-06-01 15:22 UTC. We are not that much far off the 14 hrs result obtained during testing!
Further optimization could be applied by increasing the parallelism of the smaller wiki, as long as it does not impact the performance of an-launcher1003.

In short, if monthly load gets snapshot data till 31st May, and lands on 3rd June. Any visibility change events applied to old revisions 1 or 2 June gets lost. This is corrected in the next monthly load.

Similar to monthly reconcile logic, should the incremental row_update_dt also be -1 second?

Implemented the populate-on-reconcile change discussed above:

We discussed the semantics of row_update_dt on today's sync up and agreed that to make this column more useful to other folks we should populate it when doing monthly reconciles as well (instead of setting it to NULL). The value should be the data_interval_end of the monthly reconcile job, or perhaps one second before? Will think about it a bit more.

We have done multiple test runs ingesting data from event.mediawiki_user_change_dev0, and so far have found no data quality issues.

Implemented WAP (Write-Audit-Publish) branching for both MWHistoryDeltaWriter and MWHistorySnapshotMerger.

@BTullis delivered Spark 3.5.8 and Iceberg 1.6.1 for us over at T338057, effectively taking care of this issue. Thanks @BTullis! Closing.

Added four fields to both MWHistoryDeltaWriter and MWHistorySnapshotMerger: