Problem

MWHistoryDeltaWriter issues 8 sequential MERGE INTOs (M1–M7b) and MWHistorySnapshotMerger issues a DELETE followed by two INSERTs. Between any two mutations, intermediate state is visible to downstream readers — e.g. revision rows present but page_is_deleted not yet back-patched, or a gap where all snapshot rows are absent. The SnapshotMerger code had an explicit NOTE comment flagging this gap and naming WAP as the fix.

Solution

Add optional Iceberg WAP (Write-Audit-Publish) branching to both jobs. When --wap_branch <name> is supplied, all reads and writes target a private Iceberg branch via the dot-suffix notation (table.branch_<name>). On success, a single CALL catalog.system.fast_forward(...) atomically advances main. Downstream readers see only the pre-run state or the fully-written state.

Design notes

• Branch-suffix approach (not spark.wap.branch session conf): reads and writes are explicit in every SQL string, so intra-run reads — including revert_seed (M1/M2) and user_provenance (M7b) — are also scoped to the branch. M7b in particular must read the branch to see user create events inserted by M7 in the same run.
• --wap_branch is optional (no default). Old Airflow DAGs without the flag continue writing directly to main unchanged.
• Branch naming convention is left to Airflow (the job does not construct the name).
• DROP BRANCH IF EXISTS before CREATE BRANCH makes retries idempotent after a failed run.
• No write.wap.enabled table property needed — branch-suffix approach does not require it.

Remaining work

• Airflow DAG changes: pass --wap_branch <generated_name> to both Spark jobs
• Cluster smoke test: verify branch appears/disappears cleanly in Iceberg branch listing during a real daily run