Actually, now that I think more about it, because we were able to reuse the page_content_change schema, this task may just be an Airflow task reusing the existing process_events.py (formerly events_merge_into.py), but pointing it to the new table from the reconciliation stream.

gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/900

Draft: analytics: dumps: add reconciliation ingestion

To test, we can override the DagProperty hive_wikitext_raw_table with xcollazo.wikitext_raw_rc2_wiki_partitioned_plus_bf_on_rev_id.

gmodena updated https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/91

content_history: parse action api v2 format.

Change #1088275 had a related patch set uploaded (by Gmodena; author: Gmodena):

[operations/deployment-charts@master] dse-k8s-services: mw-dump: version bump image

https://gerrit.wikimedia.org/r/1088275

gmodena updated https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/92

content_history: explicit check on content format.

gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/92

content_history: explicit check on content format.

In T368746#10277534, @xcollazo wrote:

To test, we can override the DagProperty hive_wikitext_raw_table with xcollazo.wikitext_raw_rc2_wiki_partitioned_plus_bf_on_rev_id.

f/up from a longer thread on slack.

I've been testing this DAG on a development Airflow instance and have validated that the following integrations work:

• Events produced in the reconciliation topic are enriched.
• Enriched reconciliation data is loaded into HDFS/Hive via the regular Gobblin event ingestion pipeline.
• Page content, visibility, and enriched data are merged into the content history table with the https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/900 dag .
• Upon reconciliation MERGE INTO, the content history table is updated and aligns with the reconciliation enrichment topics.

@xcollazo is there any other test you'd like to do at this stage?

@xcollazo is there any other test you'd like to do at this stage?

LGTM!

gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/900

analytics: dumps: add reconciliation consumer

Change #1088275 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s-services: mw-dump: version bump image

https://gerrit.wikimedia.org/r/1088275

@gmodena this is WAD, correct? Can we close?