Page MenuHomePhabricator
Create Task
Maniphest T418467

Enrich "parent" HTML using diffs
Closed, ResolvedPublic
Actions

Description

During the HTML enrichment pipeline, we need to also enrich the event with the parent revision HTML content. To do that, we are doing the following:

  • Use the new schema designed for it
  • If the page change has a parent_rev_id, call the HTML endpoint to get the content.
  • Given the main HTML (html_canonical) and the parent HTML, compute a unified_diff and store it in the event.
  • Make sure the unified_diff can be used to build the parent HTML completely.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
stream: mw-page-html-content-change-enrich-nextoperations/deployment-chartsmaster+1 -1
stream: mediawiki.page_html_content_changeoperations/deployment-chartsmaster+2 -4
stream: mw-page-html-content-change-enrich-nextoperations/deployment-chartsmaster+2 -2
component: mediawiki.page_html_content_change.dev0operations/mediawiki-configmaster+4 -5
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Enrich Parent HTML with Diffsrepos/data-engineering/mediawiki-event-enrichment!110javiermontonfeature/html-enrich-parentmain
Customize query in GitLab

Related Objects
Search...

Event Timeline

JMonton-WMF created this task.Feb 26 2026, 12:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 26 2026, 12:14 PM
JMonton-WMF moved this task from Next Up to In progress on the Data-Engineering (Q3 FY25/26 January 1st - March 31th) board.Feb 26 2026, 12:14 PM
CodeReviewBot added a comment.Feb 26 2026, 12:18 PM

javiermonton updated https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/110

Enrich Parent HTML with Diffs

gerritbot added a comment.Feb 26 2026, 12:29 PM

Change #1244629 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[operations/mediawiki-config@master] component: mediawiki.page_html_content_change.dev0

https://gerrit.wikimedia.org/r/1244629

gerritbot added a project: Patch-For-Review.Feb 26 2026, 12:29 PM
gerritbot added a comment.Feb 26 2026, 4:50 PM

Change #1244629 merged by jenkins-bot:

[operations/mediawiki-config@master] component: mediawiki.page_html_content_change.dev0

https://gerrit.wikimedia.org/r/1244629

Stashbot added a comment.Feb 26 2026, 4:51 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-26T16:51:32Z] <javiermonton@deploy2002> Started scap sync-world: Backport for [[gerrit:1244629|component: mediawiki.page_html_content_change.dev0 (T418467)]]

Stashbot added a comment.Feb 26 2026, 4:53 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-26T16:53:35Z] <javiermonton@deploy2002> javiermonton: Backport for [[gerrit:1244629|component: mediawiki.page_html_content_change.dev0 (T418467)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Feb 26 2026, 5:03 PM

Mentioned in SAL (#wikimedia-operations) [2026-02-26T17:03:27Z] <javiermonton@deploy2002> Finished scap sync-world: Backport for [[gerrit:1244629|component: mediawiki.page_html_content_change.dev0 (T418467)]] (duration: 11m 55s)

gerritbot added a comment.Feb 27 2026, 3:06 PM

Change #1245383 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[operations/deployment-charts@master] stream: mw-page-html-content-change-enrich-next

https://gerrit.wikimedia.org/r/1245383

gerritbot added a comment.Feb 27 2026, 3:17 PM

Change #1245383 merged by jenkins-bot:

[operations/deployment-charts@master] stream: mw-page-html-content-change-enrich-next

https://gerrit.wikimedia.org/r/1245383

gerritbot added a comment.Feb 27 2026, 4:43 PM

Change #1245410 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[operations/deployment-charts@master] stream: mediawiki.page_html_content_change

https://gerrit.wikimedia.org/r/1245410

gerritbot added a comment.Mar 2 2026, 9:34 AM

Change #1245410 merged by jenkins-bot:

[operations/deployment-charts@master] stream: mediawiki.page_html_content_change

https://gerrit.wikimedia.org/r/1245410

JMonton-WMF moved this task from In progress to In Review on the Data-Engineering (Q3 FY25/26 January 1st - March 31th) board.Mar 3 2026, 5:12 PM
CodeReviewBot added a comment.Mar 4 2026, 8:56 AM

javiermonton merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/110

Enrich Parent HTML with Diffs

gerritbot added a comment.Mar 4 2026, 10:44 AM

Change #1247962 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[operations/deployment-charts@master] stream: mw-page-html-content-change-enrich-next

https://gerrit.wikimedia.org/r/1247962

JMonton-WMF created subtask T418996: Audit and fix observability (logging and metrics) for pyflink jobs.Mar 4 2026, 11:14 AM
JMonton-WMF moved this task from In Review to Done on the Data-Engineering (Q3 FY25/26 January 1st - March 31th) board.Mar 4 2026, 11:17 AM
gerritbot added a comment.Mar 5 2026, 9:06 AM

Change #1247962 merged by JavierMonton:

[operations/deployment-charts@master] stream: mw-page-html-content-change-enrich-next

https://gerrit.wikimedia.org/r/1247962

JMonton-WMF created subtask T419258: Adatp HTML pipeline to the new diffs schema.Mar 6 2026, 3:33 PM
CodeReviewBot added a comment.Mar 13 2026, 11:36 AM

javiermonton merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/122

Fix make diff

Ottomata moved this task from Backlog to Stream Data Products on the Event-Platform board.Mar 16 2026, 2:14 PM
AKhatun_WMF closed subtask T419258: Adatp HTML pipeline to the new diffs schema as Resolved.Mar 16 2026, 4:57 PM
Ahoelzl edited projects, added Data-Engineering (Q4 FS25/26 April 1st - June 30st); removed Data-Engineering (Q3 FY25/26 January 1st - March 31th).Fri, Apr 3, 10:26 PM
Ahoelzl moved this task from Next Up to Done on the Data-Engineering (Q4 FS25/26 April 1st - June 30st) board.
Ahoelzl closed this task as Resolved.Fri, Apr 3, 10:29 PM
Ahoelzl edited projects, added Data-Engineering (Q3 FY25/26 January 1st - March 31th); removed Data-Engineering (Q4 FS25/26 April 1st - June 30st).Fri, Apr 3, 10:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits