Page MenuHomePhabricator
Create Task
Maniphest T363421

Prototype selective HTML updates in Parsoid
Closed, ResolvedPublic
Actions

Description

From WE5.3 Draft,

Draft hypothesis idea: On template edits, if we can implement an algorithm in Parsoid to reuse HTML of a page that depends on the edited template without processing the page from scratch and demonstrate 1.5x or higher processing speedup, we will have a potential incremental parsing solution for efficient page updates on template edits.

NOTE: We are only planning to implement this in the Parsoid library and test it on the command line. The actual integration with the processing pipeline will be followup work and will be more involved. In this prototype, we will start with templates that produce well-balanced DOM fragments.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Bump wikimedia/parsoid to 0.21.0-a3mediawiki/vendormaster+99 -60
Bump wikimedia/parsoid to 0.21.0-a2mediawiki/vendormaster+1 K -148
More selective update testingmediawiki/services/parsoidmaster+41 -12
Add a test for selective updatesmediawiki/services/parsoidmaster+69 -0
Record the last modification time (page_touched) for transclusionsmediawiki/services/parsoidmaster+34 -0
Provide previous parse results to parser when renderingmediawiki/coremaster+193 -52
Bump wikimedia/parsoid to 0.20.0-a18mediawiki/vendormaster+10 K -194
Add DataAccess::fetchTemplateTouched() for Parsoid dependency trackingmediawiki/coremaster+36 -1
Selective HTML Updatesmediawiki/services/parsoidmaster+335 -143
Customize query in gerrit

Related Objects
Search...

Event Timeline

ABreault-WMF created this task.Apr 24 2024, 10:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 24 2024, 10:48 PM
ABreault-WMF moved this task from Backlog to Hacking projects on the Wikimedia-Hackathon-2024 board.Apr 24 2024, 10:48 PM
Lofhi subscribed.Apr 25 2024, 12:52 AM
SDelbecque-WMF subscribed.May 2 2024, 6:40 PM
ABreault-WMF renamed this task from Prototype incremental parsing in Parsoid to Prototype selective HTML updates in Parsoid.May 3 2024, 2:59 PM
gerritbot added a comment.May 3 2024, 4:54 PM

Change #1026985 had a related patch set uploaded (by Arlolra; author: Arlolra):

[mediawiki/services/parsoid@master] [WIP] Selective HTML Updates

https://gerrit.wikimedia.org/r/1026985

gerritbot added a project: Patch-For-Review.May 3 2024, 4:54 PM
ABreault-WMF added a comment.May 5 2024, 2:52 PM

@ssastry reports on Slack,

very early iniital results show 5x-10x speedup on reparsing of pages after template edits (as reproduced on our local wikis).

╰─➤  time MW_INSTALL_PATH=../core php bin/parse.php --integrated --pageName "Hampi"  < /dev/null > /tmp/hampi.html
 MW_INSTALL_PATH=../core php bin/parse.php --integrated --pageName "Hampi" <    3.07s user 0.24s system 71% cpu 4.618 total
╭─subbu@earth ~/work/wmf/parsoid  ‹T363421*› 
╰─➤  time MW_INSTALL_PATH=../core php bin/parse.php --integrated --pageName "Hampi" --selpar --revtextfile /tmp/hampi.wt --revhtmlfile /tmp/hampi.html --editedtemplatetitle "About" < /dev/null > /tmp/hampi.new.html
 MW_INSTALL_PATH=../core php bin/parse.php --integrated --pageName "Hampi"      0.36s user 0.05s system 89% cpu 0.459 total

Rest of this work will happen in Q1. Finishing up, addressing FIXMEs, polishing it up, adding tests will probably take at least be a week's worth full time work.

Don-vip awarded a token.May 5 2024, 6:16 PM
Lofhi awarded a token.May 5 2024, 6:17 PM
Quiddity mentioned this in T361975: [Session] 🥳 Wikimedia Hackathon 2024 Project Showcase & Closing.May 13 2024, 6:51 PM
ssastry moved this task from Needs Triage to Feature requests on the Parsoid board.May 23 2024, 3:51 PM
ssastry added a project: Content-Transform-Team-WIP.
MSantos added a project: OKR-Work.Jun 3 2024, 2:31 PM
ABreault-WMF triaged this task as High priority.Jun 15 2024, 1:08 AM
gerritbot added a comment.Aug 2 2024, 5:04 PM

Change #1059404 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] Provide previous parse results to parser when rendering

https://gerrit.wikimedia.org/r/1059404

gerritbot added a comment.Aug 23 2024, 1:56 AM

Change #1026985 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Selective HTML Updates

https://gerrit.wikimedia.org/r/1026985

gerritbot added a comment.Aug 23 2024, 11:23 PM

Change #1065296 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/core@master] Add DataAccess::fetchTemplateTouched() for Parsoid dependency tracking

https://gerrit.wikimedia.org/r/1065296

gerritbot added a comment.Aug 23 2024, 11:23 PM

Change #1065297 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/services/parsoid@master] Record the last modification time (page_touched) for transclusions

https://gerrit.wikimedia.org/r/1065297

gerritbot added a comment.Aug 24 2024, 3:37 AM

Change #1059404 merged by jenkins-bot:

[mediawiki/core@master] Provide previous parse results to parser when rendering

https://gerrit.wikimedia.org/r/1059404

ssastry mentioned this in T373258: Build selective update functionality in Parsoid to efficiently support structured fragment generators.Aug 25 2024, 12:48 AM
ssastry added a parent task: T373258: Build selective update functionality in Parsoid to efficiently support structured fragment generators.
gerritbot added a comment.Aug 26 2024, 8:38 AM

Change #1066683 had a related patch set uploaded (by Isabelle Hurbain-Palatin; author: Isabelle Hurbain-Palatin):

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.20.0-a18

https://gerrit.wikimedia.org/r/1066683

gerritbot added a comment.Aug 26 2024, 11:36 AM

Change #1066683 merged by jenkins-bot:

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.20.0-a18

https://gerrit.wikimedia.org/r/1066683

Jdforrester-WMF subscribed.Aug 26 2024, 1:09 PM
ssastry mentioned this in T373776: Parsoid does not correctly render <noinclude> if used with templates.Sep 4 2024, 5:04 PM
gerritbot added a comment.Oct 22 2024, 12:47 AM

Change #1082112 had a related patch set uploaded (by Arlolra; author: Arlolra):

[mediawiki/services/parsoid@master] Add a test for selective updates

https://gerrit.wikimedia.org/r/1082112

gerritbot added a comment.Oct 22 2024, 1:24 AM

Change #1082112 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Add a test for selective updates

https://gerrit.wikimedia.org/r/1082112

gerritbot added a comment.Oct 25 2024, 10:43 PM

Change #1083283 had a related patch set uploaded (by Arlolra; author: Arlolra):

[mediawiki/services/parsoid@master] More selective update testing

https://gerrit.wikimedia.org/r/1083283

gerritbot added a comment.Oct 26 2024, 1:54 PM

Change #1083283 merged by jenkins-bot:

[mediawiki/services/parsoid@master] More selective update testing

https://gerrit.wikimedia.org/r/1083283

gerritbot added a comment.Oct 28 2024, 5:01 PM

Change #1083864 had a related patch set uploaded (by Subramanya Sastry; author: Subramanya Sastry):

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.21.0-a2

https://gerrit.wikimedia.org/r/1083864

gerritbot added a comment.Oct 28 2024, 8:54 PM

Change #1083864 merged by jenkins-bot:

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.21.0-a2

https://gerrit.wikimedia.org/r/1083864

gerritbot added a comment.Nov 4 2024, 11:03 AM

Change #1087149 had a related patch set uploaded (by Isabelle Hurbain-Palatin; author: Isabelle Hurbain-Palatin):

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.21.0-a3

https://gerrit.wikimedia.org/r/1087149

gerritbot added a comment.Nov 4 2024, 3:53 PM

Change #1087149 merged by jenkins-bot:

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.21.0-a3

https://gerrit.wikimedia.org/r/1087149

ssastry mentioned this in T346196: Wikimedia\Assert\InvariantException: Invariant failed: Expected valid DSR.Nov 4 2024, 11:48 PM
ABreault-WMF closed this task as Resolved.Nov 7 2024, 9:28 PM
ssastry mentioned this in T384060: enwiktionary: Parsoid's output includes self-links for fragment links unlike legacy output.Mar 11 2025, 8:19 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits