Page MenuHomePhabricator

Deprecate and remove non-remex Tidy modes of the core parser
Open, Needs TriagePublic

Description

Set the stage for eventual migration of the core parser to Parsoid by deprecating and removing unsupported non-tidy and non-remex-tidy modes of the core parser.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 468039 merged by jenkins-bot:
[mediawiki/extensions/Math@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/468039

Change 468020 merged by jenkins-bot:
[mediawiki/extensions/Collection@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/468020

Change 468040 merged by jenkins-bot:
[mediawiki/extensions/ZeroPortal@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/468040

Change 468036 merged by jenkins-bot:
[mediawiki/extensions/MassMessage@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/468036

Change 468029 merged by jenkins-bot:
[mediawiki/extensions/FlaggedRevs@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/468029

Change 468033 merged by jenkins-bot:
[mediawiki/extensions/MapSources@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/468033

Change 468031 merged by jenkins-bot:
[mediawiki/extensions/FundraiserLandingPage@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/468031

Change 468026 merged by jenkins-bot:
[mediawiki/extensions/ConfirmEdit@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/468026

Change 468017 merged by jenkins-bot:
[mediawiki/extensions/CollaborationKit@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/468017

Change 468857 had a related patch set uploaded (by Paladox; owner: C. Scott Ananian):
[mediawiki/extensions/ConfirmEdit@REL1_32] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/468857

Change 462753 merged by jenkins-bot:
[mediawiki/core@master] Use OutputPage::addWikiTextAsInterface() instead of untidy addWikiText()

https://gerrit.wikimedia.org/r/462753

Change 468908 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/CheckUser@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/468908

Change 468912 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/Translate@master] Avoid untidy calls to OutputPage::addWikiText()

https://gerrit.wikimedia.org/r/468912

Change 469001 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/WikiLexicalData@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/469001

Change 469003 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/SemanticGenealogy@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/469003

Change 469007 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/LinkedWiki@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/469007

Change 469009 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/MOOC@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/469009

Change 468912 merged by jenkins-bot:
[mediawiki/extensions/Translate@master] Avoid untidy calls to OutputPage::addWikiText()

https://gerrit.wikimedia.org/r/468912

Change 468908 merged by jenkins-bot:
[mediawiki/extensions/CheckUser@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/468908

Change 469001 merged by jenkins-bot:
[mediawiki/extensions/WikiLexicalData@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/469001

Change 469015 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/Video@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/469015

Change 469017 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/AdManager@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/469017

Change 469022 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/Cargo@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/469022

Change 469023 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/examples@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/469023

Change 469024 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/PageForms@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/469024

Change 469023 merged by jenkins-bot:
[mediawiki/extensions/examples@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/469023

Change 469009 abandoned by C. Scott Ananian:
Replace deprecated untidy OutputPage::addWikiText() method

Reason:
Repository has been archived.

https://gerrit.wikimedia.org/r/469009

Change 469003 abandoned by C. Scott Ananian:
Replace deprecated untidy OutputPage::addWikiText() method

Reason:
Repository has been archived.

https://gerrit.wikimedia.org/r/469003

Change 469017 merged by jenkins-bot:
[mediawiki/extensions/AdManager@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/469017

Change 469022 abandoned by Yaron Koren:
Replace deprecated untidy OutputPage::addWikiText() method

Reason:
I checked in a corresponding change: https://gerrit.wikimedia.org/r/469454 Thanks for your help!

https://gerrit.wikimedia.org/r/469022

Change 467996 merged by Jforrester:
[mediawiki/extensions/SecurePoll@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/467996

Change 463384 merged by jenkins-bot:
[mediawiki/core@master] Tidy Message::parseAsBlock() by enabling tidy in MessageCache

https://gerrit.wikimedia.org/r/463384

Change 462715 merged by jenkins-bot:
[mediawiki/core@master] Hard deprecate non-tidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/462715

Change 469882 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/core@master] Tidy OutputPage::parse() and OutputPage::parseInternal()

https://gerrit.wikimedia.org/r/469882

Change 469894 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/core@master] Deprecate OutputPage::parse() and OutputPage::parseInline()

https://gerrit.wikimedia.org/r/469894

Change 469901 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/core@master] Replace deprecated calls to OutputPage::parseInline()

https://gerrit.wikimedia.org/r/469901

Change 469934 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/core@master] Replace deprecated calls to OutputPage::parse()

https://gerrit.wikimedia.org/r/469934

Change 469942 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/LiquidThreads@master] Replace deprecated OutputPage::parse/parseInline()

https://gerrit.wikimedia.org/r/469942

Change 469946 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/AbuseFilter@master] Replace deprecated OutputPage::parse/parseInline()

https://gerrit.wikimedia.org/r/469946

Change 469949 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/CentralAuth@master] Replace deprecated OutputPage::parse/parseInline()

https://gerrit.wikimedia.org/r/469949

Change 469951 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/OAuth@master] Replace deprecated OutputPage::parseInline()

https://gerrit.wikimedia.org/r/469951

Change 469958 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/Translate@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469958

Change 469966 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/CodeReview@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469966

Change 469969 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/Collection@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469969

Change 469970 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/ConfirmEdit@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469970

Change 469973 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/DonationInterface@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469973

Change 469981 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/FlaggedRevs@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469981

Change 469986 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/TemplateSandbox@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469986

Change 469987 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/VipsScaler@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469987

Change 469973 merged by jenkins-bot:
[mediawiki/extensions/DonationInterface@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469973

Change 469894 merged by jenkins-bot:
[mediawiki/core@master] Deprecate OutputPage::parse() and OutputPage::parseInline()

https://gerrit.wikimedia.org/r/469894

Change 469901 merged by jenkins-bot:
[mediawiki/core@master] Replace deprecated calls to OutputPage::parseInline()

https://gerrit.wikimedia.org/r/469901

Change 469942 merged by jenkins-bot:
[mediawiki/extensions/LiquidThreads@master] Replace deprecated OutputPage::parse/parseInline()

https://gerrit.wikimedia.org/r/469942

Change 468857 merged by jenkins-bot:
[mediawiki/extensions/ConfirmEdit@REL1_32] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/468857

Change 469949 merged by jenkins-bot:
[mediawiki/extensions/CentralAuth@master] Replace deprecated OutputPage::parse/parseInline()

https://gerrit.wikimedia.org/r/469949

Change 469946 merged by jenkins-bot:
[mediawiki/extensions/AbuseFilter@master] Replace deprecated OutputPage::parse/parseInline()

https://gerrit.wikimedia.org/r/469946

Change 469966 merged by jenkins-bot:
[mediawiki/extensions/CodeReview@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469966

Change 469981 merged by jenkins-bot:
[mediawiki/extensions/FlaggedRevs@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469981

Change 469987 merged by jenkins-bot:
[mediawiki/extensions/VipsScaler@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469987

Change 469970 merged by jenkins-bot:
[mediawiki/extensions/ConfirmEdit@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469970

Change 469986 merged by jenkins-bot:
[mediawiki/extensions/TemplateSandbox@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469986

Change 469951 merged by jenkins-bot:
[mediawiki/extensions/OAuth@master] Replace deprecated OutputPage::parseInline()

https://gerrit.wikimedia.org/r/469951

Change 469934 merged by jenkins-bot:
[mediawiki/core@master] Replace deprecated calls to OutputPage::parse()

https://gerrit.wikimedia.org/r/469934

Change 471104 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/core@REL1_32] Deprecate OutputPage::parse() and OutputPage::parseInline()

https://gerrit.wikimedia.org/r/471104

Change 469969 merged by C. Scott Ananian:
[mediawiki/extensions/Collection@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469969

Change 471170 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/core@master] Replace deprecated call to OutputPage::parse() in HTMLForm

https://gerrit.wikimedia.org/r/471170

Change 471172 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/Collection@master] Replace deprecated OutputPage::parse() in CollectionPageTemplate

https://gerrit.wikimedia.org/r/471172

Change 471173 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/SecurePoll@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/471173

Change 471170 merged by jenkins-bot:
[mediawiki/core@master] Replace deprecated call to OutputPage::parse() in HTMLForm

https://gerrit.wikimedia.org/r/471170

Change 471172 merged by Jforrester:
[mediawiki/extensions/Collection@master] Replace deprecated OutputPage::parse() in CollectionPageTemplate

https://gerrit.wikimedia.org/r/471172

Anomie removed a subscriber: Anomie.Nov 2 2018, 3:23 PM

Change 471104 merged by jenkins-bot:
[mediawiki/core@REL1_32] Deprecate OutputPage::parse() and OutputPage::parseInline()

https://gerrit.wikimedia.org/r/471104

Change 469958 merged by jenkins-bot:
[mediawiki/extensions/Translate@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469958

Change 471173 merged by Jforrester:
[mediawiki/extensions/SecurePoll@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/471173

Change 442108 merged by jenkins-bot:
[mediawiki/core@master] Hard deprecate codepaths where tidy is disabled

https://gerrit.wikimedia.org/r/442108

Change 467972 merged by jenkins-bot:
[mediawiki/core@master] Remove most support for configuring Tidy, including Raggett

https://gerrit.wikimedia.org/r/467972

Change 469015 merged by jenkins-bot:
[mediawiki/extensions/Video@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/469015

Change 469024 abandoned by C. Scott Ananian:
Replace deprecated untidy OutputPage::addWikiText() method

Reason:
Abandoned in favor of Yaron's own version.

https://gerrit.wikimedia.org/r/469024

Change 469007 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/LinkedWiki@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/469007

Change 461797 had a related patch set uploaded (by Krinkle; owner: C. Scott Ananian):
[mediawiki/core@master] WIP: deprecate tests w/o tidy section; add html+tidy modes for all tests

https://gerrit.wikimedia.org/r/461797

(For the record: these are the last two open patches for this task:)

Change 461797 had a related patch set uploaded (by C. Scott Ananian):
[mediawiki/core@master] Deprecate tests w/o tidy section; add html+tidy modes for all tests
https://gerrit.wikimedia.org/r/461797

Change 469882 had a related patch set uploaded (by C. Scott Ananian):
[mediawiki/core@master] Tidy OutputPage::parse() and OutputPage::parseInternal()
https://gerrit.wikimedia.org/r/469882

Hey there, should this be moved to 1.35? The cut is a couple of weeks away. If it needs to go out in 1.34, is there anything I can do to help get it out of the door?

Hey there, should this be moved to 1.35? The cut is a couple of weeks away. If it needs to go out in 1.34, is there anything I can do to help get it out of the door?

I think updating parser tests to get rid of non-tidied parsing modes is probably the biggest chunk of work left. Requires (a) adding html+tidy section to every test lacking it (b) getting rid of the plain html section (c) making tidying the default and thus renaming the html+tidy to just html. In terms of patches, it could be one to three separate patches. But (a) is probably the biggest chunk of work left. Unfortunately, we are sprinting to get parsoid/php out the door by end of september, so any help with (a) is appreciated if we want to get this out in 1.34. What is the release date for 1.35? 6 months out?

Hey there, should this be moved to 1.35? The cut is a couple of weeks away. If it needs to go out in 1.34, is there anything I can do to help get it out of the door?

I think updating parser tests to get rid of non-tidied parsing modes is probably the biggest chunk of work left. Requires (a) adding html+tidy section to every test lacking it (b) getting rid of the plain html section (c) making tidying the default and thus renaming the html+tidy to just html. In terms of patches, it could be one to three separate patches.

Plus follow-up patches in extensions, presumably?

[…]

Unfortunately, we are sprinting to get parsoid/php out the door by end of september, so any help with (a) is appreciated if we want to get this out in 1.34. What is the release date for 1.35? 6 months out?

Yes, ~May 2020.

Hey there, should this be moved to 1.35? The cut is a couple of weeks away. If it needs to go out in 1.34, is there anything I can do to help get it out of the door?

I think updating parser tests to get rid of non-tidied parsing modes is probably the biggest chunk of work left. Requires (a) adding html+tidy section to every test lacking it (b) getting rid of the plain html section (c) making tidying the default and thus renaming the html+tidy to just html. In terms of patches, it could be one to three separate patches.

Plus follow-up patches in extensions, presumably?
[…]

Indeed, yes. :-(

Unfortunately, we are sprinting to get parsoid/php out the door by end of september, so any help with (a) is appreciated if we want to get this out in 1.34. What is the release date for 1.35? 6 months out?

Yes, ~May 2020.

In case it looks like we cannot make the 1.34 cut, I suppose Parsoid/PHP + this will both be part of 1.35.

Krinkle removed a subscriber: Krinkle.Sep 5 2019, 10:37 PM
cscott added a comment.Sep 6 2019, 9:25 PM

I have a multi-stage plan in my head for conversion, starting with 1) deleting existing html/php clauses where we already have an html/php+tidy clause, 2) if a marker at the top of the file is found, always read html/php as html/php+tidy, 3) one by one in core and extensions add the marker to the top of the file and update the tests to match (which should ensure that the diff is readable and only includes actual differences introduced by tidying the output, not bookkeeping changes), and then 4) require the marker/normalizing the naming of the clauses/remove the no-tidy paths in the test case.

The goal of 3 is to have a clean diff to review. If we just add html/php+tidy clauses, they show up as entirely new lines and that makes it hard to see the changes from the previous html/php clauses, and if you change the name of the clause you get a spurious line diff on every test which makes it hard to find the parts which 'really' changed.

Anyway, that's a long way of saying it's unlikely to all make 1.34, although it would be nice to get the "optionally interpret html/php as html/php+tidy" part or some similar mode switch thing to ease the pain w/ extensions which try to maintain backwards compatibility. But the extensions that want to maintain backward compatibility can do so trivially by marking all the clauses as html/php+tidy (even after that's synonymous with html/php) so this might already be taken care of. If there are other structural changes to parser test files we want to make in preparation for the tidy migration (I can't think of any at the moment, other than the clause renaming I've been discussing), then we should do those in the next few weeks.