Page MenuHomePhabricator

Deprecate and remove non-remex Tidy modes of the core parser
Open, MediumPublic

Description

Set the stage for eventual migration of the core parser to Parsoid by deprecating and removing unsupported non-tidy and non-remex-tidy modes of the core parser.

Details

ProjectBranchLines +/-Subject
mediawiki/coremaster+7 -3
mediawiki/extensions/BlueSpiceSocialmaster+1 -31
mediawiki/extensions/BlueSpiceFoundationmaster+1 -2
mediawiki/extensions/TextExtractsmaster+20 -29
mediawiki/coremaster+45 -327
mediawiki/coremaster+5 -7
mediawiki/extensions/FundraisingEmailUnsubscribemaster+4 -1
mediawiki/coremaster+403 -436
mediawiki/coremaster+81 -43
mediawiki/extensions/TinyMCEmaster+5 -5
mediawiki/skins/Nimbusmaster+5 -2
mediawiki/extensions/WikiArticleFeedsmaster+1 -1
mediawiki/coremaster+8 -171
mediawiki/extensions/WikiForummaster+2 -2
mediawiki/extensions/TranslateSvgmaster+1 -1
mediawiki/extensions/TestLanguageNameGrammarmaster+4 -1
mediawiki/extensions/QuizGamemaster+2 -2
mediawiki/extensions/PollNYmaster+2 -2
mediawiki/extensions/PictureGamemaster+3 -3
mediawiki/extensions/Athenamaster+5 -5
mediawiki/extensions/DataTransfermaster+5 -2
mediawiki/extensions/FanBoxesmaster+3 -3
mediawiki/extensions/DidYouKnowmaster+1 -1
mediawiki/extensions/Commentsmaster+2 -2
mediawiki/extensions/PageFormsmaster+6 -6
mediawiki/extensions/SportsTeamsmaster+3 -3
mediawiki/extensions/SocialProfilemaster+4 -4
mediawiki/extensions/ConfirmAccountmaster+4 -4
mediawiki/extensions/Formmaster+6 -3
mediawiki/extensions/PageFormsmaster+6 -4
mediawiki/extensions/Videomaster+13 -11
mediawiki/coremaster+33 -562
mediawiki/coremaster+13 -0
mediawiki/extensions/SecurePollmaster+3 -4
mediawiki/extensions/Translatemaster+74 -8
mediawiki/coreREL1_32+192 -28
mediawiki/extensions/Collectionmaster+1 -1
mediawiki/coremaster+1 -1
mediawiki/extensions/Collectionmaster+5 -5
mediawiki/coremaster+192 -28
mediawiki/coremaster+42 -20
mediawiki/extensions/ConfirmEditmaster+1 -1
mediawiki/extensions/OAuthmaster+4 -4
mediawiki/extensions/AbuseFiltermaster+5 -7
mediawiki/extensions/CodeReviewmaster+3 -3
mediawiki/extensions/ConfirmEditREL1_32+2 -2
mediawiki/extensions/CentralAuthmaster+6 -8
mediawiki/extensions/TemplateSandboxmaster+10 -3
mediawiki/extensions/VipsScalermaster+4 -1
mediawiki/extensions/FlaggedRevsmaster+2 -2
mediawiki/coremaster+9 -7
mediawiki/extensions/LiquidThreadsmaster+12 -7
mediawiki/extensions/DonationInterfacemaster+11 -1
mediawiki/extensions/Translatemaster+47 -13
mediawiki/coremaster+9 -0
mediawiki/coremaster+5 -3
mediawiki/extensions/SecurePollmaster+9 -9
mediawiki/extensions/Cargomaster+23 -8
mediawiki/extensions/AdManagermaster+13 -13
mediawiki/extensions/SemanticGenealogymaster+16 -16
mediawiki/extensions/MOOCmaster+16 -13
mediawiki/extensions/examplesmaster+8 -5
mediawiki/extensions/CheckUsermaster+4 -5
mediawiki/extensions/WikiLexicalDatamaster+24 -24
mediawiki/coremaster+58 -49
mediawiki/extensions/ConfirmEditmaster+2 -2
mediawiki/extensions/CollaborationKitmaster+2 -2
mediawiki/extensions/FundraiserLandingPagemaster+4 -1
mediawiki/extensions/MassMessagemaster+4 -2
mediawiki/extensions/MapSourcesmaster+9 -3
mediawiki/extensions/FlaggedRevsmaster+2 -2
mediawiki/extensions/ZeroPortalmaster+3 -3
mediawiki/extensions/Collectionmaster+4 -4
mediawiki/extensions/Mathmaster+5 -2
mediawiki/extensions/CentralAuthmaster+1 -1
mediawiki/extensions/CodeReviewmaster+5 -5
mediawiki/extensions/ArticlePlaceholdermaster+4 -4
mediawiki/extensions/CodeReviewmaster+5 -5
mediawiki/extensions/Wikibasemaster+1 -1
mediawiki/coremaster+4 -0
mediawiki/coremaster+3 -2
mediawiki/coremaster+30 -6
mediawiki/extensions/ProofreadPagemaster+1 -1
mediawiki/coremaster+172 -18
mediawiki/extensions/Translatemaster+4 -5
mediawiki/coremaster+1 -1
mediawiki/coremaster+9 -5
mediawiki/coremaster+10 -4
mediawiki/extensions/ProofreadPagemaster+1 -1
mediawiki/coremaster+10 -2
mediawiki/coremaster+22 -3
mediawiki/coremaster+20 -11
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 471104 merged by jenkins-bot:
[mediawiki/core@REL1_32] Deprecate OutputPage::parse() and OutputPage::parseInline()

https://gerrit.wikimedia.org/r/471104

Change 469958 merged by jenkins-bot:
[mediawiki/extensions/Translate@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/469958

Change 471173 merged by Jforrester:
[mediawiki/extensions/SecurePoll@master] Replace deprecated OutputPage::parse()

https://gerrit.wikimedia.org/r/471173

Change 442108 merged by jenkins-bot:
[mediawiki/core@master] Hard deprecate codepaths where tidy is disabled

https://gerrit.wikimedia.org/r/442108

Change 467972 merged by jenkins-bot:
[mediawiki/core@master] Remove most support for configuring Tidy, including Raggett

https://gerrit.wikimedia.org/r/467972

Change 469015 merged by jenkins-bot:
[mediawiki/extensions/Video@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/469015

Change 469024 abandoned by C. Scott Ananian:
Replace deprecated untidy OutputPage::addWikiText() method

Reason:
Abandoned in favor of Yaron's own version.

https://gerrit.wikimedia.org/r/469024

Change 469007 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/LinkedWiki@master] Replace deprecated untidy OutputPage::addWikiText() method

https://gerrit.wikimedia.org/r/469007

Change 461797 had a related patch set uploaded (by Krinkle; owner: C. Scott Ananian):
[mediawiki/core@master] WIP: deprecate tests w/o tidy section; add html+tidy modes for all tests

https://gerrit.wikimedia.org/r/461797

(For the record: these are the last two open patches for this task:)

Change 461797 had a related patch set uploaded (by C. Scott Ananian):
[mediawiki/core@master] Deprecate tests w/o tidy section; add html+tidy modes for all tests
https://gerrit.wikimedia.org/r/461797

Change 469882 had a related patch set uploaded (by C. Scott Ananian):
[mediawiki/core@master] Tidy OutputPage::parse() and OutputPage::parseInternal()
https://gerrit.wikimedia.org/r/469882

Hey there, should this be moved to 1.35? The cut is a couple of weeks away. If it needs to go out in 1.34, is there anything I can do to help get it out of the door?

Hey there, should this be moved to 1.35? The cut is a couple of weeks away. If it needs to go out in 1.34, is there anything I can do to help get it out of the door?

I think updating parser tests to get rid of non-tidied parsing modes is probably the biggest chunk of work left. Requires (a) adding html+tidy section to every test lacking it (b) getting rid of the plain html section (c) making tidying the default and thus renaming the html+tidy to just html. In terms of patches, it could be one to three separate patches. But (a) is probably the biggest chunk of work left. Unfortunately, we are sprinting to get parsoid/php out the door by end of september, so any help with (a) is appreciated if we want to get this out in 1.34. What is the release date for 1.35? 6 months out?

Hey there, should this be moved to 1.35? The cut is a couple of weeks away. If it needs to go out in 1.34, is there anything I can do to help get it out of the door?

I think updating parser tests to get rid of non-tidied parsing modes is probably the biggest chunk of work left. Requires (a) adding html+tidy section to every test lacking it (b) getting rid of the plain html section (c) making tidying the default and thus renaming the html+tidy to just html. In terms of patches, it could be one to three separate patches.

Plus follow-up patches in extensions, presumably?

[…]

Unfortunately, we are sprinting to get parsoid/php out the door by end of september, so any help with (a) is appreciated if we want to get this out in 1.34. What is the release date for 1.35? 6 months out?

Yes, ~May 2020.

Hey there, should this be moved to 1.35? The cut is a couple of weeks away. If it needs to go out in 1.34, is there anything I can do to help get it out of the door?

I think updating parser tests to get rid of non-tidied parsing modes is probably the biggest chunk of work left. Requires (a) adding html+tidy section to every test lacking it (b) getting rid of the plain html section (c) making tidying the default and thus renaming the html+tidy to just html. In terms of patches, it could be one to three separate patches.

Plus follow-up patches in extensions, presumably?

[…]

Indeed, yes. :-(

Unfortunately, we are sprinting to get parsoid/php out the door by end of september, so any help with (a) is appreciated if we want to get this out in 1.34. What is the release date for 1.35? 6 months out?

Yes, ~May 2020.

In case it looks like we cannot make the 1.34 cut, I suppose Parsoid/PHP + this will both be part of 1.35.

Krinkle removed a subscriber: Krinkle.Sep 5 2019, 10:37 PM
cscott added a comment.Sep 6 2019, 9:25 PM

I have a multi-stage plan in my head for conversion, starting with 1) deleting existing html/php clauses where we already have an html/php+tidy clause, 2) if a marker at the top of the file is found, always read html/php as html/php+tidy, 3) one by one in core and extensions add the marker to the top of the file and update the tests to match (which should ensure that the diff is readable and only includes actual differences introduced by tidying the output, not bookkeeping changes), and then 4) require the marker/normalizing the naming of the clauses/remove the no-tidy paths in the test case.

The goal of 3 is to have a clean diff to review. If we just add html/php+tidy clauses, they show up as entirely new lines and that makes it hard to see the changes from the previous html/php clauses, and if you change the name of the clause you get a spurious line diff on every test which makes it hard to find the parts which 'really' changed.

Anyway, that's a long way of saying it's unlikely to all make 1.34, although it would be nice to get the "optionally interpret html/php as html/php+tidy" part or some similar mode switch thing to ease the pain w/ extensions which try to maintain backwards compatibility. But the extensions that want to maintain backward compatibility can do so trivially by marking all the clauses as html/php+tidy (even after that's synonymous with html/php) so this might already be taken care of. If there are other structural changes to parser test files we want to make in preparation for the tidy migration (I can't think of any at the moment, other than the clause renaming I've been discussing), then we should do those in the next few weeks.

Change 469882 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/core@master] Remove deprecated non-tidy OutputPage::parse/parseInline methods

https://gerrit.wikimedia.org/r/469882

Change 565102 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/Form@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565102

Change 565102 merged by jenkins-bot:
[mediawiki/extensions/Form@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565102

Change 565103 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/PageForms@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565103

Change 565105 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/SocialProfile@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565105

Change 565106 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/ConfirmAccount@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565106

Change 565107 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/SportsTeams@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565107

Change 565108 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/Athena@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565108

Change 565109 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/Comments@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565109

Change 565110 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/DataTransfer@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565110

Change 565111 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/DidYouKnow@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565111

Change 565114 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/FanBoxes@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565114

Change 565106 merged by Jforrester:
[mediawiki/extensions/ConfirmAccount@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565106

Change 565116 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/FundraisingEmailUnsubscribe@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565116

Change 565103 merged by jenkins-bot:
[mediawiki/extensions/PageForms@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565103

Change 565117 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/PictureGame@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565117

Change 565105 merged by jenkins-bot:
[mediawiki/extensions/SocialProfile@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565105

Change 565119 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/PollNY@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565119

Change 565107 merged by jenkins-bot:
[mediawiki/extensions/SportsTeams@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565107

Change 565121 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/QuizGame@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565121

Change 565122 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/TestLanguageNameGrammar@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565122

Change 565109 merged by jenkins-bot:
[mediawiki/extensions/Comments@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565109

Change 565110 merged by jenkins-bot:
[mediawiki/extensions/DataTransfer@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565110

Change 565111 merged by jenkins-bot:
[mediawiki/extensions/DidYouKnow@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565111

Change 565114 merged by jenkins-bot:
[mediawiki/extensions/FanBoxes@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565114

Change 565123 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/TinyMCE@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565123

Change 565108 merged by Jforrester:
[mediawiki/extensions/Athena@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565108

Change 565117 merged by jenkins-bot:
[mediawiki/extensions/PictureGame@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565117

Change 565126 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/TranslateSvg@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565126

Change 565119 merged by jenkins-bot:
[mediawiki/extensions/PollNY@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565119

Change 565121 merged by jenkins-bot:
[mediawiki/extensions/QuizGame@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565121

Change 565127 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/WikiArticleFeeds@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565127

Change 565128 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/WikiForum@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565128

Change 565129 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/skins/Nimbus@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565129

Change 565122 merged by jenkins-bot:
[mediawiki/extensions/TestLanguageNameGrammar@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565122

Change 565126 merged by jenkins-bot:
[mediawiki/extensions/TranslateSvg@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565126

Change 469882 merged by jenkins-bot:
[mediawiki/core@master] Remove deprecated non-tidy OutputPage::parse/parseInline methods

https://gerrit.wikimedia.org/r/469882

Change 565128 merged by jenkins-bot:
[mediawiki/extensions/WikiForum@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565128

Change 565129 merged by Jack Phoenix:
[mediawiki/skins/Nimbus@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565129

Change 565127 merged by Jforrester:
[mediawiki/extensions/WikiArticleFeeds@master] Replace deprecated OutputPage::parse(), which will be removed from MW

https://gerrit.wikimedia.org/r/565127

Change 565146 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/core@master] Update parserTests to v2 format (default html/php tests to tidy)

https://gerrit.wikimedia.org/r/565146

Change 461797 merged by jenkins-bot:
[mediawiki/core@master] Support html+untidy tests and version flag in parserTests file

https://gerrit.wikimedia.org/r/461797

LGoto assigned this task to cscott.Jan 28 2020, 10:01 PM
LGoto moved this task from Incoming to Doing (Roadmap Critical Path) on the Parsing-Active-Work board.

Change 565146 merged by jenkins-bot:
[mediawiki/core@master] Update parserTests to v2 format (default html/php tests to tidy)

https://gerrit.wikimedia.org/r/565146

ssastry triaged this task as Medium priority.Feb 27 2020, 1:11 AM

Change 585330 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/core@master] Remove codepaths which ran parser in 'untidy' mode

https://gerrit.wikimedia.org/r/585330

Change 585331 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/core@master] Deprecate ParserOptions::getTidy() and ParserOptions::setTidy()

https://gerrit.wikimedia.org/r/585331

Change 585330 merged by jenkins-bot:
[mediawiki/core@master] Remove codepaths which ran parser in 'untidy' mode

https://gerrit.wikimedia.org/r/585330

Change 585331 merged by jenkins-bot:
[mediawiki/core@master] Deprecate ParserOptions::getTidy() and ParserOptions::setTidy()

https://gerrit.wikimedia.org/r/585331

Change 593825 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/BlueSpiceSocial@master] Always tidy the parser output

https://gerrit.wikimedia.org/r/593825

Change 593820 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/core@master] Deprecate a few more tidy-related methods

https://gerrit.wikimedia.org/r/593820

Change 593827 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/BlueSpiceFoundation@master] Use MWTidy::tidy() directly

https://gerrit.wikimedia.org/r/593827

Change 593879 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/extensions/TextExtracts@master] Tidy is no longer configurable in MW 1.35

https://gerrit.wikimedia.org/r/593879

Change 593827 merged by jenkins-bot:
[mediawiki/extensions/BlueSpiceFoundation@master] Use MWTidy::tidy() directly

https://gerrit.wikimedia.org/r/593827

Change 593825 merged by jenkins-bot:
[mediawiki/extensions/BlueSpiceSocial@master] Always tidy the parser output

https://gerrit.wikimedia.org/r/593825

Change 593820 merged by jenkins-bot:
[mediawiki/core@master] Deprecate a few more tidy-related methods

https://gerrit.wikimedia.org/r/593820