Page MenuHomePhabricator

Ensure all the features of parse.js are covered by parse.php
Closed, ResolvedPublic

Description

Here's what's currently available in parse.js,

  • --help, -h Show this help message [boolean] [default: false]
  • --version Show version number [boolean]
  • --wt2html Wikitext -> HTML [boolean] [default: false]
  • --html2wt HTML -> Wikitext [boolean] [default: false]
  • --wt2wt Wikitext -> HTML -> Wikitext
  • --html2html HTML -> Wikitext -> HTML
  • --selser Use the selective serializer to go from HTML to
  • --normalize Normalize the output as parserTests would do. Use
  • --body_only Just return the body, without any normalizations
  • --config Path to a config.yaml file. Use --config w/ no [ NOT RELEVANT ]
  • --oldtext The old page text for a selective-serialization
  • --oldtextfile File containing the old page text for a
  • --oldhtmlfile File containing the old HTML for a
  • --domdiff File containing the diff-marked HTML for used
  • --inputfile File containing input as an alternative to stdin
  • --pbin Input pagebundle JSON [default: ""]
  • --pbinfile Input pagebundle JSON file [default: ""]
  • --pboutfile Output pagebundle JSON to file [default: false]
  • --offline Shortcut to turn off various network fetches
  • --record Record http requests for later replay [ NOT RELEVANT ]
  • --replay Replay recorded http requests for later replay [ NOT RELEVANT ]
  • --verbose Log at level "info" as well
  • --useWorker Use a worker farm [boolean] [default: false] [ NOT RELEVANT ]
  • --linting Parse with linter enabled
  • --loadWMF Use WMF mediawiki API config [ WON'T SUPPORT ]
  • --prefix Which wiki prefix to use; e.g. "enwiki" for [ WON'T SUPPORT ]
  • --domain Which wiki to use; e.g. "en.wikipedia.org" for
  • --oldid Oldid of the given page. [default: null]
  • --outputContentVersion The acceptable content version.[default: "2.1.0"]
  • --pageName The page name, returned for {{PAGENAME}}. If no
  • --pageBundle Output pagebundle JSON [boolean] [default: false]
  • --scrubWikitext Apply wikitext scrubbing while serializing. This
  • --contentmodel The content model of the input. Defaults to
  • --wrapSections Output <section> tags (default false)
  • --debug Provide optional flags. Use --debug=help for
  • --trace Use --trace=help for supported options
  • --dump Dump state. Use --dump=help for supported options
  • --genTest Generates token transformer and DOM pass tests. [ WON'T SUPPORT ]
  • --genTestOut Output file to use for token transformer tests [ WON'T SUPPORT ]
  • --genDirectory Output directory to use for DOM tests [ WON'T SUPPORT ]
  • --genTestFragments Enable fragment generation in DOM genTest output [ WON'T SUPPORT ]
  • --fetchConfig Whether to fetch the wiki config from the server [ NOT RELEVANT ]
  • --fetchTemplates Whether to fetch included templates recursively [ NOT RELEVANT ]
  • --fetchImageInfo Whether to fetch image info via the API [ NOT RELEVANT ]
  • --expandExtensions Whether we should request extension tag [ NOT RELEVANT ]
  • --usePHPPreProcessor Whether to use the PHP preprocessor to expand [ NOT RELEVANT ]
  • --addHTMLTemplateParameters Parse template parameters to HTML and add them to
  • --maxdepth Maximum expansion depth [default: 40]
  • --apiURL http path to remote API, e.g.
  • --rtTestMode Test in rt test mode (changes some parse &
  • --useBatchAPI Turn on/off the API batching system [ NOT RELEVANT ]
  • --phpConfigFile PHP config to splice into the JS pipelines [ NOT RELEVANT ]
  • --color Enable color output Ex: --no-color

vs what's in parse.php,

> php bin/parse.php --help

Omnibus script to convert between wikitext and HTML, and roundtrip wikitext or
HTML. Supports a number of options pertaining to pointing at a specific wiki or
enabling various features during these transformations.

If no options are provided, --wt2html is enabled by default.
See --help for detailed usage help.

Usage: php parse.php [--body_only|--help|--html2wt|--oldhtmlfile|--oldtextfile|--quiet|--selser|--wt2html]

    --help (-1): Display this help message
    --quiet (-1): Whether to suppress non-error output

Script specific parameters:
    --body_only: Just return the body, without any normalizations as in
        --normalize
    --html2wt: HTML -> Wikitext
    --oldhtmlfile: File containing the old HTML for a
        selective-serialization (see --selser)
    --oldtextfile: File containing the old page text for a
        selective-serialization (see --selser)
    --selser: Use the selective serializer to go from HTML to Wikitext.
    --wt2html: Wikitext -> HTML

Details

SubjectRepoBranchLines +/-
mediawiki/services/parsoidmaster+19 -1
mediawiki/services/parsoidmaster+38 -2
mediawiki/services/parsoidmaster+41 -2
mediawiki/services/parsoidmaster+13 -0
mediawiki/services/parsoidmaster+43 -7
mediawiki/services/parsoidmaster+22 -3
mediawiki/services/parsoidmaster+26 -16
mediawiki/services/parsoidmaster+27 -2
mediawiki/services/parsoidmaster+10 -1
mediawiki/services/parsoidmaster+46 -16
mediawiki/services/parsoidmaster+15 -2
mediawiki/services/parsoidmaster+22 -2
mediawiki/services/parsoidmaster+16 -17
mediawiki/services/parsoidmaster+16 -4
mediawiki/services/parsoidmaster+53 -10
mediawiki/services/parsoidmaster+37 -19
mediawiki/services/parsoidmaster+20 -8
mediawiki/services/parsoidmaster+44 -30
mediawiki/services/parsoidmaster+42 -19
mediawiki/services/parsoidmaster+10 -1
mediawiki/services/parsoidmaster+28 -12
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Trying to address T178914: parse.js should have standard arguments as part of this port might be something worth considering.

Trying to address T178914: parse.js should have standard arguments as part of this port might be something worth considering.

I could address the specific points mentioned there, but I would rather not be on the hook for coming up with a holistic vision

Trying to address T178914: parse.js should have standard arguments as part of this port might be something worth considering.

I could address the specific points mentioned there, but I would rather not be on the hook for coming up with a holistic vision

Ok ... since we might be running both parse.js and parse.php for a while during the port, I take back my recommendation (which was premised on the thought that maybe we don't need to backport the changes from the PHP version to the JS version). So, ok to just arrive at feature parity first.

Change 524915 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add option for inputfile to bin/parse.php

https://gerrit.wikimedia.org/r/524915

Arlolra updated the task description. (Show Details)

Change 525128 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid@master] Add scrubWikitext option to bin/parse.php

https://gerrit.wikimedia.org/r/525128

Change 524917 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add pageName option to bin/parse.php

https://gerrit.wikimedia.org/r/524917

Change 525128 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add scrubWikitext option to bin/parse.php

https://gerrit.wikimedia.org/r/525128

Change 525155 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid@master] Add wrapSection option to bin/parse.php

https://gerrit.wikimedia.org/r/525155

Change 525155 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add wrapSection option to bin/parse.php

https://gerrit.wikimedia.org/r/525155

Arlolra updated the task description. (Show Details)

Change 525166 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid@master] Add oldtextfile option to bin/parse.php

https://gerrit.wikimedia.org/r/525166

There should probably be better test coverage of this bin or at least fixing this,
https://github.com/wikimedia/parsoid/blob/master/bin/toolcheck.sh#L19-L21

Edit: That part got done in T228217#5362937

Change 525166 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add oldtextfile option to bin/parse.php

https://gerrit.wikimedia.org/r/525166

Change 525312 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid@master] Run the toolcheck with bin/parse.php

https://gerrit.wikimedia.org/r/525312

Change 525333 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid@master] Add addHTMLTemplateParameters options to bin/parse.php

https://gerrit.wikimedia.org/r/525333

Change 525341 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid@master] Add rtTestMode option to bin/parse.php

https://gerrit.wikimedia.org/r/525341

Change 525312 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Run the toolcheck with bin/parse.php

https://gerrit.wikimedia.org/r/525312

Change 525333 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add addHTMLTemplateParameters options to bin/parse.php

https://gerrit.wikimedia.org/r/525333

Change 525341 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add rtTestMode option to bin/parse.php

https://gerrit.wikimedia.org/r/525341

@Sbailey is going to be implementing the "--domain" functionality

Change 533616 had a related patch set uploaded (by Sbailey; owner: Sbailey):
[mediawiki/services/parsoid@master] Added --linter true option to parse.php

https://gerrit.wikimedia.org/r/533616

Change 533610 had a related patch set uploaded (by Subramanya Sastry; owner: Sbailey):
[mediawiki/services/parsoid@master] Add --domain and --apiURL support to parse.php

https://gerrit.wikimedia.org/r/533610

Change 533610 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add --domain and --apiURL support to parse.php

https://gerrit.wikimedia.org/r/533610

Change 533616 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Added --lint option to parse.php

https://gerrit.wikimedia.org/r/533616

ssastry updated the task description. (Show Details)

Change 539970 had a related patch set uploaded (by Subramanya Sastry; owner: Sbailey):
[mediawiki/services/parsoid@master] Add --trace --dump and --debug command line option support

https://gerrit.wikimedia.org/r/539970

Change 539970 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add --trace --dump and --debug command line option support

https://gerrit.wikimedia.org/r/539970

ssastry lowered the priority of this task from Medium to Low.Oct 22 2019, 8:54 PM

Change 553429 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid@master] Add --oldid option to bin/parse.php

https://gerrit.wikimedia.org/r/553429

Change 553430 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid@master] Add --normalize option to bin/parse.php

https://gerrit.wikimedia.org/r/553430

Change 553429 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add --oldid option to bin/parse.php

https://gerrit.wikimedia.org/r/553429

Change 553430 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add --normalize option to bin/parse.php

https://gerrit.wikimedia.org/r/553430

Change 554394 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add --outputContentVersion option to bin/parse.php

https://gerrit.wikimedia.org/r/554394

Change 559608 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid@master] Add --pageBundle option to bin/parse.php

https://gerrit.wikimedia.org/r/559608

Change 559608 abandoned by Arlolra:
Add --pageBundle option to bin/parse.php

Reason:
In favour of I5ee02872932c3663009b51797926260655226a9f

https://gerrit.wikimedia.org/r/559608

Change 559621 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid@master] Add --verbose option to bin/parse.php

https://gerrit.wikimedia.org/r/559621

record and replay aren't really relevant in Parsoid/PHP

color was a standard option that doesn't seem to have been used in parse.js

Change 559611 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/services/parsoid@master] Add --pageBundle and --pboutfile options to parse.php

https://gerrit.wikimedia.org/r/559611

Change 559631 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid@master] Add --maxdepth option to bin/parse.php

https://gerrit.wikimedia.org/r/559631

offline is pretty much covered by --mock and can be further discussed in T229469

contentmodel support hasn't been ported Parsoid/PHP yet, so that can be done with T236112

Change 559611 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add --pageBundle and --pboutfile options to parse.php

https://gerrit.wikimedia.org/r/559611

Arlolra updated the task description. (Show Details)

Change 559621 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add --verbose option to bin/parse.php

https://gerrit.wikimedia.org/r/559621

Change 559631 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add --maxdepth option to bin/parse.php

https://gerrit.wikimedia.org/r/559631

config is not really relevant. We could point it at a file with $wgParsoidSettings, like in,
https://github.com/wikimedia/parsoid/blob/master/tests/RTTestSettings.php
but for now using LocalSettings.php in integrated mode should suffice.

Arlolra updated the task description. (Show Details)

Change 559922 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid@master] [WIP] Add --version option to bin/parse.php

https://gerrit.wikimedia.org/r/559922

Change 559989 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid@master] Add --pbin(file) options to bin/parse.php

https://gerrit.wikimedia.org/r/559989

Change 559989 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add --pbin(file) options to bin/parse.php

https://gerrit.wikimedia.org/r/559989

Arlolra claimed this task.
Arlolra updated the task description. (Show Details)

Change 559922 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Add --version option to bin/parse.php

https://gerrit.wikimedia.org/r/559922