Page MenuHomePhabricator

Set up test infrastructure for testing the library against standard spec test suites
Open, MediumPublic

Description

DoDo does not currently have any test infrastructure to test it against standard spec test suites.

Domino has tests from W3C, WPT, and html5lib (see https://github.com/fgnass/domino/tree/master/test ). PHP has very minimal tests in https://github.com/php/php-src/tree/master/ext/libxml/tests and https://github.com/php/php-src/tree/master/ext/dom/tests but it would be good to verify compatibility using those as well. For DoDo, we are primarily interested in the DOM pieces of the spec test suites and not the HTML5 tree builder (that is well covered by RemexHtml).

So, we need to figure out how to import the W3C and WPT tests (and if necessary, any other test suites, for ex, WHATWG). Some possibilities:

  1. Some web platform tests are written in an abstract language and may be translated to PHP using XSLT.
  2. Domino (JS) used a more-or-less evil hack to run web-page-based tests; you could port this technique to PHP using a js engine or js2php converter to run the <script> portions of the tests, eg take https://github.com/web-platform-tests/wpt/tree/master/dom/nodes and run the results through js2php to make a PHP version. Most of the JS in these tests is very simple, so you wouldn't need a very smart conversion. But if you did, you could use https://github.com/phpv8/v8js, which would be fine as a dev (not run-time) dependency.
  3. Remex uses html5lib/{encoding,serializer,tokenizer/tree-construction} tests, they are somewhat relevant, but this would not add much value since it would be simply repeating tests done in the Remex library after T269262 is complete. See https://github.com/html5lib/html5lib-tests and https://github.com/web-platform-tests/wpt/tree/master/html/syntax .

The goal is *not* to achieve 100% tests passing on 100% conversion of a test suite. Instead what we're trying to do is get "as many tests passing" on "as many of the relevant tests as possible". Not all of the tests will be relevant -- for example they may target legacy HTML features we don't support, or they may test details of the JS WebIDL (https://developer.mozilla.org/en-US/docs/Mozilla/WebIDL_bindings) binding that don't make sense for the PHP WebIDL binding (T271724). And not all of the relevant tests will be runnable -- some may be written in ways which make assumptions about a browser and JS context that can't be easily translated to DoDo, and that's fine, we may in the future write PHP-native tests to cover these features or contribute changes upstream to make the upstream test less browser-specific. But the goal *should* be to run the tests from upstream "unmodified" (aka, automatically translated, not manually ported) -- the platform tests are quite frequently updated, and new features get added to the web platform on a regular basis. It should be as simple as possible to update the tests from upstream and verify that we are in compliance with the latest standard behavior (or reveal that we have not yet implemented newly-added HTML element <foo> or whatever).

Thus, the DoDo test infrastructure will likely have several kinds of "test selection" and "known failure" lists, where we identify certain subsets of the upstream test that we think make sense to run, and then compare the runs of running these against a checked-into-git list of "known failures". Tests should fail on both an unexpected fail *and* an unexpected pass -- that is how we ensure that the list of 'known failures' is kept up to date simultaneously with code changes which affect the results either positively or negatively. (This is different from the 'skip' mechanism built into many test runners, which typically doesn't provide any feedback on tests which newly pass.)

This task will be updated to include more specific information about specific test suites we want to target. Subtasks will cover the work involved in targeting each test suite since each test suite has its own peculiarities and formats and test harnesses.

Event Timeline

ssastry created this task.Dec 2 2020, 5:05 PM
ssastry triaged this task as High priority.Dec 2 2020, 5:58 PM
ssastry lowered the priority of this task from High to Medium.Dec 8 2020, 8:48 PM
cscott updated the task description. (Show Details)Jan 11 2021, 3:51 PM
cscott updated the task description. (Show Details)Jan 11 2021, 3:54 PM
cscott updated the task description. (Show Details)Jan 11 2021, 4:44 PM
ssastry edited projects, added Parsoid (Dodo); removed Parsoid.Jan 14 2021, 7:27 PM

Change 661960 had a related patch set uploaded (by Tim.abdullin; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] T269259: Test generator scaffold

https://gerrit.wikimedia.org/r/661960

Change 661965 had a related patch set uploaded (by Tim.abdullin; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] T269259: Remex integration bugfixig

https://gerrit.wikimedia.org/r/661965

Change 661965 had a related patch set uploaded (by C. Scott Ananian; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] Remex integration bugfixig

https://gerrit.wikimedia.org/r/661965

Change 661960 had a related patch set uploaded (by C. Scott Ananian; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] Test generator scaffold

https://gerrit.wikimedia.org/r/661960

Change 661965 merged by jenkins-bot:
[mediawiki/libs/Dodo@master] Remex integration bugfixig

https://gerrit.wikimedia.org/r/661965

Change 662717 had a related patch set uploaded (by Tim.abdullin; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] Moved test generator composer dependecies to general composer; optimized imports; added robo exec composer script

https://gerrit.wikimedia.org/r/662717

Change 661960 abandoned by C. Scott Ananian:
[mediawiki/libs/Dodo@master] Test generator scaffold

Reason:
Squashed this commit with If47265ebabf6ac54271e6b266d82f99462739ee0 for easier review.

https://gerrit.wikimedia.org/r/661960

Change 662780 had a related patch set uploaded (by Tim.abdullin; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] RoboFile and tasks improvements.

https://gerrit.wikimedia.org/r/662780

Change 662782 had a related patch set uploaded (by Tim.abdullin; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] Relative path to folder fix.

https://gerrit.wikimedia.org/r/662782

Change 662782 abandoned by Tim.abdullin:
[mediawiki/libs/Dodo@master] Relative path to folder fix.

Reason:
Not neccesary anymore

https://gerrit.wikimedia.org/r/662782

Change 662780 abandoned by Tim.abdullin:
[mediawiki/libs/Dodo@master] RoboFile and tasks improvements.

Reason:
Not neccesary anymore

https://gerrit.wikimedia.org/r/662780

Change 663547 had a related patch set uploaded (by Tim.abdullin; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] Remex integration bugfixig

https://gerrit.wikimedia.org/r/663547

Change 663887 had a related patch set uploaded (by C. Scott Ananian; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] Remex integration bugfixing (take 2)

https://gerrit.wikimedia.org/r/663887

Change 663887 merged by jenkins-bot:
[mediawiki/libs/Dodo@master] Remex integration bugfixing (take 2)

https://gerrit.wikimedia.org/r/663887