Page MenuHomePhabricator

Set up test infrastructure for testing the library against standard spec test suites
Open, MediumPublic

Description

DoDo does not currently have any test infrastructure to test it against standard spec test suites.

Domino has tests from W3C, WPT, and html5lib (see https://github.com/fgnass/domino/tree/master/test ). PHP has very minimal tests in https://github.com/php/php-src/tree/master/ext/libxml/tests and https://github.com/php/php-src/tree/master/ext/dom/tests but it would be good to verify compatibility using those as well. For DoDo, we are primarily interested in the DOM pieces of the spec test suites and not the HTML5 tree builder (that is well covered by RemexHtml).

So, we need to figure out how to import the W3C and WPT tests (and if necessary, any other test suites, for ex, WHATWG). Some possibilities:

  1. Some web platform tests are written in an abstract language and may be translated to PHP using XSLT.
  2. Domino (JS) used a more-or-less evil hack to run web-page-based tests; you could port this technique to PHP using a js engine or js2php converter to run the <script> portions of the tests, eg take https://github.com/web-platform-tests/wpt/tree/master/dom/nodes and run the results through js2php to make a PHP version. Most of the JS in these tests is very simple, so you wouldn't need a very smart conversion. But if you did, you could use https://github.com/phpv8/v8js, which would be fine as a dev (not run-time) dependency.
  3. Remex uses html5lib/{encoding,serializer,tokenizer/tree-construction} tests, they are somewhat relevant, but this would not add much value since it would be simply repeating tests done in the Remex library after T269262 is complete. See https://github.com/html5lib/html5lib-tests and https://github.com/web-platform-tests/wpt/tree/master/html/syntax .

The goal is *not* to achieve 100% tests passing on 100% conversion of a test suite. Instead what we're trying to do is get "as many tests passing" on "as many of the relevant tests as possible". Not all of the tests will be relevant -- for example they may target legacy HTML features we don't support, or they may test details of the JS WebIDL (https://developer.mozilla.org/en-US/docs/Mozilla/WebIDL_bindings) binding that don't make sense for the PHP WebIDL binding (T271724). And not all of the relevant tests will be runnable -- some may be written in ways which make assumptions about a browser and JS context that can't be easily translated to DoDo, and that's fine, we may in the future write PHP-native tests to cover these features or contribute changes upstream to make the upstream test less browser-specific. But the goal *should* be to run the tests from upstream "unmodified" (aka, automatically translated, not manually ported) -- the platform tests are quite frequently updated, and new features get added to the web platform on a regular basis. It should be as simple as possible to update the tests from upstream and verify that we are in compliance with the latest standard behavior (or reveal that we have not yet implemented newly-added HTML element <foo> or whatever).

Thus, the DoDo test infrastructure will likely have several kinds of "test selection" and "known failure" lists, where we identify certain subsets of the upstream test that we think make sense to run, and then compare the runs of running these against a checked-into-git list of "known failures". Tests should fail on both an unexpected fail *and* an unexpected pass -- that is how we ensure that the list of 'known failures' is kept up to date simultaneously with code changes which affect the results either positively or negatively. (This is different from the 'skip' mechanism built into many test runners, which typically doesn't provide any feedback on tests which newly pass.)

This task will be updated to include more specific information about specific test suites we want to target. Subtasks will cover the work involved in targeting each test suite since each test suite has its own peculiarities and formats and test harnesses.

Details

ProjectBranchLines +/-Subject
mediawiki/libs/Dodomaster+3 K -2 K
mediawiki/libs/Dodomaster+5 K -4 K
mediawiki/libs/Dodomaster+64 -64
mediawiki/libs/IDLeDOMmain+1 K -13
mediawiki/libs/Dodomaster+6 K -4 K
mediawiki/libs/Dodomaster+2 K -10 K
mediawiki/libs/Dodomaster+2 K -5 K
mediawiki/libs/Dodomaster+6 -3
mediawiki/libs/Dodomaster+1 K -2 K
mediawiki/libs/Dodomaster+5 K -4 K
mediawiki/libs/Dodomaster+4 K -4 K
mediawiki/libs/Dodomaster+605 -228
mediawiki/libs/Dodomaster+17 K -1 K
mediawiki/libs/Dodomaster+41 K -17 K
mediawiki/libs/Dodomaster+26 K -2 K
mediawiki/libs/Dodomaster+18 K -155
mediawiki/libs/Dodomaster+410 -259
mediawiki/libs/Dodomaster+1 K -61
mediawiki/libs/Dodomaster+51 -23
mediawiki/libs/Dodomaster+142 -91
mediawiki/libs/Dodomaster+1 -4
mediawiki/libs/Dodomaster+719 -0
mediawiki/libs/Dodomaster+3 -3
Show related patches Customize query in gerrit

Related Objects

Event Timeline

ssastry triaged this task as High priority.Dec 2 2020, 5:58 PM
ssastry lowered the priority of this task from High to Medium.Dec 8 2020, 8:48 PM

Change 661960 had a related patch set uploaded (by Tim.abdullin; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] T269259: Test generator scaffold

https://gerrit.wikimedia.org/r/661960

Change 661965 had a related patch set uploaded (by Tim.abdullin; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] T269259: Remex integration bugfixig

https://gerrit.wikimedia.org/r/661965

Change 661965 had a related patch set uploaded (by C. Scott Ananian; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] Remex integration bugfixig

https://gerrit.wikimedia.org/r/661965

Change 661960 had a related patch set uploaded (by C. Scott Ananian; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] Test generator scaffold

https://gerrit.wikimedia.org/r/661960

Change 661965 merged by jenkins-bot:
[mediawiki/libs/Dodo@master] Remex integration bugfixig

https://gerrit.wikimedia.org/r/661965

Change 662717 had a related patch set uploaded (by Tim.abdullin; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] Moved test generator composer dependecies to general composer; optimized imports; added robo exec composer script

https://gerrit.wikimedia.org/r/662717

Change 661960 abandoned by C. Scott Ananian:
[mediawiki/libs/Dodo@master] Test generator scaffold

Reason:
Squashed this commit with If47265ebabf6ac54271e6b266d82f99462739ee0 for easier review.

https://gerrit.wikimedia.org/r/661960

Change 662780 had a related patch set uploaded (by Tim.abdullin; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] RoboFile and tasks improvements.

https://gerrit.wikimedia.org/r/662780

Change 662782 had a related patch set uploaded (by Tim.abdullin; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] Relative path to folder fix.

https://gerrit.wikimedia.org/r/662782

Change 662782 abandoned by Tim.abdullin:
[mediawiki/libs/Dodo@master] Relative path to folder fix.

Reason:
Not neccesary anymore

https://gerrit.wikimedia.org/r/662782

Change 662780 abandoned by Tim.abdullin:
[mediawiki/libs/Dodo@master] RoboFile and tasks improvements.

Reason:
Not neccesary anymore

https://gerrit.wikimedia.org/r/662780

Change 663547 had a related patch set uploaded (by Tim.abdullin; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] Remex integration bugfixig

https://gerrit.wikimedia.org/r/663547

Change 663887 had a related patch set uploaded (by C. Scott Ananian; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] Remex integration bugfixing (take 2)

https://gerrit.wikimedia.org/r/663887

Change 663887 merged by jenkins-bot:
[mediawiki/libs/Dodo@master] Remex integration bugfixing (take 2)

https://gerrit.wikimedia.org/r/663887

Change 662717 merged by jenkins-bot:
[mediawiki/libs/Dodo@master] Test generator scaffold

https://gerrit.wikimedia.org/r/662717

Change 663547 abandoned by Tim.abdullin:
[mediawiki/libs/Dodo@master] Remex integration bugfixig

Reason:
Most of changes were cherry-picked into another patchset.

https://gerrit.wikimedia.org/r/663547

Change 672675 had a related patch set uploaded (by Tim.abdullin; owner: Tim.abdullin):
[mediawiki/libs/Dodo@master] DominoJS W3C tests improments.

https://gerrit.wikimedia.org/r/672675

Change 676908 had a related patch set uploaded (by Tim.abdullin; author: Tim.abdullin):

[mediawiki/libs/Dodo@master] WPT tests generation improvements

https://gerrit.wikimedia.org/r/676908

Change 672675 merged by jenkins-bot:

[mediawiki/libs/Dodo@master] DominoJS W3C test generation improvements.

https://gerrit.wikimedia.org/r/672675

Change 682021 had a related patch set uploaded (by Tim.abdullin; author: Tim.abdullin):

[mediawiki/libs/Dodo@master] WPT and W3C tests generation improvements

https://gerrit.wikimedia.org/r/682021

Change 676908 abandoned by C. Scott Ananian:

[mediawiki/libs/Dodo@master] WPT and W3C tests generation improvements

Reason:

Superceded by I5736e8bf0d9829581ecdaeac78899b32d1771b55

https://gerrit.wikimedia.org/r/676908

Change 682021 merged by jenkins-bot:

[mediawiki/libs/Dodo@master] WPT and W3C tests generation improvement

https://gerrit.wikimedia.org/r/682021

Change 684879 had a related patch set uploaded (by Tim.abdullin; author: Tim.abdullin):

[mediawiki/libs/Dodo@master] WPT and W3C tests improvements

https://gerrit.wikimedia.org/r/684879

Change 684879 merged by jenkins-bot:

[mediawiki/libs/Dodo@master] WPT and W3C tests improvements

https://gerrit.wikimedia.org/r/684879

Change 689969 had a related patch set uploaded (by Tim.abdullin; author: Tim.abdullin):

[mediawiki/libs/Dodo@master] Test and Dodo methods improvements

https://gerrit.wikimedia.org/r/689969

Change 692868 had a related patch set uploaded (by Tim.abdullin; author: Tim.abdullin):

[mediawiki/libs/Dodo@master] FilteredElementList port; getElementsByTagName implementation

https://gerrit.wikimedia.org/r/692868

Change 692895 had a related patch set uploaded (by Tim.abdullin; author: Tim.abdullin):

[mediawiki/libs/Dodo@master] Tests: fixed count() on string values; rewrite tests on robo build by default

https://gerrit.wikimedia.org/r/692895

Change 692868 abandoned by C. Scott Ananian:

[mediawiki/libs/Dodo@master] Merged changes related to test generator and Dodo development

Reason:

Not needed; did a rebase instead.

https://gerrit.wikimedia.org/r/692868

Change 692959 had a related patch set uploaded (by Tim.abdullin; author: Tim.abdullin):

[mediawiki/libs/Dodo@master] Removed additional not applicable tests; make use of Document::getBody

https://gerrit.wikimedia.org/r/692959

Change 689969 merged by jenkins-bot:

[mediawiki/libs/Dodo@master] Test harness asserts improvements; FilteredElementList port

https://gerrit.wikimedia.org/r/689969

Change 692895 merged by jenkins-bot:

[mediawiki/libs/Dodo@master] Tests: fixed count() on string values; rewrite tests on robo build by default

https://gerrit.wikimedia.org/r/692895

Change 692982 had a related patch set uploaded (by Tim.abdullin; author: Tim.abdullin):

[mediawiki/libs/Dodo@master] Added FilteredElementList::getIterator() to make iterator_to_array work for Zest querySelector

https://gerrit.wikimedia.org/r/692982

Change 692959 merged by jenkins-bot:

[mediawiki/libs/Dodo@master] Removed additional not applicable tests; make use of Document::getBody

https://gerrit.wikimedia.org/r/692959

Change 692982 abandoned by Tim.abdullin:

[mediawiki/libs/Dodo@master] Renamed FilteredElementList::length() to make iterator_to_array work for Zest querySelector

Reason:

https://gerrit.wikimedia.org/r/692982

Change 693471 had a related patch set uploaded (by Tim.abdullin; author: Tim.abdullin):

[mediawiki/libs/Dodo@master] Bug fixing in test generator and small fix for DOMImplementation::hasFeature definition

https://gerrit.wikimedia.org/r/693471

Change 693944 had a related patch set uploaded (by Tim.abdullin; author: Tim.abdullin):

[mediawiki/libs/Dodo@master] Some minor fixes to W3c and Wpt harnesses

https://gerrit.wikimedia.org/r/693944

Change 695461 had a related patch set uploaded (by Tim.abdullin; author: Tim.abdullin):

[mediawiki/libs/Dodo@master] WIP Test log processing development

https://gerrit.wikimedia.org/r/695461

Change 699200 had a related patch set uploaded (by Tim.abdullin; author: Tim.abdullin):

[mediawiki/libs/Dodo@master] Log processing improvements, preserve comments on log yml's rebuild

https://gerrit.wikimedia.org/r/699200

Change 693471 merged by jenkins-bot:

[mediawiki/libs/Dodo@master] Bug fixing in test generator and small fix for DOMImplementation::hasFeature definition

https://gerrit.wikimedia.org/r/693471

Change 693950 had a related patch set uploaded (by C. Scott Ananian; author: Tim.abdullin):

[mediawiki/libs/IDLeDOM@main] Add {Global,DocumentAndElement,Window}EventHandlers

https://gerrit.wikimedia.org/r/693950

Change 699795 had a related patch set uploaded (by Tim.abdullin; author: Tim.abdullin):

[mediawiki/libs/Dodo@master] WPT test harness bug fixing; added robo log:process after phpunit run

https://gerrit.wikimedia.org/r/699795

Change 693944 merged by jenkins-bot:

[mediawiki/libs/Dodo@master] Some fixes to W3c and Wpt harnesses

https://gerrit.wikimedia.org/r/693944

Change 695461 merged by jenkins-bot:

[mediawiki/libs/Dodo@master] Tests exection log processing, convert to readable yml

https://gerrit.wikimedia.org/r/695461

Change 693950 merged by jenkins-bot:

[mediawiki/libs/IDLeDOM@main] Add {Global,DocumentAndElement,Window}EventHandlers

https://gerrit.wikimedia.org/r/693950

Change 699200 merged by jenkins-bot:

[mediawiki/libs/Dodo@master] Log processing improvements, preserve comments on log yml's rebuild

https://gerrit.wikimedia.org/r/699200

Change 699795 merged by jenkins-bot:

[mediawiki/libs/Dodo@master] WPT test harness bug fixing; robo log:process after phpunit run

https://gerrit.wikimedia.org/r/699795

Change 700091 had a related patch set uploaded (by Tim.abdullin; author: Tim.abdullin):

[mediawiki/libs/Dodo@master] Bug fixing and documentation

https://gerrit.wikimedia.org/r/700091

Change 700091 merged by jenkins-bot:

[mediawiki/libs/Dodo@master] Bug fixing and documentation

https://gerrit.wikimedia.org/r/700091