On Lingo I came accross a little challenge. A couple tests are passing when invoked specifically, but fails as part of the main run.
Note
[X] I have sent https://gerrit.wikimedia.org/r/437459 to make the tests compatible with PHPUnit 6.
Failure:
[ ] 1) Lingo\Tests\Integration\ArticleAnnotationTest::testArticleAnnotation
```
$ php tests/phpunit/phpunit.php --testsuite extensions
There were 2 failures:
1) Lingo\Tests\Integration\ArticleAnnotationTest::testArticleAnnotation with data set #0 ('/latin.xml', 'Lorem ipsum dolor sit amet, p...oluta.', array(array('ipsum', 'ipsum-def', null, null), array('vel', 'vel-def', null, null), array('vel', 'vel-def', null, null), array('mandamus', 'mandamus-def', null, null), array('ius', 'ius-def', null, null), array('modus', 'modus-def', null, null), array('omnes', 'omnes-def', null, null), array('has', 'has-def', null, null)), '<div class="mw-parser-output"...div>\n')
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'<div class="mw-parser-output"><p>Lorem <span class="mw-lingo-term" data-lingo-term-id="e78f5438b48b39bcbdea61b73679449d">ipsum</span> dolor sit amet, pri admodum alienum at, decore facete ei cum,\n
-discere admodum dissentias quo et. Ex iusto antiopam sed, <span class="mw-lingo-term" data-lingo-term-id="689c4f6017e042e16a6315622d4fc11d">ius</span> eu stet periculis\n
-complectitur. Quas apeirian complectitur nec no. Nam <span class="mw-lingo-term" data-lingo-term-id="5ebcdbfc8779f1ac8898f70672713e85">omnes</span> nullam ex. Ne sed\n
+'<div class="mw-parser-output"><p>Lorem ipsum dolor sit amet, pri admodum alienum at, decore facete ei cum,\n
+discere admodum dissentias quo et. Ex iusto antiopam sed, ius eu stet periculis\n
+complectitur. Quas apeirian complectitur nec no. Nam omnes nullam ex. Ne sed\n
eros homero, utroque pertinacia eam eu, in est ubique latine minimum. Vel\n
...
extensions/Lingo/tests/phpunit/Integration/ArticleAnnotationTest.php:86
maintenance/doMaintenance.php:94
```
[X] 2) Lingo\Tests\Unit\ElementTest::testGetFormattedTerm_4
```
2) Lingo\Tests\Unit\ElementTest::testGetFormattedTerm_4
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'Main Page'
+'Main Page (page does not exist)'
extensions/Lingo/tests/phpunit/Unit/ElementTest.php:498
extensions/Lingo/tests/phpunit/Unit/ElementTest.php:189
maintenance/doMaintenance.php:94
```
If I run them directly by using --filter:
```
$ php tests/phpunit/phpunit.php --testsuite extensions --filter ArticleAnnotation
...
OK (3 tests, 3 assertions)
```
```
$ php tests/phpunit/phpunit.php --testsuite extensions --filter testGetFormattedTerm_4
.
OK (1 test, 5 assertions)
```
Ditto using the file that holds the tests:
```
$ php tests/phpunit/phpunit.php --testsuite extensions ../extensions/Lingo/tests/phpunit/Unit/ElementTest.php
............ 12 / 12 (100%)
OK (12 tests, 58 assertions)
```