I am not sure, if this is possible for a codesniffer, but it would be nice to validate @covers against the autoloader to see, if the covers makes sense
Feel free to close, if that is not possible.
See test case:
https://gerrit.wikimedia.org/r/#/c/366547/2/tests/phpunit/includes/specials/SpecialCreateTopicPageTest.php
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Legoktm | T171899 Validate @covers tags during normal test runs | |||
| Resolved | MaxSem | T183528 EventLogging has invalid @covers tags | |||
| Resolved | None | T183529 MobileContextTest has invalid @covers tag | |||
| Resolved | Legoktm | T183530 CirrusSearch\UtilTest has invalid @covers tag | |||
| Resolved | Legoktm | T183531 Wikibase has invalid @covers tags | |||
| Resolved | Legoktm | T183604 Can PHPUnit @covers tags cover entire files |
@Legoktm Indeed. For PHPCS to statically be aware of all classes seems unrealistic at this point. The Phan tool tries to do this, and does so quite well, but at a cost (complexity, speed). So in theory perhaps Phan could be made to validate @covers annotation in the same way that it also validates references to classes/methods in real code.
But ultimately, yeah, PHPUnit is the one evaluating these tags, and as you'd expect, it does in fact validate these already. The thing is, however, it only does so when running with coverage enabled. And while we run PHPUnit that way for small repositories (e.g. PHP libraries), we don't run PHPUnit that way for mediawiki-core, because it takes 2-3 hours to run PHPUnit tests for core when enabling coverage. Instead, we run a separate Jenkins job a few times a day. And that separate job does correctly detect invalid @covers values and fail the build accordingly. (But most people don't notice that.)
Given this task was filed based on an issue in an extension, there is a broader issue here: We can't run PHPUnit with coverage on extensions right now. To get coverage reports for extensions, we need to either run PHPUnit "from" the extension repo context somehow (instead of from core with core's phpunit.xml file), or somehow override the coverage whitelist when running tests for an extension.
In addition, once running PHPUnit with coverage is possible for extensions, we can probably do so by default for the test-pipeline jobs for extensions (so that we validate patches early, and also allows viewing Clover reports within Jenkins' UI).
Thanks for the information, than phpcs only can look for wrong tags like "@cover"[1], but not for wrong classes.
[1]https://phabricator.wikimedia.org/rMWda6235ce3a426effc7dc4d30a6c638c9ddd1886f#f714faad
We could add https://github.com/oradwell/covers-validator/tree/0.5 to our CI process?
Re-triaging under MediaWiki-Core-Tests since this is a PHPUnit issue, and not something easily doable in MW-CS.
Change 399775 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/core@master] Verify that all @covers tags are sane when running tests
Change 399950 had a related patch set uploaded (by MaxSem; owner: MaxSem):
[mediawiki/extensions/EventLogging@master] Remove bogus @covers annotation
Change 399956 had a related patch set uploaded (by MaxSem; owner: MaxSem):
[mediawiki/extensions/CirrusSearch@master] Fix @covers annotations
Change 399956 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Fix @covers annotations
Change 399775 merged by jenkins-bot:
[mediawiki/core@master] Verify that all @covers tags are sane when running tests
@covers used in tests extending MediaWikiTestCase are now checked.
Other extends of PHPUnit_Framework_TestCase can use the MediaWikiCoversValidator directly.
Yay! Just announced it too: https://lists.wikimedia.org/pipermail/wikitech-l/2017-December/089349.html