prevent modules with broken paraminfo being deployed to production wikis
Open, Needs TriagePublic

Description

Given T100775 is fixed, I think there's nothing else for me here (this task is for a pywikibot-specific issue, yes?), so unusbscribing. Thanks for the heads up though.

@greg, this is a general problem with CI and release. API clients shouldn't need to detect defective API modules on WMF servers, or any MediaWiki installation for that matter.

In addition to T100775: fancycaptchareload paraminfo failure on test wikis (May) , T108317: paraminfo for flow submodule view-topic-history fails is another case which has occurred now (August). Those are the two that the Pywikibot team is aware of, as they have broken Pywikibot tests. Have there been other cases in 2015?

IMO WMF needs unit tests to verify that changes dont break paraminfo. T100839: Run test suites for all captcha types in ConfirmEdit has gone nowhere.

Also, integration tests running on beta should detect broken modules. T100903: Run pywikibot test suite regularly on beta cluster as part of MediaWiki/Wikimedia CI would provide this. Fixing T28680: Add a way to allow action=paraminfo to dump results for all modules would also make testing this much easier, as one API call would test every module.

Finally, IMO MediaWiki paraminfo API should also better handle when a broken module is loaded to the live wiki, instead of 503 being considered an appropriate API response.

jayvdb created this task.Aug 7 2015, 1:54 PM
jayvdb updated the task description. (Show Details)
jayvdb raised the priority of this task from to Needs Triage.
jayvdb added subscribers: jayvdb, greg.
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptAug 7 2015, 1:54 PM
Restricted Application added a subscriber: Luke081515. · View Herald TranscriptAug 7 2015, 1:57 PM
Anomie added a subscriber: Anomie.Aug 9 2015, 10:24 PM

Finally, IMO MediaWiki paraminfo API should also better handle when a broken module is loaded to the live wiki, instead of 503 being considered an appropriate API response.

The API should be able to assume that extensions don't register unloadable classes as API modules. Violation of that basic assumption appears to have been the cause in both T100775 and T108317.

The API should be able to assume that extensions don't register unloadable classes as API modules. Violation of that basic assumption appears to have been the cause in both T100775 and T108317.

Yeah it should. We can probably add a structure test for this? Something that instantiates the API and attempts to instantiate all the available submodules and recurse?

The API should be able to assume that extensions don't register unloadable classes as API modules. Violation of that basic assumption appears to have been the cause in both T100775 and T108317.

Yeah it should. We can probably add a structure test for this? Something that instantiates the API and attempts to instantiate all the available submodules and recurse?

There is a test: https://gerrit.wikimedia.org/r/#/c/173874/ but each module with submodules have to add it own test, because there is no recursion (Which seems not done by Flow).

The Captcha problem would not detected because not all captcha types where tested (and that would not change a structure test because the list task is not resolved).

Krinkle set Security to None.
Krinkle removed a subscriber: pywikibot-bugs-list.
hashar added a subscriber: hashar.

We removed Beta-Cluster-Infrastructure because there is not much we can do on it, although it will certainly be used by a yet-to-be-written test suite.

Krinkle moved this task from Inbox to PHPUnit on the MediaWiki-Core-Tests board.Jul 7 2017, 5:15 AM