Finally, IMO MediaWiki paraminfo API should also better handle when a broken module is loaded to the live wiki, instead of 503 being considered an appropriate API response.

The API should be able to assume that extensions don't register unloadable classes as API modules. Violation of that basic assumption appears to have been the cause in both T100775 and T108317.

In T108322#1523073, @Anomie wrote:

The API should be able to assume that extensions don't register unloadable classes as API modules. Violation of that basic assumption appears to have been the cause in both T100775 and T108317.

Yeah it should. We can probably add a structure test for this? Something that instantiates the API and attempts to instantiate all the available submodules and recurse?

In T108322#1524624, @Legoktm wrote:

In T108322#1523073, @Anomie wrote:

The API should be able to assume that extensions don't register unloadable classes as API modules. Violation of that basic assumption appears to have been the cause in both T100775 and T108317.

Yeah it should. We can probably add a structure test for this? Something that instantiates the API and attempts to instantiate all the available submodules and recurse?

There is a test: https://gerrit.wikimedia.org/r/#/c/173874/ but each module with submodules have to add it own test, because there is no recursion (Which seems not done by Flow).

The Captcha problem would not detected because not all captcha types where tested (and that would not change a structure test because the list task is not resolved).

It has happened again: T109659: paraminfo for contenttranslationlist is broken on beta wikis

T100903: Run pywikibot test suite regularly on beta cluster as part of MediaWiki/Wikimedia CI is moving ahead to catch these types of problems.

We removed Beta-Cluster-Infrastructure because there is not much we can do on it, although it will certainly be used by a yet-to-be-written test suite.