Do you mean just the composer tests or the actual MediaWiki PHPUnit tests? The latter would need to run against an older version since 1.31+ require PHP 7...

Does running composer tests with PHP5 actually help to find incompatibilities?

It would be fine if the PHP5 tests are run against an older version of core (it would actually be a good thing), but I don't know if that is possible with the current system. Is it?

In T197561#4296154, @Nikerabbit wrote:

Does running composer tests with PHP5 actually help to find incompatibilities?

It depends what kind of incompatibilities you want to guard against. If someone tries using PHP 7.0 syntax, then MW-CS 19.x plus php5 lint will catch those. But if someone adds a call to a PHP 7 standard library function, it won't get caught unless the code is actually executed in tests.

It would be fine if the PHP5 tests are run against an older version of core (it would actually be a good thing), but I don't know if that is possible with the current system. Is it?

It is possible, DonationInterface currently uses this functionality.

It would be fine if the PHP5 tests are run against an older version of core (it would actually be a good thing), but I don't know if that is possible with the current system. Is it?

It is possible, DonationInterface currently uses this functionality.

DonationInterface is an edge case, it is Wikimedia specific and is deployed on the fundraising cluster which uses REL1_27. So we went instructing CI to target that branch (zuul-cloner --branch REL1_27).

For all other extensions, CI assumes all other repositories target the same branch. So a patch to REL1_31 is tested with MediaWiki REL1_31. There is no back compatibility job though others have expressed that need. An example is the Lingo extension which targets REL1_27. We could have a job that parse extension.json and figure out the target branch based on the minimal Mediawiki requirement. That is the subject of T196467: Ensure extension master branch still works with the indicated MediaWiki version.

I would like to find a generic solution to that problem, and avoid duplicating the DonationInterface example all other the place. It is not sustainable long term.

For Translate etc, they for now are tested with MediaWiki master branch which is dropping support for php 5.5. So soonish the tests would probably no more pass anymore at some near point in the future.

Latest MLEB also requires PHP7 now.