Oh wow, this test runs a full maintenance script.

The mw-debug-cli.split-group-0.log.gz contains some messages related to ChangeWikiConfig, but they don’t look relevant (mainly a bunch of “RequestContext::getTitle called with no title set”, some from the setup code, some from the maintenance script itself).

I haven’t found any relevant-looking recent commits so far (CommunityConfiguration mainly has l10n updates in the past few days).

Oh. The last error actually has a more useful message:

10) MediaWiki\Extension\CommunityConfiguration\Tests\Integration\ChangeWikiConfigTest::testFailsOnInvalidConfig
Output before shutdown simulation
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'Error: The property NonExistingProperty is not defined and the definition does not allow additional properties. Key: \n
+'Error: To edit this page, please solve the following task below and enter the answer in the box ([[Special:Captcha/help|more info]]):\n
 '

hCaptcha: Disable submit buttons during execution was merged 10:44 today… that’s pretty close timing, but it’s before we saw some successful merges over in T409269, so I don’t think it can be that.

Change #1202173 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/CommunityConfiguration@master] DNM: Throw bad Status for visibility

https://gerrit.wikimedia.org/r/1202173

Change #1202182 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] DNM: Empty change to test CI

https://gerrit.wikimedia.org/r/1202182

So far the error doesn’t seem to occur again in the changes linked above. Maybe it already got fixed?

Nope, it’s still happening e.g. in this build (Gerrit change).

What I have so far: in some builds – seemingly nondeterministically – the maintenance script, whose purpose is in the end to edit a wiki page (where the community configuration is stored), fails to edit the page due to a captcha check. In most tests, this shows itself as a useless “false is not true” assertion (execute() returned false); in one test (T409303#11345167), we can see a more useful message because the test expects a failure, but instead gets the captcha failure, which is unexpected.

The maintenance script uses $user = User::newSystemUser( User::MAINTENANCE_SCRIPT_USER, [ 'steal' => true ] ) as the performer, so it sounds like a bug that this system user is being asked to pass a captcha check at all.

The failure is seen in core, Wikibase, and presumably lots of other extensions; when a job fails, it always seems to fail with the same number of broken tests (10), i.e. the failure is deterministic within one job, but not deterministic across jobs. We’ve seen failures in test runs with and without gated extensions, and in PHP 8.1, 8.2 and/or 8.3, with no apparent pattern behind it.

In T409303#11345174, @Lucas_Werkmeister_WMDE wrote:

hCaptcha: Disable submit buttons during execution was merged 10:44 today… that’s pretty close timing, but it’s before we saw some successful merges over in T409269, so I don’t think it can be that.

Also that change only touched JS files. The last change to touch PHP code was SimpleCaptcha: Ensure correct instance is used on page creation, merged Monday evening; I don’t think the issue could’ve gone unnoticed for that long (all day yesterday).

Change #1202173 abandoned by Lucas Werkmeister (WMDE):

[mediawiki/extensions/CommunityConfiguration@master] DNM: Throw bad Status for visibility

Reason:

I676fa2ab88 is probably more useful

https://gerrit.wikimedia.org/r/1202173

Change #1202201 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/CommunityConfiguration@master] Print output on failure

https://gerrit.wikimedia.org/r/1202201

Okay, in this build (for the above change) we can see that the other nine tests also fail with the same error in the maintenance script:

Error: To edit this page, please solve the following task below and enter the answer in the box ([[Special:Captcha/help|more info]]):

Change #1202210 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/ConfirmEdit@master] SimpleCaptcha: Let system users skip captcha

https://gerrit.wikimedia.org/r/1202210

We recently noticed this independently within the Growth Team. Thanks Lucas for the work you started here! I'll also look.

rEABFd798dff68893: ConfirmEditHandlerTest: Re-apply test might have also been involved.

Change #1202210 merged by jenkins-bot:

[mediawiki/extensions/ConfirmEdit@master] SimpleCaptcha: Let system users skip captcha

https://gerrit.wikimedia.org/r/1202210

Change #1202282 had a related patch set uploaded (by Reedy; author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/ConfirmEdit@REL1_45] SimpleCaptcha: Let system users skip captcha

https://gerrit.wikimedia.org/r/1202282

Change #1202283 had a related patch set uploaded (by Reedy; author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/ConfirmEdit@REL1_44] SimpleCaptcha: Let system users skip captcha

https://gerrit.wikimedia.org/r/1202283

Change #1202284 had a related patch set uploaded (by Reedy; author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/ConfirmEdit@REL1_43] SimpleCaptcha: Let system users skip captcha

https://gerrit.wikimedia.org/r/1202284

Change #1202282 merged by jenkins-bot:

[mediawiki/extensions/ConfirmEdit@REL1_45] SimpleCaptcha: Let system users skip captcha

https://gerrit.wikimedia.org/r/1202282

Change #1202284 merged by jenkins-bot:

[mediawiki/extensions/ConfirmEdit@REL1_43] SimpleCaptcha: Let system users skip captcha

https://gerrit.wikimedia.org/r/1202284

Change #1202283 merged by jenkins-bot:

[mediawiki/extensions/ConfirmEdit@REL1_44] SimpleCaptcha: Let system users skip captcha

https://gerrit.wikimedia.org/r/1202283

Change #1202201 merged by jenkins-bot:

[mediawiki/extensions/CommunityConfiguration@master] Print output on failure

https://gerrit.wikimedia.org/r/1202201

Change #1202182 abandoned by Lucas Werkmeister (WMDE):

[mediawiki/extensions/Wikibase@master] DNM: Empty change to test CI

Reason:

no longer needed

https://gerrit.wikimedia.org/r/1202182

In T409303#11345859, @kostajh wrote:

rEABFd798dff68893: ConfirmEditHandlerTest: Re-apply test might have also been involved.

I guess so? (T408975 also sounds like it could have been related… some general captcha over-aggressiveness. idk)

Change #1202678 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/ConfirmEdit@master] Clean up SimpleCaptcha::canSkipCaptcha()

https://gerrit.wikimedia.org/r/1202678

Change #1202678 merged by jenkins-bot:

[mediawiki/extensions/ConfirmEdit@master] Clean up SimpleCaptcha::canSkipCaptcha()

https://gerrit.wikimedia.org/r/1202678