Page MenuHomePhabricator

beta-scap-sync-world broken after https://gerrit.wikimedia.org/r/c/1115497 was merged
Closed, ResolvedPublic

Description

After https://gerrit.wikimedia.org/r/c/1115497 was merged today
beta-scap-sync-world started failing with the following output:

06:55:04 14:55:04 Started scap sync-world: beta-scap-sync-world (build #194543)
06:55:06 14:55:06 Started cache_git_info
06:55:35 14:55:35 Finished cache_git_info (duration: 00m 28s)
06:55:35 14:55:35 Started l10n-update
06:55:39 14:55:39 Updating ExtensionMessages-master.php
06:55:45 14:55:45 Finished l10n-update (duration: 00m 09s)
06:55:45 mergeMessageFileList.php generated PHP notices/warnings:
06:55:45 Warning: socket_sendto(): Host lookup failed [-10002]: Host name lookup failure in /srv/mediawiki-staging/php-master/includes/debug/logger/monolog/LegacyHandler.php on line 234
06:55:45 
06:55:45 Build step 'Execute shell' marked build as failure

I disabled the job to avoid noise while the issue is being discussed. A revert commit has been proposed at https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/1121043

Event Timeline

FWIW, more discussion about that CA change is at T386836 (though it’s not clear to me how that could cause a host name lookup failure o_O)

A revert commit has been proposed at https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/1121043

I +2ed that revert by now, but it’ll take quite some time for gate-and-submit to reach it :|

image.png (1×719 px, 224 KB)

FWIW, more discussion about that CA change is at T386836 (though it’s not clear to me how that could cause a host name lookup failure o_O)

I added some in-place debugging to LegacyHandler.php:

public function __construct(
        $stream,
        $useLegacyFilter = false,
        $level = Logger::DEBUG,
        $bubble = true
) {
        parent::__construct( $level, $bubble );
        trigger_error("URI is {$stream}\n", E_USER_WARNING);
        $this->uri = $stream;
        $this->useLegacyFilter = $useLegacyFilter;
}

This is the result:

mergeMessageFileList.php generated PHP notices/warnings:
Warning: URI is udp://deployment-mwlog02.deployment-prep.eqiad1.wikimedia.cloud:8420/{channel}
...

deployment-mwlog02.deployment-prep.eqiad1.wikimedia.cloud is a resolvable hostname so it's unclear what's going on.

This is the message that is being logged at the time of the socket_sendto() call:

MessageCache 2025-02-19 18:28:06.710940 [608821d4d8ba83c01bd9b82d] deployment-deploy04 aawiki 1.44.0-alpha MessageCache DEBUG: MessageCache using store Wikimedia\ObjectCache\EmptyBagOStuff {"class":"Wikimedia\\ObjectCache\\EmptyBagOStuff"}

OK. I remember now. Scap runs mergeMessageFileList.php in a container that does not have network access enabled, so anything that generates a log message will trigger this type of situation.