Page MenuHomePhabricator

Mediawiki maintenance job "generate-fancycaptcha" - fatal error when trying to copy new captchas to storage
Open, Needs TriagePublic

Description

The Mediawiki maintenance cron job "generate-fancycaptcha" generates new fancy captchas for the ConfirmEdit extension.

class mediawiki::maintenance::generatecaptcha

When running this manually for T195392 to ensure it has no issues with PHP 7.2, i noticed it generates the captchas just fine (i even downloaded and looked at 2 random PNGs), but
in the next step when it tries to copy the new captchas to storage it fails horribly with unknown error -9999

Generated 10000 captchas in 258.4 seconds
Getting a list of old captchas to delete... Done.
Copying the new captchas to storage...[Mon Jun 17 01:05:02 2019] ..
Warning: Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.34.0-wmf.8/includes/debug/logger/monolog/LegacyHandler.php on line 218
...
Fatal error: require(/srv/mediawiki/php-1.34.0-wmf.8/includes/json/FormatJson.php): File not found in /srv/mediawiki/php-1.34.0-wmf.8/includes/AutoLoader.php on line 109

Full logs are in /var/log/mediawiki/generate-fancycaptcha on mwmaint1001. This has been going on for a while apparently.

Event Timeline

Dzahn created this task.Aug 9 2019, 9:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 9 2019, 9:48 PM
Dzahn updated the task description. (Show Details)Aug 9 2019, 9:50 PM
Dzahn updated the task description. (Show Details)Aug 9 2019, 10:12 PM
bd808 added a subscriber: bd808.Aug 9 2019, 10:17 PM
/var/log/mediawiki/generate-fancycaptcha/cron.log-20190806
Generating 10000 new captchas.. Done.

Generated 10000 captchas in 258.4 seconds
Getting a list of old captchas to delete... Done.
Copying the new captchas to storage...[Mon Aug  5 01:04:23 2019] [hphp] [171225:7f25d4c2e3c0:0:000001] []
Warning: Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.34.0-wmf.16/includes/debug/logger/monolog/LegacyHandler.php on line 218
[Mon Aug  5 01:04:23 2019] [hphp] [171225:7f25d4c2e3c0:0:000002] []
Fatal error: require(/srv/mediawiki/php-1.34.0-wmf.16/includes/json/FormatJson.php): File not found in /srv/mediawiki/php-1.34.0-wmf.16/includes/AutoLoader.php on line 109

Well FormatJson.php definitely exists...

legoktm@mwmaint1002:~$ file /srv/mediawiki/php-1.34.0-wmf.16/includes/json/FormatJson.php
/srv/mediawiki/php-1.34.0-wmf.16/includes/json/FormatJson.php: PHP script, UTF-8 Unicode text
legoktm@mwmaint1002:~$ php7.2 -a
Interactive mode enabled

php > $sink=socket_create(AF_INET, SOCK_DGRAM, SOL_UDP);
php > $a=socket_sendto($sink, 'legoktm test', strlen('legoktm test'), 0, 'mwlog1001.eqiad.wmnet', '8420');
php > var_dump($a);
int(12)
php > ^Dlegoktm@mwmaint1002:~$ php -a
Welcome to HipHop Debugger!
Type "help" or "?" for a complete list of commands.

Note: no server specified, debugging local scripts only.
If you want to connect to a server, launch with "-h" or use:
  [m]achine [c]onnect <servername>

hphpd> $sink=socket_create(AF_INET, SOCK_DGRAM, SOL_UDP);
$sink=socket_create(AF_INET, SOCK_DGRAM, SOL_UDP);
hphpd> $a=socket_sendto($sink, 'legoktm test', strlen('legoktm test'), 0, 'mwlog1001.eqiad.wmnet', '8420');
$a=socket_sendto($sink, 'legoktm test', strlen('legoktm test'), 0, 'mwlog1001.eqiad.wmnet', '8420');
hphpd> var_dump($a);
var_dump($a);
int(12)

Reproducible when running it manually...

legoktm@mwmaint1002:~$ PHP=php7.2 sudo -u www-data /usr/local/bin/mwscript extensions/ConfirmEdit/maintenance/GenerateFancyCaptchas.php enwiki --wordlist=/etc/fancycaptcha/words --font=/usr/share/fonts/truetype/freefont/FreeMonoBoldOblique.ttf --blacklist=/etc/fancycaptcha/badwords --fill=10000 --oldcaptcha --delete --threads=4
Generating 10000 new captchas.. Done.

Generated 10000 captchas in 261.1 seconds
Getting a list of old captchas to delete... Done.
Copying the new captchas to storage...[Fri Aug  9 23:04:16 2019] [hphp] [207070:7f565ff223c0:0:000001] [] 
Warning: Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.34.0-wmf.17/includes/debug/logger/monolog/LegacyHandler.php on line 218
[Fri Aug  9 23:04:16 2019] [hphp] [207070:7f565ff223c0:0:000002] [] 
Fatal error: require(/srv/mediawiki/php-1.34.0-wmf.17/includes/json/FormatJson.php): File not found in /srv/mediawiki/php-1.34.0-wmf.17/includes/AutoLoader.php on line 109
Krenair added a subscriber: Krenair.EditedAug 10 2019, 5:43 PM

(@MarcoAurelio attempted to reproduce this on beta but ran into other problems - I've solved those by created a directory and a couple of files - can confirm it now falls to this same error as seen in prod)

Hi @Dzahn Could you please check if this is still happening while running the script via cron and manually? Thank you.