Page MenuHomePhabricator
Create Task
Maniphest T159607

Investigate why --delete seemingly deleted all but 1 captcha file when run in cronjob
Closed, ResolvedPublic
Actions

Description

In T159581#3072702, @Tgr wrote:

Confirmed that there is only one captcha file in swift:

tgr@terbium:~$ mwscript eval.php --wiki=metawiki
> $backend = FileBackendGroup::singleton()->get( $wgCaptchaFileBackend );

> $baseDir = FileBackendGroup::singleton()->get( $wgCaptchaFileBackend )->getRootStoragePath() . '/captcha-render';

> $fl = $backend->getFileList( [ 'dir' => $baseDir ] )

> var_dump(iterator_to_array($fl))
array(1) {
  [0]=>
  string(41) "9/9/9/image_67350e1b_9996cc25d9b58fc9.png"
}

The captcha generation script does not update/clear the cache used in FancyCaptcha::pickImage*, which it probably should, but that shouldn't be able to cause anything remotely similar.

Details

SubjectRepoBranchLines +/-
Make sure Swift store operations close the source file handlemediawiki/corewmf/1.29.0-wmf.16+2 -0
Make sure Swift store operations close the source file handlemediawiki/coremaster+2 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Reedy created this task.Mar 4 2017, 4:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 4 2017, 4:48 PM
Reedy added a subtask: T159581: The same CAPTCHA image is always used across platforms and refresh.Mar 4 2017, 4:48 PM
Reedy closed subtask T159581: The same CAPTCHA image is always used across platforms and refresh as Resolved.Mar 4 2017, 4:52 PM
Reedy added a comment.Mar 4 2017, 5:21 PM

So, running a cronjob to create upto 10000... No --delete parameter, I see

Warning: socket_sendto(): Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.29.0-wmf.14/includes/debug/logger/monolog/LegacyHandler.php on line 220
Warning: socket_sendto(): Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.29.0-wmf.14/includes/debug/logger/monolog/LegacyHandler.php on line 220
Warning: socket_sendto(): Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.29.0-wmf.14/includes/debug/logger/monolog/LegacyHandler.php on line 220
Warning: socket_sendto(): Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.29.0-wmf.14/includes/debug/logger/monolog/LegacyHandler.php on line 220
Warning: socket_sendto(): Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.29.0-wmf.14/includes/debug/logger/monolog/LegacyHandler.php on line 220
Warning: md5_file(/tmp/mw-fancycaptcha-1488645777-a1b4a7/a/6/1/image_0ec90eab_a614abaa81a86580.png): failed to open stream: Too many open files in /srv/mediawiki/php-1.29.0-wmf.14/includes/libs/filebackend/SwiftFileBackend.php on line 331

Do we get the feeling something happened with the copying in in batches, but the delete apparently worked fine? Not sure why it'd end up with 1 file still, but the above... At least for the last line

I filed T159610 to stop this information from cronjob runs going into the black hole that is /dev/null

Reedy mentioned this in T159613: Host lookup failed [-9999]: Unknown error -9999.Mar 4 2017, 5:23 PM
Reedy added a subscriber: aaron.

Ping @aaron for the "Too many open files in /srv/mediawiki/php-1.29.0-wmf.14/includes/libs/filebackend/SwiftFileBackend.php on line 331" - do we know how many is too many? And why the layer let the script do that?

Reedy added a comment.Mar 4 2017, 5:24 PM
In T159607#3073192, @Reedy wrote:

Do we get the feeling something happened with the copying in in batches, but the delete apparently worked fine? Not sure why it'd end up with 1 file still, but the above... At least for the last line

Or the 1 was some sort of fluke, a file that couldn't be deleted for some other reason?

Tgr reopened subtask T159581: The same CAPTCHA image is always used across platforms and refresh as Open.Mar 5 2017, 8:48 PM
Reedy added a project: SRE-swift-storage.Mar 5 2017, 10:28 PM
• MZMcBride subscribed.Mar 5 2017, 10:32 PM
Reedy added a subtask: T159613: Host lookup failed [-9999]: Unknown error -9999.Mar 5 2017, 10:35 PM
Platonides subscribed.Mar 5 2017, 10:50 PM
In T159607#3073207, @Reedy wrote:

do we know how many is too many?

That looks like reaching the max number of open files per process (ulimit -n) or per system. Maybe there's a leak of open descriptors in that script?

Reedy added a comment.Mar 5 2017, 11:04 PM
In T159607#3074238, @Platonides wrote:
In T159607#3073207, @Reedy wrote:

do we know how many is too many?

That looks like reaching the max number of open files per process (ulimit -n) or per system. Maybe there's a leak of open descriptors in that script?

Maybe. It worked before... rECOE7de460c289041e6fd729d0209c08077ab95f1b27 changed it from being a tight loop doing each copy/move operation, to passing a big array of paths on disk, with the hope it'd be more performant

Reedy created subtask T159830: Sanity check global-multiwrite logs for ConfirmEdit usage.Mar 7 2017, 1:30 PM
Reedy added a subtask: T159605: GenerateFancyCaptchas should purge relevant caches upon running.Mar 7 2017, 1:35 PM
Reedy added a subtask: T159606: GenerateFancyCaptchas output a little spurious.
Reedy added a subtask: T159831: GenerateFancyCaptchas outputs Array for errors.Mar 7 2017, 1:41 PM
Reedy closed subtask T159831: GenerateFancyCaptchas outputs Array for errors as Resolved.Mar 7 2017, 6:26 PM
aaron added a comment.Mar 9 2017, 2:46 AM

Batching itself is fine and well within reasonable ulimits (in fact FileBackend has a hard batch operation limit anyway, on top of the concurrency limit). Any issue would either be leaks or something else.

gerritbot subscribed.Mar 9 2017, 2:51 AM

Change 341962 had a related patch set uploaded (by Aaron Schulz):
[mediawiki/core] Make sure Swift store operations close the source file handle

https://gerrit.wikimedia.org/r/341962

gerritbot added a project: Patch-For-Review.Mar 9 2017, 2:51 AM
Reedy closed subtask T159581: The same CAPTCHA image is always used across platforms and refresh as Resolved.Mar 9 2017, 2:33 PM
Reedy added a comment.Mar 9 2017, 6:41 PM
reedy@terbium:~$ /usr/local/bin/mwscript extensions/ConfirmEdit/maintenance/GenerateFancyCaptchas.php mediawikiwiki --wordlist=/etc/fancycaptcha/words --font=/usr/share/fonts/truetype/freefont/FreeMonoBoldOblique.ttf --blacklist=/etc/fancycaptcha/badwords --fill=10000 --oldcaptcha
Current number of captchas is 1000.
Generating 9000 new captchas.. Done.

Generated 9000 captchas in 1459.5 seconds
Copying the new captchas to storage...Warning: md5_file(/tmp/mw-fancycaptcha-1489075661-03e546/a/8/e/image_733e516f_a8edc35843fb98e8.png): failed to open stream: Too many open files in /srv/mediawiki/php-1.29.0-wmf.15/includes/libs/filebackend/SwiftFileBackend.php on line 331
Warning: socket_sendto(): Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.29.0-wmf.15/includes/debug/logger/monolog/LegacyHandler.php on line 220
Warning: socket_sendto(): Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.29.0-wmf.15/includes/debug/logger/monolog/LegacyHandler.php on line 220
<snip>
Warning: socket_sendto(): Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.29.0-wmf.15/includes/debug/logger/monolog/LegacyHandler.php on line 220
Warning: socket_sendto(): Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.29.0-wmf.15/includes/debug/logger/monolog/LegacyHandler.php on line 220
Warning: socket_sendto(): Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.29.0-wmf.15/includes/debug/logger/monolog/LegacyHandler.php on line 220
Warning: socket_sendto(): Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.29.0-wmf.15/includes/debug/logger/monolog/LegacyHandler.php on line 220
Warning: socket_sendto(): Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.29.0-wmf.15/includes/debug/logger/monolog/LegacyHandler.php on line 220
Warning: md5_file(/tmp/mw-fancycaptcha-1489075661-03e546/a/8/e/image_733e516f_a8edc35843fb98e8.png): failed to open stream: Too many open files in /srv/mediawiki/php-1.29.0-wmf.15/includes/libs/filebackend/SwiftFileBackend.php on line 331
Warning: socket_sendto(): Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.29.0-wmf.15/includes/debug/logger/monolog/LegacyHandler.php on line 220
Warning: socket_sendto(): Host lookup failed [-9999]: Unknown error -9999 in /srv/mediawiki/php-1.29.0-wmf.15/includes/debug/logger/monolog/LegacyHandler.php on line 220
Errored.
Warning: socket_sendto(): Host lookup failed [-10002]: Host name lookup failure in /srv/mediawiki/php-1.29.0-wmf.15/includes/debug/logger/monolog/LegacyHandler.php on line 220
Warning: implode(): Invalid arguments passed in /srv/mediawiki/php-1.29.0-wmf.15/extensions/ConfirmEdit/maintenance/GenerateFancyCaptchas.php on line 197
Removing temporary files... Done.

Whole captchas generation process took 2303.2 seconds
reedy@terbium:~$
Reedy closed subtask T159606: GenerateFancyCaptchas output a little spurious as Resolved.Mar 9 2017, 6:42 PM
gerritbot added a comment.Mar 14 2017, 8:49 PM

Change 341962 merged by jenkins-bot:
[mediawiki/core] Make sure Swift store operations close the source file handle

https://gerrit.wikimedia.org/r/341962

ReleaseTaggerBot added projects: MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)), MW-1.29-release-notes.Mar 14 2017, 9:00 PM
gerritbot added a comment.Mar 14 2017, 9:07 PM

Change 342703 had a related patch set uploaded (by Reedy; owner: Aaron Schulz):
[mediawiki/core] Make sure Swift store operations close the source file handle

https://gerrit.wikimedia.org/r/342703

gerritbot added a comment.Mar 15 2017, 6:24 PM

Change 342703 merged by jenkins-bot:
[mediawiki/core] Make sure Swift store operations close the source file handle

https://gerrit.wikimedia.org/r/342703

Stashbot subscribed.Mar 15 2017, 6:32 PM

Mentioned in SAL (#wikimedia-operations) [2017-03-15T18:32:37Z] <legoktm@tin> Synchronized php-1.29.0-wmf.16/includes/libs/filebackend/SwiftFileBackend.php: Make sure Swift store operations close the source file handle - T159607 (duration: 00m 44s)

ReleaseTaggerBot edited projects, added MW-1.29-release (WMF-deploy-2017-03-14_(1.29.0-wmf.16)); removed MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)).Mar 15 2017, 7:00 PM
Reedy removed a project: Patch-For-Review.Mar 15 2017, 7:05 PM
Krinkle removed projects: MW-1.29-release (WMF-deploy-2017-03-14_(1.29.0-wmf.16)), MW-1.29-release-notes.May 25 2017, 2:01 PM
Reedy closed this task as Resolved.Jun 12 2017, 5:20 PM

Looks like the various fixes are meaning this doesn't break all over the place now. Yay

fgiunchedi closed subtask T159830: Sanity check global-multiwrite logs for ConfirmEdit usage as Resolved.Apr 14 2020, 9:33 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits