Page MenuHomePhabricator

Run all maintenance scripts on PHP7 or HHVM
Closed, ResolvedPublic

Description

As mentioned a couple of days ago, some jobs on terbium are still running with php5; those are the jobs that run via foreachwiki as www-data which calls foreachwikiindblist.
A list of those jobs is here: P7139

There's two options.
One: fix foreachwikindblist so it doesn't use php5 but whatever /usr/bin/php points to, which on terbium would be hhvm. Someone(s) should feel confident that the above jobs run well on hhvm before such a change is deployed.
Two: wait for terbium's replacement (mwmaint1001) to become the live server. See T192092 This would happen sometime after the tin -> deploy1001 switchover, which is currently scheduled for May 25, see T175288#4205322 In this case, someone(s) should feel confident that the above jobs run well on php7.0.

Can folks discuss and get us a plan?

Related Objects

Event Timeline

Reedy updated the task description. (Show Details)May 23 2018, 6:09 PM

Change 434754 had a related patch set uploaded (by Reedy; owner: Reedy):
[operations/puppet@production] s/php5/php/ in foreachwikiindblist

https://gerrit.wikimedia.org/r/434754

Reedy added a subscriber: Reedy.May 23 2018, 6:31 PM

Stuff that uses foreachwiki, and hence, php5

modules/mediawiki/manifests/maintenance/purge_abusefilter.pp:        command => '/usr/local/bin/foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php >/var/log/mediawiki/purge_abusefilter.log 2>&1',
modules/mediawiki/manifests/maintenance/purge_checkuser.pp:        command => '/usr/local/bin/foreachwiki extensions/CheckUser/maintenance/purgeOldData.php >/dev/null 2>&1',
modules/mediawiki/manifests/maintenance/cirrussearch.pp:        command => '/usr/local/bin/foreachwiki extensions/CirrusSearch/maintenance/saneitizeJobs.php --push --refresh-freq=7200 >> /var/log/mediawiki/cirrus-sanitize/push-jobs.log 2>&1',
modules/mediawiki/manifests/maintenance/initsitestats.pp:        command  => '/usr/local/bin/foreachwiki initSiteStats.php --update > /dev/null',
modules/mediawiki/manifests/maintenance/echo_mail_batch.pp:        command => '/usr/local/bin/foreachwikiindblist /srv/mediawiki/dblists/echo.dblist extensions/Echo/maintenance/processEchoEmailBatch.php >/dev/null',
modules/mediawiki/manifests/maintenance/pageassessments.pp:        command  => '/usr/local/bin/foreachwikiindblist /srv/mediawiki/dblists/pageassessments.dblist extensions/PageAssessments/maintenance/purgeUnusedProjects.php > /dev/null',
modules/mediawiki/manifests/maintenance/cleanup_upload_stash.pp:        command => '/usr/local/bin/foreachwiki maintenance/cleanupUploadStash.php > /dev/null',
modules/mediawiki/manifests/maintenance/purge_expired_userrights.pp:        command  => '/usr/local/bin/foreachwiki maintenance/purgeExpiredUserrights.php >/dev/null 2>&1',
modules/mediawiki/manifests/maintenance/purge_securepoll.pp:        command => '/usr/local/bin/foreachwiki extensions/SecurePoll/cli/purgePrivateVoteData.php >/dev/null 2>&1',
modules/mediawiki/manifests/maintenance/update_special_pages.pp:        command  => 'flock -n /var/lock/update-special-pages /usr/local/bin/foreachwiki updateSpecialPages.php > /var/log/mediawiki/updateSpecialPages.log 2>&1',
Reedy added a comment.EditedMay 23 2018, 7:21 PM

So, running extensions/CheckUser/maintenance/purgeOldData.php under normal foreachwiki, and a hacked version of foreachwikiindblist to use hhvm gives some interesting (but expected? results).

php5: real 7m8.995s
hhvm: real 15m43.211s

HHVM takes twice as long. I guess it's the usual issue of hhvm startup being excessive for short running scripts.

Is this enough for us to care? Or for shorter running scripts, is this manageable, knowing that it will be switched to php7 eventually?

Reedy added a comment.May 23 2018, 7:26 PM

Most of the jobs aren't time sensitive... But I think the Echo and CirrusSearch ones are more important and run more regularly?

CirrusSearch runs every 2 hours, at 10 past

Echo just runs at midnight though, so less freuqent

If you're testing anyway, how long does maintenance/rebuildLocalisationCache.php take on PHP7 (ref. T191921: mwscript rebuildLocalisationCache.php takes 40 minutes on HHVM (rather than ~5 on PHP 5))? ;-)

If you're testing anyway, how long does maintenance/rebuildLocalisationCache.php take on PHP7 (ref. T191921: mwscript rebuildLocalisationCache.php takes 40 minutes on HHVM (rather than ~5 on PHP 5))? ;-)

I've no idea. Where have we got PHP7 in prod that I can run it for our config? ;)

reedy@mwmaint1001:/tmp$ php7.0 --version
PHP 7.0.27-0+deb9u1 (cli) (built: Jan  5 2018 13:51:52) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies
    with Zend OPcache v7.0.27-0+deb9u1, Copyright (c) 1999-2017, by Zend Technologies
reedy@mwmaint1001:/tmp$ php -v
HipHop VM 3.18.6-dev (rel)
Compiler: 3.18.5+dfsg-1+wmf8+deb9u1
Repo schema: 7be0062745976798500274e76b3907ae9448a681
reedy@mwmaint1001:/tmp$ php7.0 --version
PHP 7.0.27-0+deb9u1 (cli) (built: Jan  5 2018 13:51:52) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies
    with Zend OPcache v7.0.27-0+deb9u1, Copyright (c) 1999-2017, by Zend Technologies
reedy@mwmaint1001:/tmp$ PHP=php7.0 mwscript eval.php enwiki
> echo PHP_VERSION;
7.0.27-0+deb9u1
reedy@mwmaint1001:/tmp$ time PHP=php7.0 mwscript rebuildLocalisationCache.php --wiki=enwiki --outdir=/tmp/l10nstuff
<snip>
408 languages rebuilt out of 408

real	4m35.088s

reedy@mwmaint1001:/tmp$ ls -al l10ntsuff/l10n_cache-en.cdb 
-rw-r--r-- 1 www-data www-data 3956618 May 24 12:08 l10nstuff/l10n_cache-en.cdb

File size looks good/sane

reedy@tin:/srv/mediawiki-staging$ ls -al php-1.32.0-wmf.5/cache/l10n/l10n_cache-en.cdb 
-rw-r--r-- 1 l10nupdate l10nupdate 3968828 May 24 03:11 php-1.32.0-wmf.5/cache/l10n/l10n_cache-en.cdb

So then compare with hhvm on the same host as php7

reedy@mwmaint1001:/tmp$ time PHP=hhvm mwscript rebuildLocalisationCache.php --wiki=enwiki --outdir=/tmp/l10nstuff
<snip>
408 languages rebuilt out of 408

real	16m59.381s

And then php5 on tin...

reedy@tin:/tmp$ time PHP=php5 mwscript rebuildLocalisationCache.php --wiki=enwiki --outdir=/tmp/l10nstuff
<snip>
408 languages rebuilt out of 408

real	24m13.240s

I, for one, welcome our new PHP7 overlords?

Reedy added a comment.May 24 2018, 1:41 PM

And for some more fun and games... hhvm on tin

real    40m32.809s

Change 434754 merged by Giuseppe Lavagetto:
[operations/puppet@production] s/php5/php/ in foreachwikiindblist

https://gerrit.wikimedia.org/r/434754

Joe added a subscriber: Joe.May 24 2018, 2:12 PM

As of now, any new script will only run on php7 or HHVM. We can consider this task resolved.

Joe closed this task as Resolved.May 24 2018, 2:15 PM
Vvjjkkii renamed this task from Run all jobs on PHP7 or HHVM to qfcaaaaaaa.Jul 1 2018, 1:08 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii raised the priority of this task from Normal to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: gerritbot, Aklapper.
ArielGlenn renamed this task from qfcaaaaaaa to Run all jobs on PHP7 or HHVM.Jul 1 2018, 8:24 AM
ArielGlenn closed this task as Resolved.
ArielGlenn lowered the priority of this task from High to Normal.
ArielGlenn updated the task description. (Show Details)
ArielGlenn added subscribers: gerritbot, Aklapper.
Krinkle renamed this task from Run all jobs on PHP7 or HHVM to Run all maintenance scripts on PHP7 or HHVM.Oct 5 2018, 6:26 AM