Page MenuHomePhabricator

Switch cronjobs on maintenance hosts to PHP7
Open, NormalPublic

Description

We need to move all maintenance scripts to use PHP 7.2 instead of HHVM.

This can be done by simply prepending PHP=php7.2 to all the invocations of mwscript etc in the maintenance scripts.

What needs to be done in detail:

  • Separate production maintenance scripts from wikitech ones**
  • Pick a few scripts to migrate first, notify the owners, migrate
  • Migrate the wikidata dispatcher
  • Migrate all other scripts / manually run them
  • Set php7.2 as the preferred debian alternative
  • Revert all the modifications to the scripts above as we don't need to force
  • Some cronjobs are shared with wikitech, which has not been switched to PHP7. To avoid blocking this task we can keep production and wikitech maint jobs separate T222900

Related to: T219135:


Progress # (of 25)%
2496%
checked & convertedpuppet class namewhat does it docommentlogging ?
cache_warmuprun APC/Memcached warmup after resetting cachesnot a cron, just a package and filen/a
initsitestatsRe-initialise or update the site statistics tableuses 'foreachwiki', uses foreachwikiindblist, bash, hardcodes RUNNER=php/dev/null
purge_expired_userrightsRemove expired userrights from user_groups table and move them to former_user_groupsuses foreachwiki/dev/null
translationnotificationssend email notification to translatorsuses mwscript, allows overriding of PHP, ends emailwith DigestEmailer.php to a lot of users https://gerrit.wikimedia.org/r/c/operations/puppet/+/528606/var/log
uploadsconfigures wget for server-side uploads of large filesnot a cron, just a config file for wgetn/a
cirrussearchbuild completion indices for search, sanitizes cirrus jobsuses expanddblist, (PHP) but pipes it into mwscript https://gerrit.wikimedia.org/r/c/operations/puppet/+/529417 , VERY LONG running/var/log/mediawiki
pageassessmentsremove unused projects from the page_assessments_projects tableuses foreachwikiindblist/dev/null
purge_old_cx_draftspurge content translation drafts older than 455 daysuses foreachwikiindblist/var/log/mediawiki/
update_flaggedrev_statsupdates the flagged revs statistics tableuses mwscriptwikiset (bash), starts mwscript/dev/null
wikidatastarts a dispatcher instance every 3 minutes, wikibase-repo pruning and othersCheck https://grafana.wikimedia.org/d/000000156/wikidata-dispatch?refresh=1m&orgId=1 and https://grafana.wikimedia.org/d/000000239/wikidata-dispatch-script?refresh=1m&orgId=1/var/log/wikidata
cleanup_upload_stashRemove old or broken uploads from temporary uploaded file storage, clean up associated database recordsuses foreachwiki , problems on commons (T230179)/var/log/mediawiki/
pagetriageupdates expired page metadatauses mwscript, https://gerrit.wikimedia.org/r/c/operations/puppet/+/528609 . ask Ariel/var/log/mediawiki
purge_securepollPurge private data (IP, XFF, UA) from SecurePoll Votesuses foreachwiki/dev/null
updatequerypages6 separate cron jobs!. (Lonelypages, Mostcategories, Mostlinkedcategories, Mostlinkedtemplates, Uncategorizedpages, Wantedtemplates) calling updateSpecialPages.php on all clusters, one job for just en, one for all othersuses updatequerypages::cronjob, uses mwscriptwikiset/var/log/mediawiki
db_lag_statsdump MediaWiki DB lag times into statsd T149210uses mwscript/dev/null
jobqueue_statsreports the amount of lag for MediaWiki-pooled DBs to StatsDhttps://gerrit.wikimedia.org/r/c/operations/puppet/+/528287/dev/null
parsercachepurgingRemove old objects from the parser cacheuses mwscript,only works if parsercache is a mysql db , very slow!! see T230240/dev/null
readinglistspurging unneeded DB rows for extension ReadingListshttps://gerrit.wikimedia.org/r/c/operations/puppet/+/528282/var/log/mediawiki
echo_mail_batchprocesses email digest / send notifications to usersuses foreachwikiindblist/dev/null
purge_abusefilterPurge old IP Address data from AbuseFilter logsuses foreachwiki/dev/null
refreshlinksrefresh link tablesuses refreshlinks::cronjob, uses mwscriptwikiset/var/log/mediawiki
update_special_pagesupdates cached special pages "Run this script periodically if you have miser mode enabled."uses flock, uses foreachwiki updateSpecialPages.php/var/log/mediawiki
generatecaptchaGenerate fancy captchas using a python script and copy them into storageuses mwscript GenerateFancyCaptchas from ConfirmEdit extension , BROKEN -> T230245/var/log/mediawiki
purge_checkuserPurge expired rows in CheckUser and RecentChangesuses foreachwiki purgeOldData.php - did https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/528730/ but foreachwiki forces RUNNER=php apparently/dev/null
updatetranslationstats"Update stats" "https://meta.wikimedia.org/wiki/Meta:Babylon/Translation_stats as User FuzzyBotruns /usr/local/bin/characterEditStatsTranslate, uses mwscript. tested at https://meta.wikimedia.org/w/index.php?title=Meta:Babylon/Translation_stats&action=history see the edit has the php7 tag/dev/null

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
jijiki updated the task description. (Show Details)Apr 3 2019, 6:50 PM
jijiki updated the task description. (Show Details)Apr 3 2019, 6:55 PM
jijiki moved this task from Backlog/Radar to In Progress on the User-jijiki board.Apr 4 2019, 9:14 PM
jijiki updated the task description. (Show Details)May 15 2019, 7:13 PM
jijiki moved this task from Backlog to Next up on the serviceops board.Jul 5 2019, 9:28 AM
jijiki moved this task from Next up to Doing on the serviceops board.Jul 11 2019, 7:38 AM
jijiki moved this task from Doing to Next up on the serviceops board.Jul 17 2019, 7:59 AM

Change 528282 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mediawiki:maintenance: switch readinglists cron to PHP 7.2

https://gerrit.wikimedia.org/r/528282

Change 528282 merged by Dzahn:
[operations/puppet@production] mediawiki:maintenance: switch readinglists cron to PHP 7.2

https://gerrit.wikimedia.org/r/528282

Change 528287 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mediawiki:maintenance: remove non-working jobqueue_stats cron

https://gerrit.wikimedia.org/r/528287

Dzahn updated the task description. (Show Details)Mon, Aug 5, 11:31 PM

Change 528287 merged by Dzahn:
[operations/puppet@production] mediawiki:maintenance: remove non-working jobqueue_stats cron

https://gerrit.wikimedia.org/r/528287

Mentioned in SAL (#wikimedia-operations) [2019-08-05T23:34:39Z] <mutante> mwmaint1002 - remove getJobQueueLengths.php from www-data's crontab (T195392)

Change 528296 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mediawiki:maintenance: switch translationnotifications to PHP 7.2

https://gerrit.wikimedia.org/r/528296

jijiki moved this task from Next up to Doing on the serviceops board.Tue, Aug 6, 9:56 AM

Change 425027 had a related patch set uploaded (by Effie Mouzeli; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::hhvm: default php to php7 on stretch

https://gerrit.wikimedia.org/r/425027

Reedy updated the task description. (Show Details)Tue, Aug 6, 11:59 PM

Change 528296 merged by Dzahn:
[operations/puppet@production] mediawiki:maintenance: switch translationnotifications to PHP 7.2

https://gerrit.wikimedia.org/r/528296

Dzahn updated the task description. (Show Details)Wed, Aug 7, 1:14 AM
Dzahn updated the task description. (Show Details)Wed, Aug 7, 1:17 AM
Dzahn updated the task description. (Show Details)Wed, Aug 7, 1:27 AM

Change 528609 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mediawiki:maintenance: switch pagetriage cron to PHP 7.2

https://gerrit.wikimedia.org/r/528609

Dzahn updated the task description. (Show Details)Wed, Aug 7, 1:41 AM
Dzahn updated the task description. (Show Details)Wed, Aug 7, 1:55 AM
Dzahn updated the task description. (Show Details)Wed, Aug 7, 2:08 AM

Change 528612 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mediawiki:maintenance: switch db_lag_stats to PHP 7.2

https://gerrit.wikimedia.org/r/528612

Dzahn updated the task description. (Show Details)Wed, Aug 7, 2:16 AM

Change 528612 merged by Dzahn:
[operations/puppet@production] mediawiki:maintenance: switch db_lag_stats to PHP 7.2

https://gerrit.wikimedia.org/r/528612

Change 528613 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mw::maintenance: switch foreachwikiindblist to PHP 7.2

https://gerrit.wikimedia.org/r/528613

Change 528730 had a related patch set uploaded (by MarcoAurelio; owner: MarcoAurelio):
[operations/puppet@production] [WIP] mediawiki:maintenance::purge_checkuser.pp: Switch to PHP 7.2

https://gerrit.wikimedia.org/r/528730

jijiki updated the task description. (Show Details)Wed, Aug 7, 3:17 PM

Change 528875 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] mediawiki:maintenance: switch wikidata to PHP 7.2

https://gerrit.wikimedia.org/r/528875

Change 528875 merged by Effie Mouzeli:
[operations/puppet@production] mediawiki:maintenance: switch wikidata to PHP 7.2

https://gerrit.wikimedia.org/r/528875

Change 528889 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] mediawiki:maintenance: switch wikidata to PHP 7.2 (prod)

https://gerrit.wikimedia.org/r/528889

Change 528889 merged by Effie Mouzeli:
[operations/puppet@production] mediawiki:maintenance: switch wikidata to PHP 7.2 (prod)

https://gerrit.wikimedia.org/r/528889

jijiki updated the task description. (Show Details)Wed, Aug 7, 4:58 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-07T22:48:12Z] <mutante> mwmaint1002 - manually running the purgeOldData cron command to verify it with PHP 7.2 for 528730 (T195392)

Change 528730 merged by Dzahn:
[operations/puppet@production] mediawiki::maintenance::purge_checkuser.pp: Switch to PHP 7.2

https://gerrit.wikimedia.org/r/528730

Dzahn updated the task description. (Show Details)Thu, Aug 8, 8:51 PM

Change 528613 merged by Dzahn:
[operations/puppet@production] scap/mw-maint: switch foreachwikiindblist to PHP 7.2

https://gerrit.wikimedia.org/r/528613

Mentioned in SAL (#wikimedia-operations) [2019-08-08T21:40:21Z] <mutante> mwmaint1002 - manually running (weekly) echo_mail cron job (user notifications) to confirm it works after switching foreachwikiindblist to use php7.2 (T195392)

Dzahn updated the task description. (Show Details)Thu, Aug 8, 9:54 PM

Change 529190 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mediawiki:maintenance: switch translationnotifications to PHP 7.2

https://gerrit.wikimedia.org/r/529190

Change 529190 merged by Dzahn:
[operations/puppet@production] mediawiki:maintenance: switch translationnotifications to PHP 7.2

https://gerrit.wikimedia.org/r/529190

Dzahn updated the task description. (Show Details)Thu, Aug 8, 11:44 PM
Dzahn updated the task description. (Show Details)Thu, Aug 8, 11:54 PM
Dzahn updated the task description. (Show Details)Fri, Aug 9, 2:42 AM
jijiki added a comment.EditedFri, Aug 9, 3:17 PM

@Dzahn I think we should continue by switching all jobs on Monday and see what gives. We followed the same strategy with async jobs, i.e. we migrated a few jobs that stood out along with a few high traffic ones, and eventually we switched what was left with one go. What do you think? Thank you very much for the help!

jijiki awarded a token.Fri, Aug 9, 3:17 PM

Change 528609 merged by Dzahn:
[operations/puppet@production] mediawiki:maintenance: switch pagetriage cron to PHP 7.2

https://gerrit.wikimedia.org/r/528609

Dzahn updated the task description. (Show Details)Fri, Aug 9, 6:03 PM
Dzahn updated the task description. (Show Details)Fri, Aug 9, 6:25 PM

Change 529417 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mw-maintenance/cirrussearch: switch cron jobs to PHP 7.2

https://gerrit.wikimedia.org/r/529417

Change 529417 merged by Dzahn:
[operations/puppet@production] mw-maintenance/cirrussearch: switch cron jobs to PHP 7.2

https://gerrit.wikimedia.org/r/529417

Dzahn updated the task description. (Show Details)Fri, Aug 9, 9:21 PM

Change 529433 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mw-maintenance: switch generatecaptcha cron to PHP 7.2

https://gerrit.wikimedia.org/r/529433

Dzahn updated the task description. (Show Details)Fri, Aug 9, 9:53 PM

Change 529433 merged by Dzahn:
[operations/puppet@production] mw-maintenance: switch generatecaptcha cron to PHP 7.2

https://gerrit.wikimedia.org/r/529433

Change 529437 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mw-maintenance: switch characterEditStatsTranslate to PHP 7.2

https://gerrit.wikimedia.org/r/529437

Change 529437 merged by Dzahn:
[operations/puppet@production] mw-maintenance: switch characterEditStatsTranslate to PHP 7.2

https://gerrit.wikimedia.org/r/529437

Dzahn updated the task description. (Show Details)Fri, Aug 9, 10:25 PM
Dzahn removed a subscriber: Dzahn.
Dzahn updated the task description. (Show Details)Fri, Aug 9, 10:27 PM

Change 529459 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] scap: set mwscriptwikiset to use PHP7.2

https://gerrit.wikimedia.org/r/529459

Change 529459 merged by Dzahn:
[operations/puppet@production] scap: set mwscriptwikiset to use PHP7.2

https://gerrit.wikimedia.org/r/529459

Mentioned in SAL (#wikimedia-operations) [2019-08-10T01:49:11Z] <mutante> mwmaint - running (1 of 8, the one for en) refreshLinks maintenance cron manually to verify it works after switching mwscriptwikiset to PHP7.2 (T195392)

Dzahn updated the task description. (Show Details)Sat, Aug 10, 2:11 AM
Peachey88 updated the task description. (Show Details)Sat, Aug 10, 2:20 AM
Dzahn added a comment.Sat, Aug 10, 2:21 AM

@jijiki This is good to go now from my point of view.


1:49 mutante: mwmaint - running (1 of 8, the one for en) refreshLinks maintenance cron manually to verify it works after switching mwscriptwikiset to PHP7.2 (T195392)
00:52 mutante: mwmaint - running update_flaggedrevs_stats - updates the flagged revs statistics table on each wiki
00:47 mutante: mwmaint - running cirrus sanitize jobs maintenance cron
21:28 mutante: mwmaint - generating new captchas for ConfirmEdit extension by running generatecaptcha maintenance cron command
20:55 mutante: mwmaint - running update_special_pages maintenance cron manually
19:37 mutante: mwmaint - running cirrussearch maintenance jobs manually (completion indices, sanitize cirrus jobs)
17:54 mutante: mwmaint - running initsitestats maintenance job - initializes or updates statistics table on all wikis
17:19 mutante: mwmaint - running purgeParserCache maintenance cron manually with PHP 7.2 - ..slowly
16:52 mutante: mwmaint - manually running updatePageTriageQueue maintenance cron with PHP 7.2
02:30 mutante: mwmaint1002 - manually running cleanup_upload_stash maintenance cron to confirm no issues with PHP 7.2 in maintenance/cleanupUploadStash.php
02:24 mutante: mwmaint1002 - manually running purge_expired_userrights maintenance cron to confirm no issues with PHP 7.2 in maintenance/purgeExpiredUserrights.php
02:17 mutante: mwmaint1002 - manually running purge_abusefilter maintenance cron
23:48 mutante: mwmaint1002 - manually running purge_securepoll maintenance script
23:42 mutante: mwmaint1002 - manually running TranslatioNNotifications DigestEmailer maintenance cron
21:50 mutante: mwmaint1002 - manually running purgeUnusedProjects with PageAssessments extension to confirm no issues after switch to PHP7.2
21:40 mutante: mwmaint1002 - manually running (weekly) echo_mail cron job (user notifications) to confirm it works after switching foreachwikiindblist to use php7.2 (T195392)
23:34 mutante: mwmaint1002 - remove getJobQueueLengths.php from www-data's crontab (T195392)
22:48 mutante: mwmaint1002 - manually running the purgeOldData cron command to verify it with PHP 7.2 for 528730 (T195392)


new related tickets:

T230245 Mediawiki maintenance job "generate-fancycaptcha" - fatal error when trying to copy new captchas to storage
T230110 Stop forcing RUNNER=php for foreachwiki/foreachwikiindblist
T230240 purgeParserCache too slow to run once a day
T230179 clean_upload_stash maintenance cron fails to delete files on commons wiki (backend-fail-delete)


scap: set mwscriptwikiset to use PHP7.2 https://gerrit.wikimedia.org/r/c/operations/puppet/+/529459
mw-maintenance: switch characterEditStatsTranslate to PHP 7.2 - https://gerrit.wikimedia.org/r/c/operations/puppet/+/529437
mw-maintenance: switch generatecaptcha cron to PHP 7.2 - https://gerrit.wikimedia.org/r/c/operations/puppet/+/529433
mw-maintenance/cirrussearch: switch cron jobs to PHP 7.2 - https://gerrit.wikimedia.org/r/c/operations/puppet/+/529417
mediawiki:maintenance: switch pagetriage cron to PHP 7.2 - https://gerrit.wikimedia.org/r/c/operations/puppet/+/528609
mediawiki:maintenance: switch translationnotifications to PHP 7.2 - https://gerrit.wikimedia.org/r/c/operations/puppet/+/529190
scap/mw-maint: switch foreachwikiindblist to PHP 7.2 - https://gerrit.wikimedia.org/r/c/operations/puppet/+/528613
mediawiki:maintenance: switch db_lag_stats to PHP 7.2 - https://gerrit.wikimedia.org/r/c/operations/puppet/+/528612
mediawiki:maintenance: switch translationnotifications to PHP 7.2 - https://gerrit.wikimedia.org/r/c/operations/puppet/+/528296
mediawiki:maintenance: remove non-working jobqueue_stats cron - https://gerrit.wikimedia.org/r/c/operations/puppet/+/528287
mediawiki:maintenance: switch readinglists cron to PHP 7.2 - https://gerrit.wikimedia.org/r/c/operations/puppet/+/528282

Change 425027 merged by Effie Mouzeli:
[operations/puppet@production] profile::mediawiki default php to php7

https://gerrit.wikimedia.org/r/425027

jijiki added a subscriber: Dzahn.Mon, Aug 12, 3:25 PM

@Dzahn Thank you! We'll see how things are since we now have merged https://gerrit.wikimedia.org/r/425027, and continue with moving any transitional changes we did.

jijiki updated the task description. (Show Details)Mon, Aug 12, 3:26 PM
jijiki moved this task from Doing to Next up on the serviceops board.Tue, Aug 13, 8:32 AM

Change 529921 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] mw-maintenance and scap: Revert changes for PHP7 transition

https://gerrit.wikimedia.org/r/529921