Page MenuHomePhabricator
Create Task
Maniphest T434979

Periodic script failure due to "Fatal error: no version entry for `Please`"
Closed, ResolvedPublic
Actions

Description

[urbanecm@deploy1003 ~]$ kubectl logs -f job/growthexperiments-listtaskcounts-29780051
Defaulted container "mediawiki-main-app" out of: mediawiki-main-app, mediawiki-main-tls-proxy, mediawiki-main-rsyslog
extensions/GrowthExperiments/maintenance/listTaskCounts.php: Start run
extensions/GrowthExperiments/maintenance/listTaskCounts.php: Running on growthexperiments.dblist
extensions/GrowthExperiments/maintenance/listTaskCounts.php: Running on growthexperiments.dblist
Please no version entry for `Please`.
Please 
Please Fatal error: no version entry for `Please`.
Please  in /srv/mediawiki/multiversion/MWMultiVersion.php on line 700
[urbanecm@deploy1003 ~]$ kubectl logs -f job/growthexperiments-deleteoldsurveys-29779395
Defaulted container "mediawiki-main-app" out of: mediawiki-main-app, mediawiki-main-tls-proxy, mediawiki-main-rsyslog
extensions/GrowthExperiments/maintenance/deleteOldSurveys.php: Start run
extensions/GrowthExperiments/maintenance/deleteOldSurveys.php: Running on growthexperiments.dblist
extensions/GrowthExperiments/maintenance/deleteOldSurveys.php: Running on growthexperiments.dblist
Please no version entry for `Please`.
Please 
Please Fatal error: no version entry for `Please`.
Please  in /srv/mediawiki/multiversion/MWMultiVersion.php on line 700
[urbanecm@deploy1003 ~]$ kubectl logs -f job/growthexperiments-purgeexpiredmentorstatus-29779725
Defaulted container "mediawiki-main-app" out of: mediawiki-main-app, mediawiki-main-tls-proxy, mediawiki-main-rsyslog
extensions/GrowthExperiments/maintenance/purgeExpiredMentorStatus.php: Start run
extensions/GrowthExperiments/maintenance/purgeExpiredMentorStatus.php: Running on growthexperiments.dblist
extensions/GrowthExperiments/maintenance/purgeExpiredMentorStatus.php: Running on growthexperiments.dblist
Please no version entry for `Please`.
Please 
Please Fatal error: no version entry for `Please`.
Please  in /srv/mediawiki/multiversion/MWMultiVersion.php on line 700
[urbanecm@deploy1003 ~]$ kubectl logs -f job/growthexperiments-refreshpraiseworthymentees-29779695
Defaulted container "mediawiki-main-app" out of: mediawiki-main-app, mediawiki-main-tls-proxy, mediawiki-main-rsyslog
extensions/GrowthExperiments/maintenance/refreshPraiseworthyMentees.php: Start run
extensions/GrowthExperiments/maintenance/refreshPraiseworthyMentees.php: Running on growthexperiments.dblist
extensions/GrowthExperiments/maintenance/refreshPraiseworthyMentees.php: Running on growthexperiments.dblist
Please no version entry for `Please`.
Please 
Please Fatal error: no version entry for `Please`.
Please  in /srv/mediawiki/multiversion/MWMultiVersion.php on line 700
[urbanecm@deploy1003 ~]$ kubectl logs -f job/growthexperiments-updatementeedata-s1-29779935
Defaulted container "mediawiki-main-app" out of: mediawiki-main-app, mediawiki-main-tls-proxy, mediawiki-main-rsyslog
extensions/GrowthExperiments/maintenance/updateMenteeData.php: Start run
extensions/GrowthExperiments/maintenance/updateMenteeData.php: Running on growthexperiments & s1
extensions/GrowthExperiments/maintenance/updateMenteeData.php: Running on growthexperiments & s1
Please no version entry for `Please`.
Please 
Please Fatal error: no version entry for `Please`.
Please  in /srv/mediawiki/multiversion/MWMultiVersion.php on line 700
[urbanecm@deploy1003 ~]$

and more. This is not the expected state for maintenance scripts.

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
expanddblist: Check for passage of DB list correctlyrepos/releng/release!260urbanecmT434979main
Customize query in GitLab

Related Objects

Event Timeline

Urbanecm_WMF created this task.Sat, Aug 15, 2:55 PM
Restricted Application added a project: Growth-Team. · View Herald TranscriptSat, Aug 15, 2:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Urbanecm_WMF triaged this task as Unbreak Now! priority.Sat, Aug 15, 2:56 PM

This puts all of Growth-Team scripts out of order (and likely others too, I did not check).

taavi added subscribers: dancy, Reedy, taavi.Sat, Aug 15, 3:10 PM

Regression from https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/259 / T434567: expanddblist without arguments doesn't fail cleanly it seems. $$argv[1] should probably be just $argv[1]. cc @Reedy @dancy

Urbanecm_WMF added a comment.Sat, Aug 15, 3:10 PM

Caused by https://gitlab.wikimedia.org/repos/releng/release/-/commit/be27c6af50e131efcfbf4bde1f8db5a71f2bd764 authored by @Reedy and deployed yesterday (yay, Friday deploys!). expanddblist now has this code:

if ( $$argv[1] === null ) {
	exit( 'Please pass a dblist expression' );
}

Note the double $$ (PHP syntax for "access variable with a variable name"). In this case, the first argument is /srv/mediawiki/dblists/growthexperiments.dblist, meaning PHP looks up a variable named exactly that. Such variable doesn't exist, so the lookup is null, calling exit( 'Please pass a dblist expression' ).

According to PHP documentation, exit() called with a string means "print the message and exit with an exit code of 0". In other words, foreachwikiindblist interprets this as "the list of DBs is literally Please pass a dblist expression, meaning all scheduled maintenance scripts running at more than one wiki (likely, all/most of them) now run on Please DB, which fails.

Urbanecm_WMF mentioned this in T434567: expanddblist without arguments doesn't fail cleanly.Sat, Aug 15, 3:11 PM
CodeReviewBot added a project: Patch-For-Review.Sat, Aug 15, 3:25 PM

urbanecm opened https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/260

expanddblist: Check for passage of DB list correctly

Urbanecm_WMF added a project: Release-Engineering-Team.Sat, Aug 15, 3:27 PM
Urbanecm_WMF mentioned this in T434981: Changes to releng/release GitLab repo can be deployed unexpectedly.Sat, Aug 15, 3:42 PM
RhinosF1 subscribed.Sat, Aug 15, 4:38 PM
• Mikeyeak86 added a subtask: T380704: Allow filtering Bitu block log.Sat, Aug 15, 11:05 PM
Reedy removed a subtask: T380704: Allow filtering Bitu block log.Sat, Aug 15, 11:51 PM
CodeReviewBot added a comment.Sun, Aug 16, 3:50 PM

jnuche merged https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/260

expanddblist: Check for passage of DB list correctly

jnuche subscribed.Sun, Aug 16, 3:52 PM

@Urbanecm_WMF Thanks for the fix. MR merged now

jnuche lowered the priority of this task from Unbreak Now! to Needs Triage.Sun, Aug 16, 3:53 PM
Maintenance_bot removed a project: Patch-For-Review.Sun, Aug 16, 4:30 PM
Aklapper mentioned this in T435034: Special:WantedCategories on enwiki failed to update.Mon, Aug 17, 8:21 AM
Aklapper merged a task: T435034: Special:WantedCategories on enwiki failed to update.Mon, Aug 17, 8:26 AM
Aklapper added subscribers: Bearcat, Les4353.
Aklapper merged a task: T434964: MediaWiki periodic job update-flaggedrev-stats failed.Mon, Aug 17, 8:48 AM
Aklapper added subscribers: phaultfinder, Bencemac, A_smart_kitten.
Ladsgroup mentioned this in T435056: MediaWiki periodic job update-flaggedrev-stats failed.Mon, Aug 17, 10:42 AM
Johannnes89 subscribed.Mon, Aug 17, 10:47 AM
Aklapper merged a task: T435071: [[Special:Unreviewed_Pages]] is no longer being updated.Mon, Aug 17, 10:52 AM
Aklapper added a subscriber: WMS.Nemo.
Clement_Goubert edited projects, added: ServiceOps-MediaWiki, ServiceOps; removed: SRE.Mon, Aug 17, 10:54 AM
Clement_Goubert subscribed.

I'm rerunning the failed jobs I can find btw.

Clement_Goubert moved this task from Inbox to In Progress on the ServiceOps board.Mon, Aug 17, 10:54 AM
Aklapper merged a task: T435056: MediaWiki periodic job update-flaggedrev-stats failed.Mon, Aug 17, 10:54 AM
Aklapper added subscribers: Ladsgroup, Tacsipacsi.
Clement_Goubert claimed this task.Mon, Aug 17, 11:15 AM

List of failed jobs left to rerun:

kubectl get jobs -A   -o custom-columns=NAME:.metadata.name,CRONJOB:.metadata.ownerReferences[0].name,FAILED:.status.failed   --no-headers   | awk '$3==1 {print}'
checkuser-revoke-temporaryaccountviewer-29782080                checkuser-revoke-temporaryaccountviewer                1
cleanup-upload-stash-29782140                                   cleanup-upload-stash                                   1
db-lag-stats-reporter-29778779                                  db-lag-stats-reporter                                  1
echo-mail-batch-29782080                                        echo-mail-batch                                        1
globalblocking-fixglobalblockwhitelist-29780640                 globalblocking-fixglobalblockwhitelist                 1
growthexperiments-deleteoldsurveys-29779395                     growthexperiments-deleteoldsurveys                     1
growthexperiments-listtaskcounts-29782511                       growthexperiments-listtaskcounts                       1
growthexperiments-purgeexpiredmentorstatus-29779725             growthexperiments-purgeexpiredmentorstatus             1
growthexperiments-refreshpraiseworthymentees-29781135           growthexperiments-refreshpraiseworthymentees           1
growthexperiments-updateisactiveflagformentees-29772582         growthexperiments-updateisactiveflagformentees         1
growthexperiments-updatementeedata-s1-29782455                  growthexperiments-updatementeedata-s1                  1
growthexperiments-updatementeedata-s2-29782455                  growthexperiments-updatementeedata-s2                  1
growthexperiments-updatementeedata-s3-29782455                  growthexperiments-updatementeedata-s3                  1
growthexperiments-updatementeedata-s4-29782455                  growthexperiments-updatementeedata-s4                  1
growthexperiments-updatementeedata-s5-29782455                  growthexperiments-updatementeedata-s5                  1
growthexperiments-updatementeedata-s6-29782455                  growthexperiments-updatementeedata-s6                  1
growthexperiments-updatementeedata-s7-29782455                  growthexperiments-updatementeedata-s7                  1
growthexperiments-updatementeedata-s8-29782455                  growthexperiments-updatementeedata-s8                  1
growthexperiments-updatemetrics-29782350                        growthexperiments-updatemetrics                        1
growthexperiments-userimpactdelete-29782210                     growthexperiments-userimpactdelete                     1
growthexperiments-userimpactupdaterecentlyedited-29782545       growthexperiments-userimpactupdaterecentlyedited       1
growthexperiments-userimpactupdaterecentlyregistered-29782395   growthexperiments-userimpactupdaterecentlyregistered   1
initsitestats-29781900                                          initsitestats                                          1
mediamoderation-hourlyscan-29782562                             mediamoderation-hourlyscan                             1
mediamoderation-updatemetrics-29782352                          mediamoderation-updatemetrics                          1
pageassessments-cleanup-29781882                                pageassessments-cleanup                                1
purge-abusefilteripdata-29782155                                purge-abusefilteripdata                                1
purge-checkuser-29782080                                        purge-checkuser                                        1
purge-expired-blocks-29782380                                   purge-expired-blocks                                   1
purge-loginnotify-29782020                                      purge-loginnotify                                      1
purge-recent-changes-29782080                                   purge-recent-changes                                   1
purge-temporary-accounts-29781507                               purge-temporary-accounts                               1
startupregistrystats-29782535                                   startupregistrystats                                   1
testkitchen-constructiveedits-29782518                          testkitchen-constructiveedits                          1
updatequerypages-lonelypages-s1-29779260                        updatequerypages-lonelypages-s1                        1
updatequerypages-mostcategories-s1-29780700                     updatequerypages-mostcategories-s1                     1
wikimediaevents-updateperiodicmetrics-per-wiki-29782360         wikimediaevents-updateperiodicmetrics-per-wiki         1
toolhub-main-crawler-29570700                                   toolhub-main-crawler                                   1
Clement_Goubert merged a task: T434952: MediaWiki periodic job initsitestats failed.Mon, Aug 17, 11:44 AM
Clement_Goubert merged a task: T434966: MediaWiki periodic job cleanup-upload-stash failed.
Clement_Goubert closed this task as Resolved.Mon, Aug 17, 11:57 AM

All jobs should now be re-running or completed.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits