Page MenuHomePhabricator

MediaWiki production config change not/randomly picked up by startup module after deployment
Closed, DuplicatePublic

Description

Earlier today I successfully deployed a trivial config change: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/545779/

sync-proxies: 100% (ok: 9; fail: 0; left: 0)                                    
09:12:02 Finished sync-proxies (duration: 00m 02s)
09:12:02 Started sync-apaches
sync-apaches: 100% (ok: 311; fail: 0; left: 0)                                  
09:12:09 Finished sync-apaches (duration: 00m 07s)
09:12:09 Check php-fpm cache...
09:12:15 Synchronized wmf-config/InitialiseSettings.php: T234853 Re-enable performance perception survey on ruwiki (duration: 01m 04s)

And now I'm seeing that the startup module randomly picks up the change (appearing, then disappearing), depending on which app server generates the startup module:

~/Downloads $ curl -s "https://ru.wikipedia.org/w/load.php?lang=en&modules=startup&only=scripts&raw=1&skin=vector" | grep perceived

(generated by mw1331.eqiad.wmnet)

~/Downloads $ curl -s "https://ru.wikipedia.org/w/load.php?lang=en&modules=startup&only=scripts&raw=1&skin=vector" | grep perceived
"ext.quicksurveys.survey.reader-demographics-ru","mmmma"],["ext.quicksurveys.survey.perceived-performance-survey","8wwz6"]]);mw.config.set({"debug":!1,"skin":"vector","stylepath":"/w/skins","wgUrlProtocols":"bitcoin\\:|ftp\\:\\/\\/|ftps\\:\\/\\/|geo\\:|git\\:\\/\\/|gopher\\:\\/\\/|http\\:\\/\\/|https\\:\\/\\/|irc\\:\\/\\/|ircs\\:\\/\\/|magnet\\:|mailto\\:|mms\\:\\/\\/|news\\:|nntp\\:\\/\\/|redis\\:\\/\\/|sftp\\:\\/\\/|sip\\:|sips\\:|sms\\:|ssh\\:\\/\\/|svn\\:\\/\\/|tel\\:|telnet\\:\\/\\/|urn\\:|worldwind\\:\\/\\/|xmpp\\:|\\/\\/","wgArticlePath":"/wiki/$1","wgScriptPath":"/w","wgScript":"/w/index.php","wgSearchType":"CirrusSearch","wgVariantArticlePath":!1,"wgActionPaths":{},"wgServer":"//ru.wikipedia.org","wgServerName":"ru.wikipedia.org","wgUserLanguage":"en","wgContentLanguage":"ru","wgTranslateNumerals":!0,"wgVersion":"1.35.0-wmf.2","wgEnableAPI":!0,"wgEnableWriteAPI":!0,"wgFormattedNamespaces":{"-2":"Медиа","-1":"Служебная","0":"","1":
"link":"Reader-demographics-2-link","instanceTokenParameterName":"entry.1791119923","isInsecure":!1},{"audience":[],"name":"perceived-performance-survey","question":"ext-quicksurveys-performance-internal-survey-question","description":null,"module":"ext.quicksurveys.survey.perceived-performance-survey","coverage":0,"platforms":{"desktop":["stable"]},"privacyPolicy":"ext-quicksurveys-performance-internal-survey-privacy-policy","type":"internal","answers":["ext-quicksurveys-example-internal-survey-answer-positive","ext-quicksurveys-example-internal-survey-answer-neutral","ext-quicksurveys-example-internal-survey-answer-negative"],"shuffleAnswersDisplay":!0,"freeformTextLabel":null}],"wgCentralNoticeActiveBannerDispatcher":"//meta.wikimedia.org/w/index.php?title=Special:BannerLoader","wgCentralSelectedBannerDispatcher":"//meta.wikimedia.org/w/index.php?title=Special:BannerLoader","wgCentralBannerRecorder":"//ru.wikipedia.org/beacon/impression","wgCentralNoticeSampleRate":0.01,

(generated by mw1239.eqiad.wmnet)

~/Downloads $ curl -s "https://ru.wikipedia.org/w/load.php?lang=en&modules=startup&only=scripts&raw=1&skin=vector" | grep perceived

(generated by mw1262.eqiad.wmnet)

I went on the actual web servers and could see that InitialiseSettings.php has the up-to-date content:

gilles@mw1262:~$ cat /srv/mediawiki/wmf-config/InitialiseSettings.php | grep T187299
		// T187299
		// T187299
		// T187299
		// T187299
		// T187299
gilles@mw1262:~$

This feels like a caching issue at the PHP level, where it doesn't reliably pick up the updated config file.

Event Timeline

Gilles created this task.Oct 24 2019, 10:24 AM
Restricted Application added subscribers: Liuxinyu970226, Aklapper. · View Herald TranscriptOct 24 2019, 10:24 AM
Gilles removed Gilles as the assignee of this task.Oct 24 2019, 10:24 AM
Gilles updated the task description. (Show Details)
Gilles renamed this task from MediaWiki config change not picked up after deployment to MediaWiki production config change not picked up after deployment.Oct 24 2019, 10:26 AM
Gilles assigned this task to Krinkle.
Gilles triaged this task as High priority.

Assigning this to you @Krinkle so you can do more digging on the ResourceLoader side of things.

Gilles renamed this task from MediaWiki production config change not picked up after deployment to MediaWiki production config change not picked up by startup module after deployment.Oct 24 2019, 10:28 AM
Gilles renamed this task from MediaWiki production config change not picked up by startup module after deployment to MediaWiki production config change not/randomly picked up by startup module after deployment.

By the looks of the survey dashboard, the issue might have gone away, but it's probably because another change to InitialiseSettings.php was just deployed: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/545816/

11:35 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: 3a5cb68: Permission changes of move-rootuserpages assignment at commonswiki (T236359) (duration: 01m 00s)
Restricted Application removed a subscriber: Liuxinyu970226. · View Herald TranscriptOct 24 2019, 2:19 PM