Beta Cluster not picking up new messages
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	Jdforrester-WMF
	Dec 19 2020, 8:41 PM

Description

Noticed for the new IPInfo extension, and new messages in VisualEditor. Not sure if this also affects changes to existing messages?

Details

	Subject	Repo	Branch	Lines +/-
	Disable PHP L10n in beta cluster	operations/mediawiki-config	master	+0 -15

Customize query in gerrit

Related Objects

Mentioned In: T260599: Deploy IP Info extension to beta cluster
Mentioned Here: rMSCA34eb198614f9: Feature flag PHP L10n generation
T268634: Deploy Scap version 3.16.0-1
rMW815def7ae258: Add flag to rebuildLocalisationCache.php to skip MessageBlobStore clear

Event Timeline

Jdforrester-WMF created this task.Dec 19 2020, 8:41 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 19 2020, 8:41 PM

Looking on deployment-deploy01, /srv/mediawiki-staging/php-master/cache/l10n has both the CDB and the PHP files, but the latter haven't been touched since 24 November (the former have been touched a few minutes ago).

The new strings are in the CDB files but not the PHP ones, FWICT.

Did someone "switch off" the wrong one?

Jdforrester-WMF added projects: Performance-Team, Release-Engineering-Team.Dec 19 2020, 8:48 PM

Tchanders mentioned this in T260599: Deploy IP Info extension to beta cluster.Dec 19 2020, 8:50 PM

Only obvious patch in config or MW core in this area is 815def7ae2580843c00df2568446416998054785 which I think is innocent. Did we have a new version of scap get deployed?

Might be caused by the new scap release? T268634: Deploy Scap version 3.16.0-1 is suspiciously timed…

Jdforrester-WMF triaged this task as High priority.Dec 19 2020, 9:33 PM

Urbanecm subscribed.Dec 19 2020, 11:50 PM

So, in 34eb198614f9f8b1a353f0e66fbd024943697d28, @thcipriani added a feature flag to scap, to solve "disk space beta issues". However, due to your own commit, beta uses php static array for caching l10n, and subsequently, beta uses stale file for caching l10n.

I didn't dig into "since when this happens", as I don't consider that answer important.

I tried to set that feature flag in beta, however, that created a LOT of error messages like:

rsync: rename failed for "/srv/mediawiki/php-master/cache/l10n/zh-cn.l10n.php" (from php-master/cache/l10n/.~tmp~/zh-cn.l10n.php): No such file or directory (2)
rsync: rename failed for "/srv/mediawiki/php-master/cache/l10n/zh-hans.l10n.php" (from php-master/cache/l10n/.~tmp~/zh-hans.l10n.php): No such file or directory (2)

so I reverted the hack back. I'm not sure if this is a bug in scap (would be understandable, as php static files are not used anywhere AFAICS) or somewhere else.

Considering the above info, what's the best solution? Make scap generate php static files, or remove James's override for beta?

Ah, yes, that patch (included in scap 3.16.0) does indeed look like the cause. My commit from 18 months ago should have been reverted if static file generation is actually being shelved as a plan, which AFAIU isn't definitely the case?

Change 651228 had a related patch set uploaded (by Ahmon Dancy; owner: Ahmon Dancy):
[operations/mediawiki-config@master] Disable PHP L10n in beta cluster

https://gerrit.wikimedia.org/r/651228

gerritbot added a project: Patch-For-Review.Dec 21 2020, 5:36 PM

dancy subscribed.Dec 21 2020, 5:41 PM

Change 651228 merged by jenkins-bot:
[operations/mediawiki-config@master] Disable PHP L10n in beta cluster

https://gerrit.wikimedia.org/r/651228

Maintenance_bot removed a project: Patch-For-Review.Dec 21 2020, 6:11 PM

Thanks everyone for investigating and deploying over the holidays, this is really helpful to Anti-Harassment.

Krinkle removed a project: Performance-Team.Dec 21 2020, 7:51 PM