Page MenuHomePhabricator

image sizes not displayed on beta
Closed, DuplicatePublicBUG REPORT

Description

  • Go to any file on beta cluster
  • Read the text below the image

What happens?:

What should have happened instead?:
Variables are normally defined resulting in something like "Size of this preview: 800 × 479 pixels. Other resolutions: 320 × 191 pixels | 640 × 383 pixels | 1,024 × 613 pixels | 1,280 × 766 pixels | 2,567 × 1,536 pixels. Original file ‎(2,567 × 1,536 pixels, file size: 1.3 MB, MIME type: image/jpeg) "

Event Timeline

AlexisJazz updated the task description. (Show Details)
AlexisJazz updated the task description. (Show Details)

I couldn't find anything that might connect this issue to the risky patch (T281152#7144507) and it might be an issue with the l10n cache of beta cluster. I think we should wait until deployment to group0 and see if it happens there too

Specially since I can't reproduce the issue locally at all.

it might be an issue with the l10n cache of beta cluster.

Could be, as e.g. https://meta.wikimedia.beta.wmflabs.org/wiki/Special:CentralAuth/Test shows ⧼centralauth-admin-notblocked⧽.

Ladsgroup removed subscribers: Pchelolo, tstarling.

it might be an issue with the l10n cache of beta cluster.

Could be, as e.g. https://meta.wikimedia.beta.wmflabs.org/wiki/Special:CentralAuth/Test shows ⧼centralauth-admin-notblocked⧽.

Yeah. It doesn't look like it's related to the file metadata refactor then.

Majavah added a subscriber: Majavah.

Yeah, that looks like something broken with beta itself. The relevant message keys are present in deployment-mediawiki11:/srv/mediawiki/php-master/cache/l10n/upstream/l10n_cache-en.cdb.json, I'm trying to figure out if there are any relevant logs

T285119: beta: Error: invalid magic word 'ns' and T285118: beta: Error: Unsupported operand types - though they're more symptoms rather than the cause (pointing at some l10n cache related issue)

@dancy Hi, do you imagine the recent scap python3 migration might be causing these?

https://logstash-beta.wmcloud.org (works now, if not clean up /var/log/logstash/logstash-json.log and restart logstash.service on deployment-logstash[04-06]) is full of php unserialize errors in localization related code, so something is definitely going wrong when generating the localization cache files, and given the files itself appear to be fully invalid (instead of having the wrong contents or something like that) my best guess is something in scap changed

Scap just calls a MW maintenance script to make them…

@dancy Hi, do you imagine the recent scap python3 migration might be causing these?

@dancy will be out next week as will I. @LarsWirzenius may be able to take a look early next week. Otherwise, it'll be the following week before we're in a position to take a look.

We'v been porting Scap to Python3 latetly, and this _might_ be Unicode string confusion problem somewhere. I don't know that it is, though, nor can I see where it might be.

I tried downgrading scap in beta to match the version in prod (by running sudo apt-get install scap=3.17.1-1 on deployment-deploy01.deployment-prep) to see if that narrows down the issue.

a similar issue is happening here
https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version
version: 1.37.0-alpha (1f66af0) 09:37, 24 June 2021

image.png (368×706 px, 36 KB)

on test wiki
version: 1.37.0-wmf.11 (32a113b) 22:42, 23 June 2021

image.png (448×758 px, 55 KB)

Mentioned in SAL (#wikimedia-releng) [2021-06-28T11:59:22Z] <majavah> downgrade scap to 3.17.1-1 (matching production) on deployment-mediawiki11, testing for T285125

I tried downgrading scap in beta to match the version in prod (by running sudo apt-get install scap=3.17.1-1 on deployment-deploy01.deployment-prep) to see if that narrows down the issue.

I downgraded an appserver (deployment-mediawiki11) too and force regenerated English language caches like this:

taavi@deployment-mediawiki11:/srv/mediawiki/php-master/cache/l10n$ sudo rm l10n_cache-en.cdb 
taavi@deployment-mediawiki11:/srv/mediawiki/php-master/cache/l10n$ sudo -u mwdeploy /usr/bin/scap cdb-rebuild
l10n merge: 100% (ok: 439; fail: 0; left: 0)                                    
12:05:31 Updated 1 CDB files(s) in /srv/mediawiki/php-master/cache/l10n

The issue was fixed after this. To confirm it was really a problem with the latest scap versions I re-upgraded scap (sudo apt-get upgrade) and force regenerated English caches once again. The issues was back, so that confirms it's a Scap issue.