Page MenuHomePhabricator

Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking)
Closed, ResolvedPublic

Description

The goal of the beta cluster is to "provide an exact copy of the WMF production cluster (including db hosts, apaches, logging, squids, varnish caching, image scalers, etc.) using Wikimedia Labs instances". In the case of image scalers, it really falls short of this goal though; the setup is not even remotely similar:

  • prod is using swift to store the files, beta uses NFS - T64835
  • deployment-upload is not puppetized at all (according to the beta config overview page) so there are probably all sorts of differences in installed packages.
  • prod uses custom Varnish code to redirect thumbnail misses to Swift and then to the scalers (according to (1)), beta uses nginx with a 404 handler php script which is as far as I can tell not tracked in git and seems to be a modified version of a several years old revision of MediaWiki's thumb.php
  • prod uses standalone image scaler servers, on beta image scaling seems to be done by the standard apache backends (swift has thumbhost set to the text cache's internal IP) Obsolete per T187899.

... and probably more, given that the beta file server was set up a long time ago, and not maintained since.

Related Objects

Event Timeline

Tgr created this task.Dec 19 2014, 12:13 AM
Tgr raised the priority of this task from to Needs Triage.
Tgr updated the task description. (Show Details)
Tgr changed Security from none to None.
Tgr added a subscriber: Tgr.
greg renamed this task from Thumbnail generation should happen via the same setup in the beta cluster and in production to Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking).Dec 19 2014, 12:19 AM
greg triaged this task as Normal priority.
greg updated the task description. (Show Details)
greg updated the task description. (Show Details)Dec 19 2014, 12:23 AM
greg updated the task description. (Show Details)
greg added a subscriber: greg.Dec 19 2014, 1:04 AM
Jdforrester-WMF moved this task from Untriaged to Backlog on the Multimedia board.Sep 4 2015, 6:27 PM
Restricted Application added subscribers: Steinsplitter, Matanya. · View Herald TranscriptSep 4 2015, 6:27 PM
Restricted Application added a subscriber: Luke081515. · View Herald TranscriptOct 8 2015, 10:10 AM
Hinote added a subscriber: Hinote.Oct 30 2015, 10:04 AM
Restricted Application added a project: Commons. · View Herald TranscriptDec 2 2015, 10:24 PM
Tgr added a comment.Apr 1 2016, 11:30 AM

Video also seems broken (example - upload and metadata extraction is successful but neither the file itself nor transcodes nor thumbnails are accessible). And errors related to the scaler (example) do not seem to be logged anywhere.

greg added a comment.Apr 1 2016, 9:22 PM
In T84950#2168051, @Tgr wrote:

Video also seems broken (example - upload and metadata extraction is successful but neither the file itself nor transcodes nor thumbnails are accessible). And errors related to the scaler (example) do not seem to be logged anywhere.

Those were due to T131322: deployment-upload won't start, upload.beta.wmflabs.org down. The examples look fixed now.

Tgr added a comment.Apr 2 2016, 6:27 AM

The videos work now; the thumbnails still look broken to me. That could be the same problem of limited thumbnail parameter handling.

Restricted Application added a subscriber: Poyekhali. · View Herald TranscriptMay 23 2016, 12:17 PM
Krenair updated the task description. (Show Details)Jul 4 2016, 10:31 PM

Once I'm done with T64835, I believe the only difference left should be that production swift uses rendering.svc.eqiad.wmnet which is a load balancer for imagescalers (production can have load balancers, and does have imagescalers), whereas deployment-prep swift uses deployment-cache-text04 to hit any MW app server (labs can't have load balancers AFAIK, and we don't have any imagescalers - yet?)

Change 302398 had a related patch set uploaded (by Alex Monk):
beta: Get rid of old unused upload.beta.wmflabs.org apache config

https://gerrit.wikimedia.org/r/302398

Change 302398 merged by Yuvipanda:
beta: Get rid of old unused upload.beta.wmflabs.org apache config

https://gerrit.wikimedia.org/r/302398

I've also finally got around to deploying https://gerrit.wikimedia.org/r/#/c/298397/ which converted the whole separate realm filebackend config files into a single file with a small realm branch to minimise the differences there

Krinkle updated the task description. (Show Details)Feb 23 2018, 1:31 AM
Krenair closed this task as Resolved.Jul 30 2018, 9:17 PM
Krenair claimed this task.

I think between my NFS -> Swift migration and Thumbor this has been done. If anyone else disagrees feel free to reopen.

Tgr awarded a token.Jul 31 2018, 12:38 PM