Page MenuHomePhabricator

Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking)
Closed, ResolvedPublic

Description

The goal of the beta cluster is to "provide an exact copy of the WMF production cluster (including db hosts, apaches, logging, squids, varnish caching, image scalers, etc.) using Wikimedia Labs instances". In the case of image scalers, it really falls short of this goal though; the setup is not even remotely similar:

  • prod is using swift to store the files, beta uses NFS - T64835
  • deployment-upload is not puppetized at all (according to the beta config overview page) so there are probably all sorts of differences in installed packages.
  • prod uses custom Varnish code to redirect thumbnail misses to Swift and then to the scalers (according to (1)), beta uses nginx with a 404 handler php script which is as far as I can tell not tracked in git and seems to be a modified version of a several years old revision of MediaWiki's thumb.php
  • prod uses standalone image scaler servers, on beta image scaling seems to be done by the standard apache backends (swift has thumbhost set to the text cache's internal IP) Obsolete per T187899.

... and probably more, given that the beta file server was set up a long time ago, and not maintained since.

Related Objects

Event Timeline

Tgr raised the priority of this task from to Needs Triage.
Tgr updated the task description. (Show Details)
Tgr changed Security from none to None.
Tgr added a subscriber: Tgr.
greg renamed this task from Thumbnail generation should happen via the same setup in the beta cluster and in production to Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking).Dec 19 2014, 12:19 AM
greg triaged this task as Medium priority.
greg updated the task description. (Show Details)
greg updated the task description. (Show Details)

Video also seems broken (example - upload and metadata extraction is successful but neither the file itself nor transcodes nor thumbnails are accessible). And errors related to the scaler (example) do not seem to be logged anywhere.

In T84950#2168051, @Tgr wrote:

Video also seems broken (example - upload and metadata extraction is successful but neither the file itself nor transcodes nor thumbnails are accessible). And errors related to the scaler (example) do not seem to be logged anywhere.

Those were due to T131322: deployment-upload won't start, upload.beta.wmflabs.org down. The examples look fixed now.

The videos work now; the thumbnails still look broken to me. That could be the same problem of limited thumbnail parameter handling.

Once I'm done with T64835, I believe the only difference left should be that production swift uses rendering.svc.eqiad.wmnet which is a load balancer for imagescalers (production can have load balancers, and does have imagescalers), whereas deployment-prep swift uses deployment-cache-text04 to hit any MW app server (labs can't have load balancers AFAIK, and we don't have any imagescalers - yet?)

Change 302398 had a related patch set uploaded (by Alex Monk):
beta: Get rid of old unused upload.beta.wmflabs.org apache config

https://gerrit.wikimedia.org/r/302398

Change 302398 merged by Yuvipanda:
beta: Get rid of old unused upload.beta.wmflabs.org apache config

https://gerrit.wikimedia.org/r/302398

I've also finally got around to deploying https://gerrit.wikimedia.org/r/#/c/298397/ which converted the whole separate realm filebackend config files into a single file with a small realm branch to minimise the differences there

Krenair claimed this task.

I think between my NFS -> Swift migration and Thumbor this has been done. If anyone else disagrees feel free to reopen.