Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Tgr
	Dec 19 2014, 12:13 AM

Description

The goal of the beta cluster is to "provide an exact copy of the WMF production cluster (including db hosts, apaches, logging, squids, varnish caching, image scalers, etc.) using Wikimedia Labs instances". In the case of image scalers, it really falls short of this goal though; the setup is not even remotely similar:

prod is using swift to store the files, beta uses NFS - T64835
deployment-upload is not puppetized at all (according to the beta config overview page) so there are probably all sorts of differences in installed packages.
prod uses custom Varnish code to redirect thumbnail misses to Swift and then to the scalers (according to (1)), beta uses nginx with a 404 handler php script which is as far as I can tell not tracked in git and seems to be a modified version of a several years old revision of MediaWiki's thumb.php
~~prod uses standalone image scaler servers, on beta image scaling seems to be done by the standard apache backends (swift has thumbhost set to the text cache's internal IP)~~ Obsolete per T187899.

... and probably more, given that the beta file server was set up a long time ago, and not maintained since.

Details

	Subject	Repo	Branch	Lines +/-
	beta: Get rid of old unused upload.beta.wmflabs.org apache config	operations/puppet	production	+0 -31

Customize query in gerrit

Related Objects
Search...

View Standalone Graph

This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.

Status	Assigned	Task
		· · ·
Resolved	ori	T71757 Images with extra parameters (e.g. low-quality JPEG) are not rendered in beta
Resolved	MoritzMuehlenhoff	T112421 Update rsvg on the image scalers to 2.40.16 (to solve several SVG rendering issues)
Resolved	• AlexMonk-WMF	T129586 /mnt/upload7 does not exist anywhere, yet it is referenced in multiple places in wmf-config
Open	None	T87220 Minimize infrastructure differences between Beta Cluster and production
Resolved	Krenair	T84950 Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking)
Resolved	• AlexMonk-WMF	T64835 Setup a Swift cluster on beta-cluster to match production
Resolved	Krenair	T116816 beta cluster missing vips command needed to render tiffs and pngs, pnmtojpeg for DjVus
Resolved	fgiunchedi	T142289 Setup deployment-imagescaler host(s) in Beta Cluster
		· · ·

Event Timeline

Tgr created this task.Dec 19 2014, 12:13 AM

Tgr raised the priority of this task from to Needs Triage.

Tgr updated the task description. (Show Details)

Tgr added projects: MediaWiki-File-management, Beta-Cluster-Infrastructure, Multimedia.

Tgr changed Security from none to None.

Tgr subscribed.

greg renamed this task from Thumbnail generation should happen via the same setup in the beta cluster and in production to Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking).Dec 19 2014, 12:19 AM

greg triaged this task as Medium priority.

greg updated the task description. (Show Details)

greg added a subtask: T64835: Setup a Swift cluster on beta-cluster to match production.

greg updated the task description. (Show Details)Dec 19 2014, 12:23 AM

greg updated the task description. (Show Details)

greg subscribed.Dec 19 2014, 1:04 AM

greg moved this task from To Triage to Backlog on the Beta-Cluster-Infrastructure board.Jan 12 2015, 10:32 PM

Tgr added a subtask: T71757: Images with extra parameters (e.g. low-quality JPEG) are not rendered in beta.Feb 10 2015, 8:39 PM

hashar mentioned this in T102953: Completely remove Beta Cluster dependency on NFS.Jun 22 2015, 7:52 PM

hashar added a parent task: T102953: Completely remove Beta Cluster dependency on NFS.

Krenair mentioned this in T102963: Beta Cluster uploads (new and viewing existing files/thumbnails, including captchas) broken due to WMF Labs NFS outage.Jun 22 2015, 7:53 PM

greg removed a parent task: T102953: Completely remove Beta Cluster dependency on NFS.Jun 24 2015, 10:26 PM

Tgr mentioned this in T111132: Thumbnail scaling broken on beta.Sep 2 2015, 5:29 AM

Jdforrester-WMF moved this task from Untriaged to Backlog on the Multimedia board.Sep 4 2015, 6:27 PM

Restricted Application added subscribers: Steinsplitter, Matanya. · View Herald TranscriptSep 4 2015, 6:27 PM

hashar removed a subtask: T71757: Images with extra parameters (e.g. low-quality JPEG) are not rendered in beta.Oct 8 2015, 10:10 AM

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptOct 8 2015, 10:10 AM

hashar added a parent task: T71757: Images with extra parameters (e.g. low-quality JPEG) are not rendered in beta.Oct 8 2015, 10:11 AM

Sorry for the mess in the blocked-by/blocking. The dependency tree is now:

T71757: Images with extra parameters (e.g. low-quality JPEG) are not rendered in beta
- This bug: T84950: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking)
  - T64835: Setup a Swift cluster on beta-cluster to match production

hashar mentioned this in T64835: Setup a Swift cluster on beta-cluster to match production.Oct 8 2015, 10:16 AM

hashar mentioned this in T114998: Provide Swift object store(s) for the labs projects.Oct 8 2015, 10:24 AM

hashar changed the status of subtask T64835: Setup a Swift cluster on beta-cluster to match production from Open to Stalled.Oct 8 2015, 10:27 AM

Krenair mentioned this in T116816: beta cluster missing vips command needed to render tiffs and pngs, pnmtojpeg for DjVus.Oct 27 2015, 8:20 PM

Hinote subscribed.Oct 30 2015, 10:04 AM

waldyrious subscribed.Oct 30 2015, 12:19 PM

Tgr mentioned this in T112421: Update rsvg on the image scalers to 2.40.16 (to solve several SVG rendering issues).Dec 2 2015, 9:42 PM

Ricordisamoa subscribed.Dec 2 2015, 10:24 PM

Restricted Application added a project: Commons. · View Herald TranscriptDec 2 2015, 10:24 PM

zhuyifei1999 moved this task from Incoming to Thumbnail and file renderings on the Commons board.Dec 3 2015, 9:37 AM

zhuyifei1999 moved this task from Thumbnail and file renderings to Tracking bugs on the Commons board.Dec 3 2015, 10:02 AM

hashar changed the status of subtask T64835: Setup a Swift cluster on beta-cluster to match production from Stalled to Open.Jan 20 2016, 2:14 PM

Tgr added a parent task: T112421: Update rsvg on the image scalers to 2.40.16 (to solve several SVG rendering issues).Mar 8 2016, 7:58 PM

hashar mentioned this in T129586: /mnt/upload7 does not exist anywhere, yet it is referenced in multiple places in wmf-config.Mar 29 2016, 11:03 AM

Video also seems broken (example - upload and metadata extraction is successful but neither the file itself nor transcodes nor thumbnails are accessible). And errors related to the scaler (example) do not seem to be logged anywhere.

In T84950#2168051, @Tgr wrote:

Video also seems broken (example - upload and metadata extraction is successful but neither the file itself nor transcodes nor thumbnails are accessible). And errors related to the scaler (example) do not seem to be logged anywhere.

Those were due to T131322: deployment-upload won't start, upload.beta.wmflabs.org down. The examples look fixed now.

The videos work now; the thumbnails still look broken to me. That could be the same problem of limited thumbnail parameter handling.

Krenair subscribed.Apr 10 2016, 12:19 AM

akosiaris added a parent task: T129586: /mnt/upload7 does not exist anywhere, yet it is referenced in multiple places in wmf-config.Apr 20 2016, 11:15 AM

Danny_B added a project: Tracking-Neverending.May 23 2016, 12:17 PM

Restricted Application added a subscriber: Poyekhali. · View Herald TranscriptMay 23 2016, 12:17 PM

Jdforrester-WMF added a subtask: T116816: beta cluster missing vips command needed to render tiffs and pngs, pnmtojpeg for DjVus.Jun 23 2016, 9:27 AM

Krenair updated the task description. (Show Details)Jul 4 2016, 10:31 PM

Once I'm done with T64835, I believe the only difference left should be that production swift uses rendering.svc.eqiad.wmnet which is a load balancer for imagescalers (production can have load balancers, and does have imagescalers), whereas deployment-prep swift uses deployment-cache-text04 to hit any MW app server (labs can't have load balancers AFAIK, and we don't have any imagescalers - yet?)

• AlexMonk-WMF closed subtask T64835: Setup a Swift cluster on beta-cluster to match production as Resolved.Jul 13 2016, 8:37 PM

• AlexMonk-WMF updated the task description. (Show Details)Jul 13 2016, 8:41 PM

Change 302398 had a related patch set uploaded (by Alex Monk):
beta: Get rid of old unused upload.beta.wmflabs.org apache config

https://gerrit.wikimedia.org/r/302398

Krenair mentioned this in rOPUP2f3692af2435: beta: Get rid of old unused upload.beta.wmflabs.org apache config.Aug 2 2016, 10:03 AM

Change 302398 merged by Yuvipanda:
beta: Get rid of old unused upload.beta.wmflabs.org apache config

https://gerrit.wikimedia.org/r/302398

yuvipanda mentioned this in rOPUP0019fd6305b0: beta: Get rid of old unused upload.beta.wmflabs.org apache config.Aug 2 2016, 10:30 PM

greg moved this task from Backlog to Epics / Tracking on the Beta-Cluster-Infrastructure board.Aug 5 2016, 8:49 PM

greg mentioned this in T142289: Setup deployment-imagescaler host(s) in Beta Cluster.Aug 6 2016, 8:53 AM

greg created subtask T142289: Setup deployment-imagescaler host(s) in Beta Cluster.

Krenair added a parent task: T87220: Minimize infrastructure differences between Beta Cluster and production.Aug 24 2016, 11:34 PM

I've also finally got around to deploying https://gerrit.wikimedia.org/r/#/c/298397/ which converted the whole separate realm filebackend config files into a single file with a small realm branch to minimise the differences there

fgiunchedi closed subtask T142289: Setup deployment-imagescaler host(s) in Beta Cluster as Resolved.Jul 24 2017, 9:29 AM

Krinkle updated the task description. (Show Details)Feb 23 2018, 1:31 AM