Page MenuHomePhabricator

On beta commons, thumbnailing of 3D files is broken still
Closed, ResolvedPublic

Description

https://commons.wikimedia.beta.wmflabs.org/wiki/File:Crystal-NEW.stl

This file, uploaded since thumbnails for other files was fixed (see other files around it at https://commons.wikimedia.beta.wmflabs.org/wiki/Special:NewFiles), does not thumbnail correctly. I tried to debug, but apparently my logstash-fu is insufficient to glean anything useful.

As far as I can tell, thumbnailing is working locally, so it's probably another beta configuration issue that we missed.

Details

Related Gerrit Patches:
operations/debs/python-thumbor-wikimedia : masterAdd missing THREED2PNG_PATH

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 12 2017, 5:39 PM
mmodell added a subscriber: Gilles.

Nobody from Release-Engineering-Team are familiar with Thumbor, maybe @Gilles can take a look at it?

Thumbor doesn't support those yet, you guys have to figure out how to deploy 3d2png properly first. There's a change cherry-picked for it on beta, but it keeps generating errors: T166013: deployment-imagescaler01 has no mwdeploy user and the executable is nowhere in sight.

fixed (or rather, worked around) T166013 by cherry picking https://gerrit.wikimedia.org/r/#/c/365891/

mmodell triaged this task as Medium priority.Jul 18 2017, 3:28 AM

@MarkTraceur: does this resolve the issue for you? If so then we just need to get the patch merged so that it doesn't have to remain cherry picked forever.

Gilles added a comment.EditedJul 18 2017, 4:49 PM

Thumbor has no support for STL yet, and it can't get it until 3d2png is available. The first step is getting 3d2png to be on the server at all and function properly. I see that the sources are there now:

gilles@deployment-imagescaler01:~$ ls -al /srv/deployment/3d2png/deploy/
total 32
drwxr-xr-x   6 mwdeploy mwdeploy 4096 Jul 18 03:27 .
drwxr-xr-x   3 mwdeploy mwdeploy 4096 Jul 18 03:26 ..
drwxr-xr-x   9 mwdeploy mwdeploy 4096 Jul 18 03:27 .git
-rw-r--r--   1 mwdeploy mwdeploy  120 Jul 18 03:27 .gitmodules
-rw-r--r--   1 mwdeploy mwdeploy  109 Jul 18 03:26 .gitreview
drwxr-xr-x 207 mwdeploy mwdeploy 4096 Jul 18 03:27 node_modules
lrwxrwxrwx   1 mwdeploy mwdeploy   16 Jul 18 03:27 package.json -> src/package.json
drwxr-xr-x   2 mwdeploy mwdeploy 4096 Jul 18 03:27 scap
drwxr-xr-x   3 mwdeploy mwdeploy 4096 Jul 18 03:27 src
gilles@deployment-imagescaler01:~$

But I don't see it installed as a command. If I try to run it crudely via the source, it doesn't find the libraries:

gilles@deployment-imagescaler01:~$ node /srv/deployment/3d2png/deploy/src/3d2png.js Crystal-NEW.stl 800x600 foo.png
THREE.WebGLRenderer 84
THREE.WebGLRenderer: TypeError: Cannot read property 'getShaderPrecisionFormat' of undefined
/srv/deployment/3d2png/deploy-cache/revs/92c7a1b729cf67ed96a2272a5fceaada750b02c0/node_modules/three/build/three.js:19592
						extension = gl.getExtension( 'WEBGL_depth_texture' ) || gl.getExtension( 'MOZ_WEBGL_depth_texture' ) || gl.getExtension( 'WEBKIT_WEBGL_depth_texture' );
						              ^

TypeError: Cannot read property 'getExtension' of undefined
    at Object.get (/srv/deployment/3d2png/deploy-cache/revs/92c7a1b729cf67ed96a2272a5fceaada750b02c0/node_modules/three/build/three.js:19592:21)
    at new WebGLRenderer (/srv/deployment/3d2png/deploy-cache/revs/92c7a1b729cf67ed96a2272a5fceaada750b02c0/node_modules/three/build/three.js:20028:14)
    at new ThreeDtoPNG (/srv/deployment/3d2png/deploy-cache/revs/92c7a1b729cf67ed96a2272a5fceaada750b02c0/src/3d2png.js:22:18)
    at Object.<anonymous> (/srv/deployment/3d2png/deploy-cache/revs/92c7a1b729cf67ed96a2272a5fceaada750b02c0/src/3d2png.js:207:10)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)

Is it supposed to be installed as a command somewhere not in the path? Are you using a more complex command to invoke it from source?

Gilles added a comment.EditedJul 18 2017, 5:04 PM

I think I've found the actual command in mediawiki-config, but xvfb-run errors:

gilles@deployment-imagescaler01:~$ /usr/bin/xvfb-run -a -s -ac -screen 0 1280x1024x24 /srv/deployment/3d2png/deploy/src/3d2png.js Crystal-NEW.stl 800x600 foo.png
/usr/bin/xvfb-run: 183: /usr/bin/xvfb-run: 0: not found

OK, finally found the right command to invoke it:

gilles@deployment-imagescaler01:~$ DISPLAY=:99 /usr/bin/xvfb-run -a -s "-ac -screen 0 1280x1024x24" /srv/deployment/3d2png/deploy/src/3d2png.js Crystal-NEW.stl 800x600 foo.png
THREE.WebGLRenderer 84
THREE.WebGLRenderer: TypeError: _canvas.addEventListener is not a function
THREE.WebGLRenderer: WEBGL_depth_texture extension not supported.
THREE.WebGLRenderer: OES_texture_float extension not supported.
THREE.WebGLRenderer: OES_texture_float_linear extension not supported.
THREE.WebGLRenderer: OES_texture_half_float extension not supported.
THREE.WebGLRenderer: OES_texture_half_float_linear extension not supported.
THREE.WebGLRenderer: OES_standard_derivatives extension not supported.
THREE.WebGLRenderer: OES_element_index_uint extension not supported.

Which succeeds, generating this image:

Which means we can pick up work on T161719: Add STL support (with 3d2png) to Thumbor

So is this waiting for T161719: Add STL support (with 3d2png) to Thumbor or is there something that needs to happen here? If so, by who?

Gilles added a comment.Aug 9 2017, 1:54 PM

Yes, now that Thumbor is serving all thumbnail traffic, T161719 is the blocker for this task. @MarkTraceur and I got a solution working locally yesterday, so I expect that it'll be working in beta and possibly production in the next couple of weeks.

Change 373595 had a related patch set uploaded (by Matthias Mullie; owner: Matthias Mullie):
[operations/debs/python-thumbor-wikimedia@master] Add missing THREED2PNG_PATH

https://gerrit.wikimedia.org/r/373595

Change 373595 abandoned by Matthias Mullie:
Add missing THREED2PNG_PATH

Reason:
Based on discussion (email), this path does not belong in this repo, but in config somewhere else (puppet)

https://gerrit.wikimedia.org/r/373595

I'm happy to report that it works now: https://commons.wikimedia.beta.wmflabs.org/wiki/File:Crystal-NEW.stl

But a number of things are temporary and needs to be made permanent:

  • the THREED2PNG_PATH config line needs to be puppetized.
  • the cherry-picked change to 3d2png needs to be properly deployed
  • T174746 needs to be fixed (already working on it)
MarkTraceur moved this task from Doing to Next up on the Multimedia board.Sep 7 2017, 4:16 PM
MarkTraceur removed a project: Patch-For-Review.
Gilles moved this task from Backlog to Radar on the Thumbor board.Sep 8 2017, 7:11 AM
MarkTraceur raised the priority of this task from Medium to High.Sep 11 2017, 4:25 PM
MarkTraceur moved this task from Next up to Needs code review on the Multimedia board.

@matthiasmullie this is still in "code review" on the Multimedia-Team-Working-Board ... shouldn't it be in 'to do' seeing as you have some puppet work to do?

This is now deployed properly (on beta)

Looks good now on Beta. Tried a couple of articles and some different browsers.

ABorbaWMF closed this task as Resolved.Sep 29 2017, 12:12 AM

Forgot to resolve

Cparle removed a subscriber: Cparle.Oct 11 2017, 1:16 PM