Page MenuHomePhabricator

Support the glTF 2.0 3D file format
Open, Needs TriagePublic

Description

This royalty-free standard format was enabled today on Facebook, which means it's likely to take off a bit (and can be exported from Blender).

The format is described at https://www.khronos.org/gltf/ supports textures and animations (which STL doesn't) and is designed to be efficient for transmission and client-side interpretation, which would work nicely for us.

Obviously this'd be a fair bit of work, not least the security concerns. Would be interesting to see what Facebook have done in this regard, and whether we could reuse their work if it's FLOSS.

It looks like most of what we need is in three.js, which we already use for the stl rendering, so rough plan for using the .glb binary variant would be:

  • add .glb and proper mime type to core's mime.info
  • add a validation hook for uploaded files to 3D (magic number, proper sections, valid json, no external uri references)
  • split ThreeDHandler into a base class, with stl and glb subclasses to do validation and extraction differently for each supports type
  • extract asset copyright and generator fields from the json as metadata when present, expose them in the table on File: page
  • include three.js's gltf loader in 3D and 3d2png's js modules
  • change thumbor config to pass glb through the 3d2png handler
  • add mmv plugin config for glb as well as stl
  • change varnish config to gzip the glb files on transfer

Event Timeline

Restricted Application added a project: Multimedia. · View Herald TranscriptFeb 21 2018, 12:27 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
brion added a subscriber: brion.Feb 21 2018, 1:10 PM

Note glTF is supported by the BabylonJS engine - https://doc.babylonjs.com/how_to/gltf - which is under Apache 2 license. I don't know how heavyweight it is for a minimal viewer (it's a fairly full-featured 3d engine which can be used for games and such as well as simple model viewing).

brion added a comment.Feb 21 2018, 1:13 PM

Facebook seems to be using three.js for viewing -- https://threejs.org/docs/#examples/loaders/GLTFLoader

brion added a comment.Feb 21 2018, 1:26 PM

Peeking at the spec, the problem is going to be with external files for textures and binary data -- JSON glTF files can either contain their own data as base-64 data: URIs, or reference external files, so a glTF file *can* be standalone but doesn't *have* to be.

They'll need to be validated on upload to make sure no absolute URIs are used (evil!) or relative URI references left dangling (won't work).

There are also binary glTF files, which is a sort of archive format that bundles the JSON file along with a binary data chunk containing images/etc. It has a clear magic number which should make detection of file type easy, but the ordering of the data inside the binary chunk seems to be up to the glTF JSON description. (As with other binary file types it would be possible to embed Files We Don't Like semi-invisibly, such as embedding copyrighted images or sounds or movies. How much this is an issue I can't say.)

brion added a comment.Feb 21 2018, 1:40 PM

One more thing -- in the binary format, the raw binary chunk comes after the JSON chunk and has no footer/trailer at the end, so it'd be possible to create a binary glTF file that's parseable as .zip/.jar which uses a trailer instead of a header. Our existing upload-time checks should catch this case.

They'll need to be validated on upload to make sure no absolute URIs are used (evil!) or relative URI references left dangling (won't work).

Yeah, not too hard. Theoretically we could transform-on-upload with absolute URIs and re-package, but that feels icky.

Gilles added a subscriber: Gilles.EditedFeb 23 2018, 9:23 PM

Why not have the upload UI ask for referenced files (if any) and always roll it all into a single document, since that's supported?

donmccurdy added a subscriber: donmccurdy.EditedFeb 24 2018, 12:04 AM

Hi, one of the three.js contributors (and disclaimer, glTF working group member) here.

Thought I'd point out that Facebook is requiring all uploads use the binary (.glb) version of glTF — they don't accept any external files whatsoever. It would be reasonable to make the same requirement here. The Blender exporter has a feature to embed everything to binary, and options exist for packing existing glTF files into a single .glb document as well. Let me know if I can help with anything. :)

brion added a comment.Feb 24 2018, 4:18 AM

Nice, that should simplify things if we only accept binary .glb. :) As far as I can tell from spec a binary file can still reference external resources if it wants, but the validation step could detect that easily and return an explanatory error message.

Oh -- and as a todo note we'll want to add http-level gzip compression for the .glb files too, just as we added for .stl. Easy to do and should save bandwidth on the JSON part of the blob and the meshes. (Textures are pre compressed, but gzip won't hurt on them.)

brion added a comment.Feb 24 2018, 4:23 AM

(I like the idea of letting you bundle multiple files together and do the transform in the UI, I just expect it to be tricky to integrate into UploadWizard. Will be faster to get going if we leave that to external tooling.)

brion updated the task description. (Show Details)Feb 24 2018, 9:07 AM
brion updated the task description. (Show Details)Feb 24 2018, 12:54 PM

As far as I can tell from spec a binary file can still reference external resources if it wants...

That's correct; it's atypical but certainly possible. There's an official validator as well, which reports external resources along with many other issues, e.g. —

resources: [
  {pointer: "/buffers/0", mimeType: "application/gltf-buffer", storage: "external", uri: "scene.bin", byteLength: 2617976}
  {pointer: "/images/0", mimeType: "image/png", storage: "external", uri: "textures/lainmat_baseColor.png", image: {…}}
  {pointer: "/images/1", mimeType: "image/png", storage: "external", uri: "textures/lainmatAO_baseColor.png", image: {…}}
  {pointer: "/images/2", mimeType: "image/png", storage: "external", uri: "textures/lainmatshadeless_baseColor.png", image: {…}}
]

It might be best to reject uploads with any outright errors, as we can't guarantee those will work in future versions of three.js.

Did a quick peek at the GLTFLoader for three.js. Couple things would need to be changed upstream to work in Node for 3d2png:

  • it tries to use window.TextDecoder if available to decode the UTF-8 JSON blob; there's a fallback but it throws a runtime error before it gets to the fallback because window is undefined.
  • it uses the Blob constructor and URL.createObjectURL to create web-loadable URIs for textures out of the buffer views. There is no Blob constructor, and no URL, in the node environment. Need to change it to use some other kind of loader?
  • even if you had blob URLs, three's ImageLoader uses DOM stuff which won't work. Appears to be a pluggable system by regex, so easy to add support for a fake protocol like 'blob:'

Change 415495 had a related patch set uploaded (by Brion VIBBER; owner: Brion VIBBER):
[3d2png@master] [WIP] Tweaking 3d2png to support glTF binary (.glb)

https://gerrit.wikimedia.org/r/415495

brion added a comment.Mar 1 2018, 1:40 AM

Provisional patch to 3d2png doesn't render successfully yet, but does load the scene data. ;)

brion added a comment.Mar 1 2018, 8:35 PM

Now renders the "Avocado.glb" and "Boombox.glb" sample files successfully. :)

Had to adjust the near frustum on the camera from 1 to 0.01 since glTF files often seem to use meters for units, and our centering was putting us at a pretty close position within 1 meter. Also made a tweak to the node texture loading hack which makes textures actually work (use THREE.Texture, not THREE.DataTexture even though it sounds like what I want it doesn't work).

Shouldn't be super difficult to use the GLTFLoader in the web view once that frustum tweak is in (it won't need the rest of the node-specific loader tweaks, just a slight adjustment to centering to handle passing through Scene and Object3D objects to find the mesh).

Looking great! A few notes that may be helpful:

From spec, on coordinate systems:

  • glTF uses a right-handed coordinate system, that is, the cross product of +X and +Y yields +Z. glTF defines +Y as up. The front of a glTF asset faces +Z.
  • The units for all linear distances are meters.

And, although I haven't tested this in Node.js, CommonJS imports _should_ let you pull the specific versions associated with your three.js version this way:

const THREE = require('three');
require('three/examples/js/loaders/STLLoader');
require('three/examples/js/loaders/GLTFLoader');

var loader = new THREE.GLTFLoader();

This won't work with ES6 modules unfortunately. The community-maintained npm packages for individual loaders tend to be pretty out of date.

Hey I implemented gltf support in facebook so happy to jump in here however i can. I would definitely advise to start with a limited format set, ie just use glb, trying to convert everyone elses models into what you can understand is a daunting proposition.

You'll also need to figure out what extensions you want to support. At fb we support the KHR_materials_unlit extension as a lot of photogrammetry models use it, or models that want to bake really fancy lighting into the textures. Sketchfab seems to have a lot of SpecularGlossiness models which we try to convert to regular PBR but i wouldn't recommend it, at least to start.

Hi @Msfeldstein

Thanks for joining the conversation. I think we'll have a lot more questions for you soon :)

Hey I implemented gltf support in facebook so happy to jump in here however i can. I would definitely advise to start with a limited format set, ie just use glb, trying to convert everyone elses models into what you can understand is a daunting proposition.
You'll also need to figure out what extensions you want to support. At fb we support the KHR_materials_unlit extension as a lot of photogrammetry models use it, or models that want to bake really fancy lighting into the textures. Sketchfab seems to have a lot of SpecularGlossiness models which we try to convert to regular PBR but i wouldn't recommend it, at least to start.

@brion is it too late to chime in and offer Babylon.js support?

We have everything available out of the box including a 1st class glTF support: https://github.com/cx20/gltf-test
The minimal viewer size could be around 200K (gzipped) depending on options you want

You can try the sandbox here: https://sandbox.babylonjs.com to see how it will render

Also wanted to mention the Babylon.js viewer: http://doc.babylonjs.com/extensions/the_babylon_viewer which can really reduce development time

Hi,

I'd like to contribute our Darth Suzanne model to the development process.

(under CC0)
It was exported using Verge3D for Blender (a Three.js-based plug-in).
Here is how it looks online: https://cdn.soft8soft.com/demo/applications/pbr/pbr.html
The source files for this glb model are available in the Verge3D distro: https://www.soft8soft.com/get-verge3d/

Hi @SantaWinsAgain, thanks for your comment. This task is about making the MediaWiki software support that file format.
After that has happened you are more than welcome to donate such files to Wikimedia Commons. Thanks!

brion added a comment.Apr 29 2018, 5:56 PM

@Deltakosh babylonjs is something to consider too! One thing is we need a headless mode to render flat thumbnails as well as the live viewer... We've managed to get three.js running under node with headless WebGL, though sometimes it's a little funky to add new features to it. Is there a headless mode available for babylon as well? (In theory we could change from node to headless Chrome or Firefox but that's a whole other thing to arrange with ops, probably. :)

It is also working and we have several users already doing it.
We went to Firefox on our side for ops (running visual tests for incoming PR)

Feel free to ping me directly if you need help: Davca@microsoft.com

JeanFred moved this task from Backlog to File format support on the 3D board.May 23 2018, 12:07 PM

This is really great additional 3D model support is being worked on. Can I ask if there is much documentation around converting popular 3D file formats like .OBJ and .PLY to this format? I'd hate for people to not use this due to lack of good quality documentation for easy process to convert the files to this format (like what happened with audio and video before .webm).

This is really great additional 3D model support is being worked on. Can I ask if there is much documentation around converting popular 3D file formats like .OBJ and .PLY to this format? I'd hate for people to not use this due to lack of good quality documentation for easy process to convert the files to this format (like what happened with audio and video before .webm).

Easy conversion & transfer was in the scope of Wikimedia Sverige’s 3D2Commons project − unsure of the current status but maybe worth checking out with them?

I found an online converter with links to offline converters, which may be helpful

http://52.4.31.236/convertmodel.html

I'd strongly suggest that effort is put into supporting .OBJ soon as this is by far the most commonly used 3D file format and is supported by most of the online 3D model sites, 3D models on Wikimedia is unlikely to be popular without it. Here is a helpful run down of popular 3D formats https://all3dp.com/3d-file-format-3d-files-3d-printer-3d-cad-vrml-stl-obj/

DivadH added a subscriber: DivadH.May 26 2018, 9:03 AM

Short version: +1 for glTF

Long vversion:
glTF is the (currently) only format the Sketchfab Download API can output

On Sketchfab there are about 100 000 CC-licensed models (see https://twitter.com/albn/status/986267640883707909 ). Some are likely NC and so out of scope for Commons. I know the cultural heritage content from GLAMs the best and certainly there's a lot of good content for it.

The tool JeanFred mentions that the Swedish National Heritage Board (where I work) and Wikimedia Sweden were aiming to develop had as its first idea to basically be for Sketchfab what Flickr2Commons is for Flickr. But when we started the project the Sketchfab Download API was not released so we changed direction. If glTF support is coming to Commons the original idea would be in play and the National Heritage Board could entirely fund or co-fund development, (depends on total cost of course) of such a Sketchfab2Commons style tool.