Page MenuHomePhabricator

Support the glTF 2.0 3D file format
Open, Needs TriagePublic

Description

This royalty-free standard format was enabled today on Facebook, which means it's likely to take off a bit (and can be exported from Blender).

The format is described at https://www.khronos.org/gltf/ supports textures and animations (which STL doesn't) and is designed to be efficient for transmission and client-side interpretation, which would work nicely for us.

Obviously this'd be a fair bit of work, not least the security concerns. Would be interesting to see what Facebook have done in this regard, and whether we could reuse their work if it's FLOSS.

It looks like most of what we need is in three.js, which we already use for the stl rendering, so rough plan for using the .glb binary variant would be:

  • add .glb and proper mime type to core's mime.info
  • add a validation hook for uploaded files to 3D (magic number, proper sections, valid json, no external uri references)
  • split ThreeDHandler into a base class, with stl and glb subclasses to do validation and extraction differently for each supports type
  • extract asset copyright and generator fields from the json as metadata when present, expose them in the table on File: page
  • include three.js's gltf loader in 3D and 3d2png's js modules
  • change thumbor config to pass glb through the 3d2png handler
  • add mmv plugin config for glb as well as stl
  • change varnish config to gzip the glb files on transfer

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

As far as I can tell from spec a binary file can still reference external resources if it wants...

That's correct; it's atypical but certainly possible. There's an official validator as well, which reports external resources along with many other issues, e.g. —

resources: [
  {pointer: "/buffers/0", mimeType: "application/gltf-buffer", storage: "external", uri: "scene.bin", byteLength: 2617976}
  {pointer: "/images/0", mimeType: "image/png", storage: "external", uri: "textures/lainmat_baseColor.png", image: {…}}
  {pointer: "/images/1", mimeType: "image/png", storage: "external", uri: "textures/lainmatAO_baseColor.png", image: {…}}
  {pointer: "/images/2", mimeType: "image/png", storage: "external", uri: "textures/lainmatshadeless_baseColor.png", image: {…}}
]

It might be best to reject uploads with any outright errors, as we can't guarantee those will work in future versions of three.js.

Did a quick peek at the GLTFLoader for three.js. Couple things would need to be changed upstream to work in Node for 3d2png:

  • it tries to use window.TextDecoder if available to decode the UTF-8 JSON blob; there's a fallback but it throws a runtime error before it gets to the fallback because window is undefined.
  • it uses the Blob constructor and URL.createObjectURL to create web-loadable URIs for textures out of the buffer views. There is no Blob constructor, and no URL, in the node environment. Need to change it to use some other kind of loader?
  • even if you had blob URLs, three's ImageLoader uses DOM stuff which won't work. Appears to be a pluggable system by regex, so easy to add support for a fake protocol like 'blob:'

Change 415495 had a related patch set uploaded (by Brion VIBBER; owner: Brion VIBBER):
[3d2png@master] [WIP] Tweaking 3d2png to support glTF binary (.glb)

https://gerrit.wikimedia.org/r/415495

Provisional patch to 3d2png doesn't render successfully yet, but does load the scene data. ;)

Now renders the "Avocado.glb" and "Boombox.glb" sample files successfully. :)

out.png (2,000×1,500 px, 298 KB)
boombox.png (2,000×1,500 px, 276 KB)

Had to adjust the near frustum on the camera from 1 to 0.01 since glTF files often seem to use meters for units, and our centering was putting us at a pretty close position within 1 meter. Also made a tweak to the node texture loading hack which makes textures actually work (use THREE.Texture, not THREE.DataTexture even though it sounds like what I want it doesn't work).

Shouldn't be super difficult to use the GLTFLoader in the web view once that frustum tweak is in (it won't need the rest of the node-specific loader tweaks, just a slight adjustment to centering to handle passing through Scene and Object3D objects to find the mesh).

Looking great! A few notes that may be helpful:

From spec, on coordinate systems:

  • glTF uses a right-handed coordinate system, that is, the cross product of +X and +Y yields +Z. glTF defines +Y as up. The front of a glTF asset faces +Z.
  • The units for all linear distances are meters.

And, although I haven't tested this in Node.js, CommonJS imports _should_ let you pull the specific versions associated with your three.js version this way:

const THREE = require('three');
require('three/examples/js/loaders/STLLoader');
require('three/examples/js/loaders/GLTFLoader');

var loader = new THREE.GLTFLoader();

This won't work with ES6 modules unfortunately. The community-maintained npm packages for individual loaders tend to be pretty out of date.

Hey I implemented gltf support in facebook so happy to jump in here however i can. I would definitely advise to start with a limited format set, ie just use glb, trying to convert everyone elses models into what you can understand is a daunting proposition.

You'll also need to figure out what extensions you want to support. At fb we support the KHR_materials_unlit extension as a lot of photogrammetry models use it, or models that want to bake really fancy lighting into the textures. Sketchfab seems to have a lot of SpecularGlossiness models which we try to convert to regular PBR but i wouldn't recommend it, at least to start.

Hi @Msfeldstein

Thanks for joining the conversation. I think we'll have a lot more questions for you soon :)

Hey I implemented gltf support in facebook so happy to jump in here however i can. I would definitely advise to start with a limited format set, ie just use glb, trying to convert everyone elses models into what you can understand is a daunting proposition.

You'll also need to figure out what extensions you want to support. At fb we support the KHR_materials_unlit extension as a lot of photogrammetry models use it, or models that want to bake really fancy lighting into the textures. Sketchfab seems to have a lot of SpecularGlossiness models which we try to convert to regular PBR but i wouldn't recommend it, at least to start.

@brion is it too late to chime in and offer Babylon.js support?

We have everything available out of the box including a 1st class glTF support: https://github.com/cx20/gltf-test
The minimal viewer size could be around 200K (gzipped) depending on options you want

You can try the sandbox here: https://sandbox.babylonjs.com to see how it will render

Also wanted to mention the Babylon.js viewer: http://doc.babylonjs.com/extensions/the_babylon_viewer which can really reduce development time

Hi,

I'd like to contribute our Darth Suzanne model to the development process.

(under CC0)
It was exported using Verge3D for Blender (a Three.js-based plug-in).
Here is how it looks online: https://cdn.soft8soft.com/demo/applications/pbr/pbr.html
The source files for this glb model are available in the Verge3D distro: https://www.soft8soft.com/get-verge3d/

Hi @SantaWinsAgain, thanks for your comment. This task is about making the MediaWiki software support that file format.
After that has happened you are more than welcome to donate such files to Wikimedia Commons. Thanks!

@Deltakosh babylonjs is something to consider too! One thing is we need a headless mode to render flat thumbnails as well as the live viewer... We've managed to get three.js running under node with headless WebGL, though sometimes it's a little funky to add new features to it. Is there a headless mode available for babylon as well? (In theory we could change from node to headless Chrome or Firefox but that's a whole other thing to arrange with ops, probably. :)

It is also working and we have several users already doing it.
We went to Firefox on our side for ops (running visual tests for incoming PR)

Feel free to ping me directly if you need help: Davca@microsoft.com

This is really great additional 3D model support is being worked on. Can I ask if there is much documentation around converting popular 3D file formats like .OBJ and .PLY to this format? I'd hate for people to not use this due to lack of good quality documentation for easy process to convert the files to this format (like what happened with audio and video before .webm).

In T187844#4224944, @Mrjohncummings wrote:

This is really great additional 3D model support is being worked on. Can I ask if there is much documentation around converting popular 3D file formats like .OBJ and .PLY to this format? I'd hate for people to not use this due to lack of good quality documentation for easy process to convert the files to this format (like what happened with audio and video before .webm).

Easy conversion & transfer was in the scope of Wikimedia Sverige’s 3D2Commons project − unsure of the current status but maybe worth checking out with them?

I found an online converter with links to offline converters, which may be helpful

http://52.4.31.236/convertmodel.html

I'd strongly suggest that effort is put into supporting .OBJ soon as this is by far the most commonly used 3D file format and is supported by most of the online 3D model sites, 3D models on Wikimedia is unlikely to be popular without it. Here is a helpful run down of popular 3D formats https://all3dp.com/3d-file-format-3d-files-3d-printer-3d-cad-vrml-stl-obj/

Short version: +1 for glTF

Long vversion:
glTF is the (currently) only format the Sketchfab Download API can output

On Sketchfab there are about 100 000 CC-licensed models (see https://twitter.com/albn/status/986267640883707909 ). Some are likely NC and so out of scope for Commons. I know the cultural heritage content from GLAMs the best and certainly there's a lot of good content for it.

The tool JeanFred mentions that the Swedish National Heritage Board (where I work) and Wikimedia Sweden were aiming to develop had as its first idea to basically be for Sketchfab what Flickr2Commons is for Flickr. But when we started the project the Sketchfab Download API was not released so we changed direction. If glTF support is coming to Commons the original idea would be in play and the National Heritage Board could entirely fund or co-fund development, (depends on total cost of course) of such a Sketchfab2Commons style tool.

May I reanimate this request? Gltf/GLB have become a kind of standard for 3D models in the last years. WikiMedia supports STL-Files for a while now. But STL is only for 3D-Printing and includes no colours, no textures. The viewer https://www.mediawiki.org/wiki/Extension:3DAlloy is based on ThreeJS and so could also render GLB.

I see at last two main use cases:

  • Showing models in WikiPedia pages. I would be great, to see the Cydney Opera or a Police Box(Tardis) textured.
  • Placing the models on a map/OpenStreetMap. A reference in the OSM tagging enables map viewer to load and show the Opera or the Tardis at the right places.
bvibber subscribed.

This got brought up recently and I think it's a reasonably self-contained and not huge work blob. :) Adding myself to do a spike test bringing branch up to date and either making it work or deciding we should redo it differently with more resources.

Change #1026883 had a related patch set uploaded (by Bvibber; author: Bvibber):

[mediawiki/extensions/3D@master] WIP update of three.js to r164 and add gltf loader

https://gerrit.wikimedia.org/r/1026883

Thank you @bvibber!

no lighting? needs fixing obviously

I don't think I can comment on the changeset, so chiming in here. three.js made some significant changes to the lighting system in r155. There's a migration guide for lighting. That said, if you're after good general-purpose lighting for unknown input files, I would strongly recommend THREE.RoomEnvironment, as shown in this example.

RoomEnvironment is relatively foolproof to set up, and will work much better for physically-based rendering (PBR) materials — as frequently found in the glTF format — than point/spot/directional/ambient lighting. It's a very lightweight implementation of image-based lighting (IBL) and you'll find it used elsewhere in viewers like https://modelviewer.dev/ and https://gltf.report/.

RoomEnvironment is relatively foolproof to set up, and will work much better for physically-based rendering (PBR) materials — as frequently found in the glTF format — than point/spot/directional/ambient lighting.

eeeeexcellent, that'll be the target of my hacking tonight & tomorrow to a) restore lighting and b) get gtlf working :D

Sketchfab has announced it will be closing/migrating to Epic's new 'Fab' marketplace by 2025..

Mid-October
Fab will launch! The Sketchfab Store will close; models that had previously been available for purchase will still be viewable on Sketchfab but will no longer be available for purchase there. If you have purchased content from the >Sketchfab Store, you will continue to have access to it through the Purchases tab of your Sketchfab profile page.

If you migrated CC-BY or Standard licensable models to Fab, those Sketchfab model pages will remain available to view on Sketchfab but will offer a link out to Fab for users who might wish to download or purchase your models. >Models that were available on Sketchfab under other license types, e.g., Editorial or CC-BY-SA, will remain available to view on Sketchfab but will not provide a link to Fab on the model page.

Future roadmap
In 2025, we plan to stop offering downloadable content on Sketchfab. All creators who have not already moved their CC-BY licensable models to Fab will be invited to do so in advance of this change. We plan to ensure that >existing integrations that use Sketchfab’s Download API to access the free content library will continue to work until new APIs are made available on Fab. Also in 2025, Sketchfab’s enterprise solutions will transition to Fab.

it'd be good to have gltf support in place on commons by the time this happens.

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/3D/+/1026883 was verified, does this mean the updated viewer can be tested somewhere?

No, it means that the code passed an automated validation check. Anyone is free to set up a local MediaWiki instance, install the 3D extension, and test code changes.

Hi @bvibber, are you currently working on the extension? Otherwise I would give it a try. It would be great if you could take a look at my comment on 1026883.

I have created a prototype for a viewer that also supports GLB files in the ‘mediawiki’ style. Only simple GLB files and KTX2 compression are supported. I am not yet satisfied with the positioning of the models, especially in view of the thumbnails. Help and feedback are very welcome.

You could choose from different free STL and GLB models, or upload your own data. Is KTX2 compression required?

Github Page: https://opendem.github.io/WikiMediaExtension3D_Test_GLB_Format/
Github: https://github.com/OpenDEM/WikiMediaExtension3D_Test_GLB_Format

It can take a while for the larger models to load, no ProgressBar is implemented here.

I have created a prototype for a viewer that also supports GLB files in the ‘mediawiki’ style. Only simple GLB files and KTX2 compression are supported. I am not yet satisfied with the positioning of the models, especially in view of the thumbnails. Help and feedback are very welcome.

Great! I would be happy if the mediawiki enables GLB to.
My test today:
https://github.com/OpenDEM/WikiMediaExtension3D_Test_GLB_Format/issues/2

As I have seen this was already looked promising in tested. What is a roadblock or any issue that this feature was not include in mediawiki (and supported in the whole wiki project) as of of yet?

I would recommend to the uploader to use the original scale as international unit: 1 should be 1 meter/metre

Looking great! A few notes that may be helpful:

From spec, on coordinate systems:

  • glTF uses a right-handed coordinate system, that is, the cross product of +X and +Y yields +Z. glTF defines +Y as up. The front of a glTF asset faces +Z.
  • The units for all linear distances are meters.

And, although I haven't tested this in Node.js, CommonJS imports _should_ let you pull the specific versions associated with your three.js version this way:

const THREE = require('three');
require('three/examples/js/loaders/STLLoader');
require('three/examples/js/loaders/GLTFLoader');

var loader = new THREE.GLTFLoader();

This won't work with ES6 modules unfortunately. The community-maintained npm packages for individual loaders tend to be pretty out of date.

Another hello here...I do a lot of work with the Khronos group (keepers of glTF) and can hopefully help. We view the ability to natively display 3D (via glTF) as quite important and a big win! I'm technically clueless, (not totally) but can put the right people together ...and I've been doing this web 3D stuff for ages. New to phabricator as well so be kind..feel free to ping me.

Hi @Sressler, Nice to have you with us. We are documenting the current status of the work here: https://commons.wikimedia.org/wiki/Commons:Textured_3D
There is also a Telegram Group.
There will also be a video conference on Sunday 23.02.2025 at 16:00 UTC.

sent you an email request for the video link...thanks in advance

I send you an email at 11:59, and now again. If the mail has not arrived please let me know.

I got the email..but I don't see any link to the jitsi call?
thanks..Sandy

Change #1308736 had a related patch set uploaded (by Amdrel; author: Amdrel):

[mediawiki/extensions/3D@master] [WIP] Add glTF support

https://gerrit.wikimedia.org/r/1308736

Change #1308746 had a related patch set uploaded (by Amdrel; author: Amdrel):

[3d2png@master] [WIP] Add glTF thumbnail support

https://gerrit.wikimedia.org/r/1308746

Change #1310187 had a related patch set uploaded (by Amdrel; author: Amdrel):

[mediawiki/core@master] Improve application/json mimetype for glTF files

https://gerrit.wikimedia.org/r/1310187

Change #1310188 had a related patch set uploaded (by Amdrel; author: Amdrel):

[mediawiki/extensions/3D@master] [WIP] Extract and show metadata for glTF files

https://gerrit.wikimedia.org/r/1310188

Change #1310187 merged by jenkins-bot:

[mediawiki/core@master] Improve application/json mimetype for glTF files

https://gerrit.wikimedia.org/r/1310187

Change #1311133 had a related patch set uploaded (by Amdrel; author: Amdrel):

[operations/software/thumbor-plugins@master] Add glTF engine and tests

https://gerrit.wikimedia.org/r/1311133

Change #1311134 had a related patch set uploaded (by Amdrel; author: Amdrel):

[operations/deployment-charts@master] thumbor: Add gltf engine and value

https://gerrit.wikimedia.org/r/1311134

Change #1311139 had a related patch set uploaded (by Amdrel; author: Amdrel):

[operations/puppet@production] cache: gzip glb files on transfer (model/gltf-binary)

https://gerrit.wikimedia.org/r/1311139

I've submitted patches that address the acceptance criteria of this ticket. The environment ones I have marked as WIP, but the extension and thumbnailer patches are marked as ready for review: https://gerrit.wikimedia.org/r/q/topic:%22T187844%22

Thank you very much for your work @Amdrel!
I have successfully tested the following Khronos Examples(https://github.com/KhronosGroup/glTF-Sample-Models/tree/main/2.0):

  • human_mouth.stl
  • Menger_sopnge_2stl
  • Spericon.stl
  • Higt_quality_skull.stl
  • Avocado.glb
  • Duck.glb
  • DamagedHelmet.glb
  • coffeemat.glb(KTX compression)
  • SimpleMesh.gltf (embedded)
  • RiggedSimple.gltf (embedded)
  • Duck.gltf (embedded)

Support for animations would, of course, be great. It shouldn't be too much work; see https://github.com/OpenDEM/WikiMediaExtension3D_Test_GLB_Format/tree/main.

Support for animations would, of course, be great. It shouldn't be too much work; see https://github.com/OpenDEM/WikiMediaExtension3D_Test_GLB_Format/tree/main

We can track this in a follow-up ticket perhaps. I left it out for now because it introduces a design dependency if you want users to be able to play/pause/select specific animations. I did see that it was present in your implementation.

CC: @mikez-WMF

We can track this in a follow-up ticket perhaps. I left it out for now because it introduces a design dependency if you want users to be able to play/pause/select specific animations. I did see that it was present in your implementation.

Thanks

Sounds good to me! When the 3D wish work wraps up, we can also highlight related tickets and ideas for more visibility.

@Amdrel @mikez-WMF Is there any way I can help out here to move the project forward?

@OpenDEM
Thank you for pinging us! This is still very much a top priority in the wishlist space - it has many dependencies for reviews, which @Amdrel is working through. These just take time and iteration since this isn't a space that many reviewers may be working on regularly, so it has to compete with their other commitments. We are very thankful for all who have been helping the wish become a reality this year, and look forward to its completion.
In case there's something actionable for others to help with, we will keep you and others in mind to help get things done. :)