Page MenuHomePhabricator

Add "Level of Detail" (LoD) implementation for 3D files
Open, MediumPublic

Description

Problem:
Large 3D files produce several problems

Background:
Similar to images or (even more common) video files an Level of Detail (LoD) would be really necessary for the 3D MediaViewer. As 3D files can get very large easily - there's already an nearly 10 MiB example on the beta cluster, up to 100 MiB is realistic for meshs including textures - it also might break the users browser if you try to load an 1 mio poly mesh using older hardware or an mobile device.
Some well known websites providing 3D media via WebGL are already serving LoD as option (including size info) right in their media viewer, similar as video viewers do.

Event Timeline

Aklapper renamed this task from Level of Detail (LoD) for 3D files to Add "Level of Detail" (LoD) implementation for 3D files.May 5 2017, 10:40 AM

@Reaper35 would you please share some URLs for the LoD options with STL files?

Depending on technical complexity, perhaps one option would be to advise the user about file size if it's going to be a really big file (CC @Nirzar for design guidance). Another might entail some sort of restriction around file size and whether a file is eligible for Media Viewer viewing from certain contexts.

@dr0ptp4kt: The LoD should and would be independent from the used filetype. The procedure would be the same as with images: The server needs to create an lower leveled/detailed, additional file. Sketchfab for example uses (or used) .obj files in that way.
By the way: Has somewhere been discussed which filetypes we are able to use and which not?

Note: Related to the file size in general: https://www.mediawiki.org/wiki/Topic:Tpxc759itxeijefe

@Reaper35, thanks - are there some particular URLs on Sketchfab with steps to demonstrate downscaling/LoD? I may be visiting the wrong files or having a user error in finding the functionality! :)

Regarding file types, there was some discussion on T3790: Allow uploading of 3D files to Wikimedia Commons. For first release (and probably for the foreseeable future), STL is the sole supported format.

@dr0ptp4kt: Oh I'm sorry, it seems that Sketchfab changed their UI. :-/
It's already up to ~1.5 years ago that I first engaged with this topic, I don't know when I last visited Sketchfab. In the past they served an option in their viewer showing several LoDs including file size and/or vertice/face count (can't remember exactly). Today they only left an "Texture" parameter with the only available option "HD" (or the only available for my, maybe my hardware is too old - if they are able to recognize this). Even not for files with 1 and 2 mio faces: https://sketchfab.com/models/00754d1a71b543d1ac6bb22a0f01b255 and https://sketchfab.com/models/6898d4c2514d4920960f2f85c2b162f9 for example.
(You might find more searching for 'photogrammetry' (-> https://sketchfab.com/search?q=photogrammetry&sort_by=-pertinence&type=models), my laptop tends to crash since a while on high load. ...)

I hope OBJ files including textures will be supported in future, as it seems to be an good choice as Sketchfab uses it.

Edit: But they have some other nice options too: lightning, wireframe and navigation options, and also an "Help" section with an 'Reset Camera View' button as mentioned somewhere else here.

In this example https://sketchfab.com/models/fa94d763e89d437e84c31d5110a3c6a4 you might see that the file is loaded step by step. That's not the same as the requested feature, but might be of interest anyway (more in technological way as in an way to use in Wikipedia I guess). As the original, download-able file has nearly 200 MiB in size, there must still be an server-side LoD functionality.

OT: The annotation feature would be really useful for Wikipedia, too and was already thinking of creating an task for this - even when it has lower priority yet.

Thanks, @Reaper35. I think what may be prudent at this point is for us to try to reach out to their team and see if they have any technical advice that would be applicable with our stack. I'll send a message.

@dr0ptp4kt: That's an nice idea! Hope you keep us up to date about the state. Thank you.
I guess you mean their LoD implementation, or you might also ask for their .obj sanitization?

@Reaper35 I spoke with Cedric at Sketchfab about the LoD stuff. I also followed up with the question about .obj sanitization by email. Do note that our design manager @Nirzar is returning from some time off (was actually on vacation when I at-mentioned earlier) and will need some time to get caught back up and to weigh in on the UX around LoD.

First, Cedric said one approach with .obj checks is verifying in a Docker instance that the file is valid (well formed) and erroring out otherwise. I believe with .obj a more over-riding concern is the higher likelihood of really big files (which would likely mean a greater need for attention to LoD). This said, @Gilles do you have a sense of whether the .obj file format is inherently problematic from a security point of view? Although the plan is to launch first with .stl support, it would be interesting to know whether .obj would be easily enough greenlit later on (assuming we could get over other hurdles like large textures and the need to LoD them).

Here were other notes from a face-to-face conversation and a little bit of follow up.

NOTES START

I think you've possibly explored parts of this already, so hopefully not too much duplicate detail.

  • Cedric noted that contexts of use and the characteristics of the file format is important in thinking about LoD. For example, with file formats prone to heavy texture usage you likely need to focus on LoD on texture. With STLs it's conceivable that LoD even on shapes and not textures may turn out to in fact need attention (although we'd learn along the way as uploads start rolling in). In practice, the textures on texture-heavy file formats (e.g., .obj) do turn out to be a significant drag on client performance.
  • This said, one approach may be, for example, LoD downsampling a file that has, say 500K triangles down to 100K triangles if it turns out it actually is a problem with the STLs.
  • Many, although not all, UAs that support WebGL well also have reasonable hardware. On desktop/laptop it's conceivable things will usually work well. On mobile web it may not be as clear cut.
  • We spoke about ways to potentially detect device capabilities. In essence, it usually comes down to user agent support. Sites like caniuse, http://webglreport.com/, and http://webglstats.com/ (the last one was having a loading problem when I last checked) can be helpful.
  • Supposing that LoD downsampling still is the primary option to ensure a smooth experience as a necessity (and there's no good workaround in the UX), http://www.openmesh.org/ may be worth exploring.

NOTES END

Following my speaking with Cedric, I noticed the the Three.js (component used in our 3D implementation) documentation at https://threejs.org/docs/index.html#manual/introduction/WebGL-compatibility-check suggests a simple way to check for WebGL compatibility. I'm unsure if we have this factored in yet, but this may be worth applying if not. @MarkTraceur @matthiasmullie @Gilles do you happen to know if in practice there's a bootstrap check for this compatibility?

Given that ResourceLoader in effect blacklists a number of browsers, generally taking JavaScript execution out of play on class C devices, and this snippet from the documentation might further narrow the list, I wonder if this might be an option to reduce the risk of overwhelming UAs on lower hardware devices. It doesn't address the notion of shrinking the download size of the full STL, so I wonder if there's a simple UX treatment about file size that might reduce the risk of really big downloads, particularly on metered cellular connections. There's also navigator.connection JavaScript stuff that may be a lighter weight way to infer connection parameters and avoid bugging people on otherwise fast and unmetered/high data cap connections for devices that are mobile and support the Network API; desktop UAs purportedly don't support it but those tend to not be as big of an issue; there are exceptions to every rule, naturally. Safari mobile apparently doesn't support this property, either, although iOS users are said to be less sensitive to bandwidth consumption (I've seen conflicting anecdotal evidence - nonetheless it's easy to detect the UA and warn accordingly).

Regarding mobile web: at the moment, this feature *is* currently constrained to desktop, although we do eventually want it to be running on mobile web (the so called "mdot" sites you get on a mobile phone). You may have noticed that as it is users can in fact switch to Desktop mode from a mobile device and this feature even works on modern browsers. The componentizing of Media Viewer so that it runs on mobile web (where there's a separate code base at the moment for media viewing, a vestige of the past) is a separate effort, one that may take several months or longer.

In looking around the OpenMesh documentation it does seem there's some documentation around file formats, so perhaps it's possible to import and re-export through its interfaces (although establishing new binaries on our servers is a task unto itself). See http://www.openmesh.org/Daily-Builds/Doc/a03978.html. I haven't fully read the documentation or tried to build, but wanted to note this in case we eventually pursue LoD and someone is following these notes.

The problem with .obj, or any format that supports textures, is that those textures often rely on external files. Obj in particular tends to use TARGA files, which have very poor compression ratio (it uses RLE). The main issue is that you're dealing with the complexity of external files. Which you can solve either by letting people upload a ZIP file containing everything required, or having a custom uploader where people upload the model first, then they're asked to provide the needed textures one by one, for example (think red links for each texture). In either case, since the final URL for files on Commons is going to be different than what's hardcoded inside the .obj or .mtl file, you'll need to parse those and replace mentions of .mtl and .tga files to point them to the right location. I think there might be precedent with SVG files that can have dependencies, but I can't remember if/how that was solved.

And that doesn't solve the problem of .tga files being huge. For that particular problem, you'll probably need to take things further and have the uploaded .mtl files point to JPG versions of the TGA files, and have the threejs obj loader be happy if it gets JPGs rather than TGAs. That requires writing thumbnailing code to render TGAs as JPGs, too.

As you can see, supporting any format with textures is a project of its own. You could however, support .obj without materials and textures (only loading the geometry). I don't think that would be more challenging that STL.

As for automatically generating different versions of the file with lower LoD, that sounds great and it would make sense to always load an initial version below a certain file size before loading the original, regardless of platform. Simply because even if you have a fast internet connection, you're still waiting for the whole thing to be downloaded before you get 3D.

As for the WebGL check, I don't think there is one right now, which means that the file does load on browsers that support our level of JS compatibility but might not render if WebGL is unavailable. ThreeJS supports a Canvas fallback in case WebGL is missing, might be worth having as a fallback. And when neither Canvas nor WebGL are available, ideally we don't want to load the 3D file at all.

@Gilles, very helpful update!

I'll open a separate task around capability detection.

@Nirzar is going to analyze potential work-around treatments that require less effort.

dr0ptp4kt triaged this task as Medium priority.Jun 13 2017, 7:23 PM

Quick note about glTF (T187844) regarding level of detail meshes and textures:

  • textures in glTF are compressed (PNG or JPEG) and can be embedded in the file (both text and binary versions, though binary is much more efficient)
  • in binary format, meshes and textures appear after the JSON-level scene description, so _in theory_, if the file lays out its resources in the proper order you can stream in multiple levels of detail by loading resources only when needed
    • that's not supported by three.js's current glTF loader, though

There is probably ongoing work from various folks on doing this 'right' in glTF... https://github.com/KhronosGroup/glTF/issues/1045 is probably of interest.

I would encourage the adoption of well tested open source libraries for 3D visualization that allow progressive streaming of 3D models.
3DHop would be a perfect solution.
http://vcg.isti.cnr.it/3dhop/demo.php
It supports compressed multiresolution streaming of 3D models, supports point cloud and it has already been used in a number of 3D archival projects, like http://zamaniproject.org/ , http://cabinetdecuriosites3d.mnhn.fr/fr/content/cro-magnon-1-homo-sapiens ,

and it would allow a definitely better experience of 3D.

Removing @Nirzar as assignee, as that account has been disabled.

https://highlightstory.com/accent-furniture-or-occasional-furniture-the-difference/
https://gambleinsights.com/how-do-you-play-the-letterle/
https://bankingdecision.com/everything-about-mykohlscard-com-login-portal/
https://lawordinance.com/all-you-need-to-know-about-the-high-court-of-karnataka-bangalore/
https://vvipproperty.com/halloween-and-some-amazing-jack-olantern-ideas/
https://vehiclesdetail.com/steering-wheel-cleaner-and-its-car-wash-services/
https://selfcareadvise.com/what-dinosaur-has-500-teeth-fully-explained/
https://solidbusinessidea.com/effortless-transitions-with-moving-companies-movingapt/
https://growwebtraffic.com/what-is-insta-stalker-unveiling-the-complications-and-impact/
https://holidaytakeoff.com/25th-island-of-greece/
https://techmarkettoday.com/
https://academicsession.com/
https://stylefashionguru.com/
https://technologyremark.com/
https://studyimprovement.com/
https://valuablestudy.com/
https://growacademics.com/
https://financingease.com/
https://salarysaving.com/
https://itmagnate.com/
https://keyheadlines.com/how-to-make-a-career-in-genetics-in-india/
https://multiplysaving.com/
https://pileupmoney.com/
https://citynewscast.com/
https://growupguys.com/
https://primenewstime.com/
https://topnotchstory.com/
https://techieresearch.com/
https://expertremark.com/
https://startupreviewer.com/
https://showingtrue.com/
https://deposittalk.com/
https://bankingcashier.com/
https://itsoftwaretips.com/
https://reliableapplication.com/write-for-us/
https://techsourcecode.com/marketing-technology-trends-for-2021/
https://educatedleader.com/
https://generaleducator.com/
https://entrepreneursprofit.com/
https://magazineheadlines.com/
https://readprimenews.com/
https://coverpagestory.com/
https://outfitstyling.com/
https://homefamilyfun.com/write-for-us/
https://homeinteriortip.com/
https://journeyupdates.com/where-to-stay-in-bali-with-family-the-answer-is-here/
https://stylefitnesstips.com/write-for-us/
https://youthstyletips.com/
https://latestreporting.com/
https://cityoutdoorfun.com/
https://stepouthome.com/smart-travel-tip-7-things-to-look-out-for-when-dealing-with-a-travel-agency/
https://nextfashiontrend.com/eco-friendly-footwear-going-green-with-footwear/
https://bankingschemes.com/
https://wellbeingstuff.com/
https://growlifeskills.com/do-you-know-how-to-be-an-engaging-and-highly-effective-educator/
https://mostrecenttech.com/
https://itbusinesstrend.com/write-for-us/
https://happylifeview.com/
https://fundayplans.com/a-few-great-sunday-funday-ideas-for-you-to-know/
https://wholefamilyfun.com/
https://managesalary.com/write-for-us/
https://expenseshelp.com/
https://fastnewstime.com/what-kind-of-applications-can-be-build-with-node-js/
https://topchangemakers.com/business-loan-oregon-go-ahead-and-boost-your-profit-by-expanding-business/
https://topbusinessicon.com/a-new-era-of-work-at-home-entrepreneurs/
https://friendfamilyfun.com/feed-your-pet-with-the-right-food-order-online-for-canidae-dog-food/
https://shoppingrightly.com/learning-street-style-fashion-with-fashion-blogs/
https://fullprooftips.com/5-reasons-why-fixed-deposit-is-the-best-investment/
https://lumpsumamount.com/
https://nursingability.com/
https://primenewsroom.com/
https://businessesgoals.com/7-ways-crm-helps-improve-customer-experience/
https://citybigstory.com/
https://techgadgettime.com/
https://newssofar.com/
https://nationnewstime.com/
https://dailyoutfitidea.com/
https://youthbeautytips.com/
https://latestnewscast.com/
https://presentdaynews.com/
https://minibignews.com/
https://newsroombroadcast.com/
https://softwareswatch.com/
https://techglobally.com/
https://primenewscast.com/
https://metrocityexpress.com/
https://gambletonight.com/
https://winjackpotprize.com/
https://safecasinofun.com/
https://shopproperly.com/top-products-for-the-best-eye-makeup/
https://necessaryshopping.com/style-trends-for-teenagers-best-style-advice/
https://slimfitnessguru.com/
https://gamblepassion.com/
https://youthcareertips.com/
https://shoppingthirst.com/
https://edgylifestyle.com/fashion-and-futurism-a-look-at-sci-fi-game-fashion/
https://reputablecar.com/2021/11/
https://cashcryptohelp.com/
https://getcryptotips.com/
https://vviphousing.com/advice-on-property-for-sale-and-property-to-rent-improve-your-property-investment/
https://healthdietguru.com/
https://uprighthabits.com/
https://classifieds.highlightstory.com/classifieds/browse-ads/101/churches-temples
https://fastitresult.com
https://pentionschemes.com
https://bankingreturn.com
https://pennyreturn.com
https://basicethics.com
https://tutorspick.com
https://casinoinsides.com
https://legalcomments.com
https://lawfulvoice.com
https://lawsbasics.com
https://lavishhousing.com
https://poshlocalities.com
https://vehiclesjourney.com
https://carroadtrip.com
https://businessesprofit.com
https://moodystyling.com
https://foodieperson.com
https://sellingtaste.com
https://legallysimpler.com
https://legitprocess.com
https://dancepartyfun.com
https://fullpartyfun.com
https://goodprofessions.com
https://bigadvices.com
https://campingstays.com
https://rightpetcare.com
https://childnecessity.com
https://jewellerymakeup.com
https://jewellerylooks.com
https://sportingspeak.com
https://careerindemand.com
https://electronicsspeak.com
https://appinstalling.com
https://techstartupinfo.com
https://softwarespackage.com
https://techtoolhelp.com
https://techaccessary.com
https://reputableproduct.com
https://uptomarkfashion.com
https://silksymbol.com
https://luxurymaintain.com
https://salariedbenefits.com
https://foodrichness.com
https://selfwelfare.com
https://setuppriority.com
https://joindreamjob.com
https://selectingcareer.com
https://schoolgrasp.com
https://schoolingsense.com
https://studentsyllabus.com
https://shopdailytips.com
https://shoppingsuperbly.com
https://drivebestcars.com
https://finestautomobile.com
https://vehiclefacelift.com
https://travellersthrill.com
https://safesolotrip.com
https://partiesoutdoor.com
https://explorecityfun.com
https://townactivities.com
https://stylinglooks.com
https://outfitsstyling.com
https://seniorswelfare.com
https://elderpeoplecare.com
https://mennecessity.com
https://boyessentials.com
https://girlsnecessity.com
https://femaleessential.com
https://schoolboycare.com
https://leadersquality.com
https://leadersplans.com
https://personstrength.com
https://achieveplans.com
https://writeupidea.com
https://bulletinstime.com
https://newspaperglance.com
https://readfullnews.com
https://businessesgain.com
https://salesnetprofit.com
https://cryptosgems.com
https://casinostricks.com
https://cbdquantity.com
https://cannabidioluses.com
https://cannabisworth.com
https://lifestylesymbol.com
https://biggerlifestyle.com
https://upkeepstyle.com
https://homesimpression.com
https://locationdecor.com
https://residencelook.com
https://petupbringing.com
https://endearingpet.com/
https://bigsmallpet.com/
https://healthsupervise.com/
https://fitnesspractise.com/
https://beautyappearance.com/
https://nursingskillset.com/
https://nursingworker.com/
https://familyjoytime.com/
https://familyfunplan.com/
https://lavishmarriage.com/
https://dentalsitting.com/
https://regularmarketing.com/
https://selfuplifting.com/
https://picssession.com/
https://parentbabycare.com/
https://travellingease.com/
https://kidsnecessity.com/