Maniphest T191802

[Epic] Determine a strategy to store files between 5 and 100 GB
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	Dereckson
	Apr 9 2018, 2:44 PM

Description

More and more, we're going to encounter requests to store very large videos in lossless format. Those videos will be huge, we already received requests for 16 GB and 18 GB files from @Goryeo.

Current Wikimedia infrastructure uses swift as a storage backend, but without support for large object.

MediaWiki isn't equipped either to send large objects to swift, as there is a need to divide the file in several parts < 5 GB each.

We need to determine a strategy at infrastructure and MediaWiki levels to store those large files.

References:

https://docs.openstack.org/swift/latest/overview_large_objects.html

Related Objects
Search...

Status	Assigned	Task
Resolved	Dereckson	T191572 Please upload large file to Wikimedia Commons
Open	None	T191802 [Epic] Determine a strategy to store files between 5 and 100 GB
Resolved	Bawolff	T191805 Allow Mediawiki to store file size greater than 32 bits
Resolved	aaron	T348294 FSFileBackend spends a lot of time generating uneeded sha1 hashes that are expensive for large files
Resolved	ABran-WMF	T348183 Apply schema change for changing img_size, oi_size, us_size, and fa_size to BIGINT
Open	None	T149847 RFC: Use content hash based image / thumb URLs

Event Timeline

Dereckson created this task.Apr 9 2018, 2:44 PM

Dereckson renamed this task from Determine if/how we can store file size greater than 32 bits to {Epic] Determine a strategy to store files between 5 and 100 Gb.Apr 9 2018, 2:59 PM

Dereckson removed Dereckson as the assignee of this task.

Dereckson updated the task description. (Show Details)

Dereckson added subscribers: aaron, fgiunchedi.

Dereckson added a subtask: T191805: Allow Mediawiki to store file size greater than 32 bits.Apr 9 2018, 3:15 PM

zhuyifei1999 subscribed.Apr 9 2018, 3:32 PM

Bawolff closed subtask T191805: Allow Mediawiki to store file size greater than 32 bits as Declined.Apr 9 2018, 4:24 PM

revi renamed this task from {Epic] Determine a strategy to store files between 5 and 100 Gb to [Epic] Determine a strategy to store files between 5 and 100 Gb.Apr 12 2018, 7:52 AM

revi subscribed.

I think it'd be good to do this. Commons is aimed to provide high quality educational multimedia. The more quality of our resources, the more useful they'll be. Thanks.

aaron added a subtask: T149847: RFC: Use content hash based image / thumb URLs.Apr 12 2018, 5:34 PM

• Mholloway subscribed.Apr 12 2018, 5:35 PM

This is related to T149847 in that we would *have* to stop moving file content around in Special:MovePage just to rename files.

Yann subscribed.Jun 7 2018, 4:28 PM

Urbanecm mentioned this in T202157: Please upload a large file to Wikimedia Commons.Aug 17 2018, 9:24 PM

Meisam subscribed.Aug 17 2018, 9:26 PM

Framawiki subscribed.Aug 19 2018, 8:57 AM

Krenair subscribed.May 20 2019, 12:21 AM

Urbanecm mentioned this in T224606: Upload 11 GB file "Bergensbanen full-length.webm" to Wikimedia Commons.May 29 2019, 6:46 PM

Urbanecm mentioned this in T219081: Please upload two ~15GB files to Wikimedia Commons.Jun 2 2019, 9:18 PM

GPSLeo subscribed.Jul 16 2019, 12:18 PM

With 4K video becoming more and more commonplace, 4GB isn't always enough. https://commons.wikimedia.org/wiki/File:Politparade.webm and https://commons.wikimedia.org/wiki/File:Genderwahn.webm are encoded to fit within MediaWiki's limit, which isn't a good sign. And 9 Mbps for 4K.. Well.. Could be worse, but.. not ideal. And they lack 1440p and 2160p transcodes, presumably due to the file size limit.

And that's one hour of video. Better don't make anything longer.. I recently uploaded https://commons.wikimedia.org/wiki/File:How_to_de-package_and_expose_a_GPU_flip_chip_die.webm from YouTube. 10 minutes, 1.07GB. So within 40 minutes this would go over 4GB. And 8K is even worse: https://commons.wikimedia.org/wiki/File:First_8K_Video_from_Space_-_Ultra_HD_VP9.webm (21.75 Mbps) and https://commons.wikimedia.org/wiki/File:Ghost_Towns_in_8K_GoPro_be_Hero.webm. (33.7 Mbps)

Bugreporter reopened subtask T191805: Allow Mediawiki to store file size greater than 32 bits as Open.Jan 14 2020, 11:24 PM

RhinosF1 subscribed.Mar 30 2021, 4:55 PM

Ltrlg subscribed.Apr 2 2021, 8:06 AM

Zabe subscribed.Apr 12 2021, 9:00 AM

Urbanecm mentioned this in T281520: Create Wikinewsie's Portal wiki.Apr 29 2021, 11:50 PM

Askeuhd subscribed.May 19 2022, 7:53 AM

TheresNoTime removed a subscriber: RhinosF1.Dec 15 2022, 11:36 PM

Frostly subscribed.Oct 2 2023, 2:52 AM

Bawolff mentioned this in T191805: Allow Mediawiki to store file size greater than 32 bits.Oct 4 2023, 1:10 PM

Bugreporter renamed this task from [Epic] Determine a strategy to store files between 5 and 100 Gb to [Epic] Determine a strategy to store files between 5 and 100 GB.Oct 15 2023, 3:04 PM

Bugreporter updated the task description. (Show Details)

Frostly added a project: Epic.Oct 15 2023, 5:03 PM

Bawolff closed subtask T191805: Allow Mediawiki to store file size greater than 32 bits as Resolved.Oct 27 2023, 10:07 AM

GeneralNotability subscribed.Oct 29 2023, 1:20 AM

CptViraj subscribed.Oct 29 2023, 2:35 AM

Personally, I think 5 GiB is plenty. Our purpose is education, not entertainment. We don't need 8K videos to explain how mitochondria work. 480p works fine. 1080p is probably overkill. And 4320p (8K) is just totally unnecessary. What use case is excluded by requiring that people downsize giant video files?

Being able to show freely licensed educational content on big screens is "not entertainment". You're still free to screen 480p in your cinema if your audience enjoys it, despite of a "totally unnecessary" 4320p version potentially also existing. :)

In T191802#9320060, @Nosferattus wrote:

Personally, I think 5 GiB is plenty. Our purpose is education, not entertainment. We don't need 8K videos to explain how mitochondria work. 480p works fine. 1080p is probably overkill. And 4320p (8K) is just totally unnecessary. What use case is excluded by requiring that people downsize giant video files?

My use case for publishing 8K (and in future 9K, 10K or 11K) videos is the possibility to make video captures and crops of these and still end ap with for example a portrait of Greta Thunberg in a reasonable resolution for a jpeg. Also I make videos of events that will may be only become interesting to historians in a hundred years, when i am long dead. I have this documents in 8K and i can now upload them in high quality, but this would be lost,, if i uploaded only 480p. But I am all for deleting this videos of mitochandria, should they ever be needed you can always make new one's, it is not as if mitochondria have changed a lot since cinema was invented 47 years ago.

add: TIFFs and PDFs (and sometimes even PNGs) are bigger than 4GiB too.

https://archive.org/details/Scotichronicon was extracted from 4K video. The text is readable but not particularly sharp. This is the only digital version of the Scotichronicon in existence as far as I know, and it exists by virtue of CGP Grey including a time-lapse in one of his videos where he shows himself turning the pages as he searches the book for a name. If the original video had been 8K120 there would be fewer missing pages and the text would be crisper. If it had been 1080p.. woof.

https://commons.wikimedia.org/wiki/File:Doria_Ragland_VOA.jpg was extracted from a 720p video. Clearly 480p would have been more than enough. https://commons.wikimedia.org/wiki/File:Bryn_Kenney_2015.jpg was extracted from a "probably overkill" 1080p source. I wouldn't call that overkill, and that's the best image we have of him.

Bugreporter2 awarded a token.Nov 10 2023, 1:54 PM

TheDJ mentioned this in T351023: Raise hard limit for transcode sizes to 4 GiB or higher.Nov 15 2023, 1:37 PM