Page MenuHomePhabricator

Move large file metadata to external store
Open, Needs TriagePublic

Assigned To
None
Authored By
Ladsgroup
Feb 17 2025, 3:18 PM
Project Tags
Referenced Files
F58412134: Figure_1_log2.png
Feb 17 2025, 3:18 PM
F58412123: Figure_1.png
Feb 17 2025, 3:18 PM

Description

This was partially done for pdf and djvu before. But the metadata field still is quite a large portion (above 80%) of image (and filerevision table).

I built a graph of cumulative size of img_metadata order by size descending (and lumped every 1000 images to 1 to avoid graph from exploding).

Figure_1.png (1,920×1,001 px, 37 KB)
Normal
Figure_1_log2.png (1,920×1,001 px, 30 KB)
logscale

Based on that and first derivative (which means just comparing sizes), the threshold seems to be around 1400 bytes which only impacts 21% of the files but taking away ~40% of the data.

This should be done after T28741: Migrate file tables to a modern layout (image/oldimage; file/filerevision; add primary keys) is done.

Event Timeline

Bugreporter renamed this task from Move large img_metadata to external store to Move large file metadata to external store.EditedFeb 19 2025, 3:10 AM
Bugreporter subscribed.

Note filearchive table also has a column fa_metadata which is comparible with img_metadata, oi_metadata and fr_metadata and thus contains file metadata albeit in a smaller scale. This table is not yet touched in T28741 and the fate of the table may be decided in future T20493.

Marostegui moved this task from Ready to Blocked on the DBA board.