Page MenuHomePhabricator

Ability to upload and render fonts in Wikimedia wikis
Open, Needs TriagePublicFeature

Assigned To
None
Authored By
Ebrahim
Jun 15 2024, 6:35 PM
Referenced Files
F55498420: image.png
Jun 20 2024, 4:25 PM
F55498416: image.png
Jun 20 2024, 4:25 PM
F55498411: image.png
Jun 20 2024, 4:25 PM
F55490749: image.png
Jun 20 2024, 10:04 AM
Tokens
"Love" token, awarded by Huji."Like" token, awarded by Izno."Love" token, awarded by Bawolff.

Description

Feature summary:

It will be fantastic if Wikimedia Commons (and maybe local wikis also) can be used as a repository of open licensed fonts in .ttf/.otf format. Both as webfonts for the use of individual wikis and as media file rendered and shown as .png/.svg files on wiki pages demand along the text.

Use case(s):

Benefits:

Back in 2011 where T20692 and T27220 are closed we didn't have support for .stl 3d objects either also but I think this type of request can feel more familiar now.

I was thinking about this for sometime and even created an personal MediaWiki extension to test the idea, technical wise this isn't more complicated than STL for example as not much is needed in frontend JavaScript stuff (at least from the beginning) but the backend rasterizer can use the following for a MediaWiki's ImageHandler (creating image thumbnail of a font, very similar to the way .svg, .tiff and .pdf are handled):

  • pango-view to use as the rasterizer (already installed at least in tools-login.wmflabs servers) with this https://mail.gnome.org/archives/gtk-i18n-list/2010-June/msg00000.html it can render only using an individual font. You can already invoke FONTCONFIG_PATH=. pango-view --text 'test' --no-display --output out.png (or .svg) in wmflabs servers.
  • fc-query (also already installed on wmflabs so shouldn't be unfamiliar, a use like fc-query <.ttf file> --format=%{fullname}) to retrieve metadata of each individual font to be shown in Common's metadata section
  • Optional: https://github.com/khaledhosny/ots for sanitation of file being uploaded (used in Chrome and Firefox also). Decades ago fonts used to trig weakness of non hardened software, nowadays with existence extensively fuzzed software libraries that isn't a concern. We already have .ttf fonts embedded in PDF files which are allowed in Commons and this can be considered as a non wrapped in PDF version of that.

License wise https://commons.wikimedia.org/wiki/Template:OFL and https://commons.wikimedia.org/wiki/Template:Apache are most popular free fonts licenses already accepted in Wikimedia Commons.

We perhaps can start just with .otf/.ttf file format and provide .woff and woff2 on the fly sometime later the same way we transcode video files somewhere more temporal.

We already have hacky ways to upload fonts, both to embed them in PDF files and upload PDFs in Commons which is already possible, as PDF files actually embed fonts in themselves, and to encode them into base64 text and put them in interface CSS files but the proposal here is to have fonts in more proper way so we can organize them using established tools we already have here such as categories and galleries. There are even software that inspect individual fonts like [https://github.com/khaledhosny/ots OTS], used by both Chrome and Firefox to make sure individual fonts can't impose any danger for end users, on any case this isn't worse than JS scripting capabilities of SVG which we've managed to overcome over the years.

Fonts also provide metadata akin to images EXIF in themselves, it's called the naming table https://learn.microsoft.com/en-us/typography/opentype/spec/name and they usually provide information like the original name of the font, author, date of creation and the copyright, among many other font related strings, and this can be parsed using software like fontconfig (the fc-query command), already installed in Wikimedia servers.

And even better than a .png output, a SVG output can be generated pango-view --text aksljdkalsjdlkasjd --no-display --output a.svg --font="Sans 2000" && svgo -p 0 a.svg (instead of svgo a small script can remove mantissa part of paths which because of the applied scale don't matter anymore), similar to the way .svg is provided by math extension but we can default to that here.

Or to put it differently, this is a request for a different <math>, minus math layout but with custom fonts. And multifont is out of scope here as many uses will be single font or even single character like https://en.wiktionary.org/wiki/Appendix:Unicode/Ancient_Greek_Musical_Notation but maybe we can address multifont drawing in a separate API where we have a drawing API I suggested in 2015 https://meta.wikimedia.org/wiki/Community_Wishlist_Survey_2015/Miscellaneous#Collaborative_way_to_generate_SVG/PNG_graphs_using_Lua_modules T66460 using fonts uploaded this way first.

Event Timeline

Ebrahim updated the task description. (Show Details)

Allowing local wiki to use free web fonts where they need, maybe site wide (which may is covered by ULS

AFAIU this exists, see https://www.mediawiki.org/wiki/Universal_Language_Selector/WebFonts

AFAIU this exists, see

Sure, thanks for mentioning it. There are indeed some overlaps with ULS just on webfont part and that's why I even myself mentioned ULS also. Originally ULS was intended to provide fonts for wikis otherwise would be unreadable but now we see some miscellaneous font like RailwaySans, LinuxLibertine, UnifrakturMaguntia and OpenDyslexic (as listed in https://www.mediawiki.org/wiki/Universal_Language_Selector/WebFonts/WebFonts_Status ) are also provided. Perhaps if we had something similar to the proposed extension, we could let ULS developers to reach to their main focus on the issue they are to tackle, making wikis being completely unreadable without the fonts readable and remove those fonts when not needed and keep other less used fonts like RailwaySans to be managed by the community.

In Persian Wikipedia we even have a number of base64 encoded free fonts embedded in CSS as gadgets for users to enable from the preference. Our wiki was indeed readable without those fonts but some users like different fonts for their workflow and I'm afraid proposing addition of all that free fonts to ULS may overwhelm MediaWiki's developers while this can be done much more efficiently in the community level and more effiecntly if we had the purposed feature, somehow like introduction of Scribunto/Lua to reduce the need for MediaWiki developer themselves to handle complicated logics in MediaWiki and let the community develop them themselves not by inefficiently combining simple magic words as some of our template were doing before Lua or by base64 font in CSS, suboptimal solutions, and please consider that web font was only one part of this idea, those number of dumped glyphs and Emojis on Commons for example are hard to be improved with ULS and anything it can provide.

Ebrahim updated the task description. (Show Details)
Jdforrester-WMF subscribed.

This proposal is interesting, but potentially a very big piece of work; I'll tag in the ULS extension in particular and the Language team in general for their view, as it is mostly their world of product/community support.

Ebrahim updated the task description. (Show Details)

Here is a very very initial proof of concept of a MediaWiki extension I suggest,

https://github.com/ebraminio/FontHandler

image.png (1×1 px, 198 KB)

It doesn't even render arbitrary text, the point of proposed feature, [[File:Font.ttf]] syntax works and I hope I can make it work like [[File:Font.ttf|text=something]]

If I understand correctly rendering part should be moved to thumbor (or a separate service, like mathoid?) and metadata generation to shellbox.

(and sorry for the repeated edits of task description, hopefully it wouldn't be needed anymore)

Now the custom text logic seems to work more or less on the extension https://github.com/ebraminio/FontHandler which hopefully can show better what I'm suggesting here,

[[File:NotoSansOldPersian-Regular.ttf|text=𐎡𐎢𐎣𐎤𐎥𐎦𐎧𐎨𐎩]]

image.png (512×1 px, 58 KB)

[[File:NotoSansEgyptianHieroglyphs-Regular.ttf|text=𓉐𓁶𓆓𓌓𓃾]]

image.png (556×1 px, 50 KB)

[[File:NotoEmoji-VariableFont wght.ttf|text=😊]]

(wished the render was colorful and guess it will be colorful on the final version using pango's python API on thumbor hopefully)

image.png (502×880 px, 35 KB)

Sorry for the repeated messages, hopefully this is the last of them for now.

Above I've shown how I purpose the syntax for the extension, e.g. [[File:NotoEmoji-VariableFont wght.ttf|text=😊]] but aside text= there can be other parameters to have a more complete work which most of them aren't even hooked in the current state of the prototyped extension https://github.com/ebraminio/FontHandler but I'm bringing it up here in hope we can see better what will be needed eventually,

[
  // Text this font will be rendered with
  'fonthandler_text' => 'text',
  // Like SVG, exact same text can be drawn different for differently languages of the same script
  'img_lang' => 'lang',
  // Possible values: ltr/rtl/ttb/btt, default is auto detect
  'fonthandler_dir' => 'dir',
  // Similar to font-feature-settings of CSS, fonts can have parameter to tweak the rendering
  'fonthandler_features' => 'features',
  // Similar to font-variation-settings of CSS, variable fonts have a mechanism to tweak variable axes
  // For example a font with variable weight can be turned into Regular, Bold, Thin upon user request
  'fonthandler_variations' => 'variations',
]

An additional parameter like unicodes=U+0054,U+0068,U+0065,U+0020,U+0071 can be imagined for passing individual unicodes instead of text but perhaps would be too low level and overkill.

And maybe ULS can cooperate with list of fonts defined in a MediaWiki: page to show them in dropdown of a specific wiki, moving addition and update of fonts to community management instead of developers.

Thinking more I believe we can also consider possibility of allowing .ttf/.otf without any preview or use as a webfont as the first step, considering https://commons.wikimedia.org/wiki/Category:Noto_fonts it's already obvious what it can be useful for, archiving source fonts in addition to dumped glyph of each set and with the vision to reverse that and only keep the source font remove all the glyph dumps (or no longer uploading newer glyphs dumps).

Or even let's see what a non-wmf MediaWiki would be better to have for .ttf/.otf files upload support, say for example this non-wmf wiki apparently has enabled .ttf enabled, https://www.thpatch.net/wiki/File:Cirno.ttf what we can offer from MediaWiki side (sanitization or other concerns you have) to make such an upload for a non-wmf instance of MediaWiki to suit better for such upload outside wmf.

Somehow I like contribute my time to make this happen, specially code and software wise, even without any mandate from you to do the actual enabling in wmf side at the end, but I don't know what is the needed steps.

Allowing local wiki to use free web fonts where they need, maybe site wide (which may is covered by ULS

AFAIU this exists, see https://www.mediawiki.org/wiki/Universal_Language_Selector/WebFonts

The people maintaining that extension have stated (e.g. here: T166138#7220936) that they don't want to become a generic repository for webfonts though, and there are quite a few requests for webfonts that have been stalled for years because we don't have anywhere else to put them. Even requests which are clearly within the scope of ULS (like T350550) take a long time to be implemented.

People have tried to solve issues like these by using Google Fonts, but were told they're not allowed to load third-party resources. People have tried to solve that problem by using Toolforge (e.g. T166138#7223384), but were told that's still a third-party resource.
People have suggested using Commons, but were told that depends on Commons approving it (T166138#7223790).

We really need somewhere we can put useful fonts easily, whether permanently (because they're out of scope for ULS), or temporarily (until they get added to ULS). If Commons doesn't want fonts to be uploaded there, then I would suggest creating a small separate wiki (something like "webfonts.wikimedia.org") with upload limited to a group of trusted users, so that licenses can be verified while still letting it be managed on-wiki by the community (and if something like that happens, I would be willing to help).

@Nikki Thanks for the very helpful context, based on that I've created a RfC to see what Commons community also think about it. https://commons.wikimedia.org/wiki/Commons:Requests_for_comment/Hosting_of_free_fonts_in_Commons

Ebrahim updated the task description. (Show Details)