Page MenuHomePhabricator

Allow music scores to be uploaded to Wikimedia Commons
Open, Needs TriagePublic

Description

An RfC was held in November 2018, demonstrating consensus for allowing multiple music notation file types on Wikimedia Commons.

Allowing music notation files to be uploaded to Commons would make it easier to share public-domain sheet music and would enable users to more easily edit each other's scores. (Currently, scores are either uploaded to Commons as images or used directly in pages through the Score extension, with benefits and disadvantages for each approach.)

Files would be rendered either by LilyPond through the Score extension (T201637), or through new extensions based on other open-source software (primarily MuseScore and Verovio). In addition, it may also be desirable to allow the score files to be rendered as audio using the normal file syntax. However, there is no requirement to enable thumbnails or playback before it becomes possible to upload files to Commons, and enabling thumbnails would not be very useful for Braille music anyway.


File types:

  • .mei – MEI
  • .ly – LilyPond
  • .abc – ABC
  • .mscz/.mscx – MuseScore
  • .mxl/.musicxml – MusicXML
  • .brf – Braille

Checklist (overall)

  • Check if no one has beat you to it (Score extension exists, no support for other file types)
  • make sure the file type is one of the filetypes allowed for uploading (all of them are neither whitelisted nor blacklisted)
  • Determine whether thumbnail generation is necessary before allowing uploads
  • Do you need the file format to be supported on WMF websites? (yes)

Support for uploads

  • Add support for the MIME types
  • Add the file types to $wgFileExtensions

Thumbnailing and embedding

  • Determine if you need support in MediaWiki core
  • Create MediaHandler subclasses for filetypes
  • Register your MediaHandler

WMF sites checklist

  • Free and open
  • The formats should not allow execution of code (not completely sure that all of them definitely can't)
  • Security review of new extensions
  • Thumbnail format should be commonly supported (PNG definitely possible for LilyPond/ABC/MuseScore/MusicXML, PDF possible for all except Braille)
  • Thumbnail (esp. when interactive) should be usable and accessible to screen reader users(?)
  • Interactive content should not reflow the page
  • Interactive content should perform (JS needs to be added only to the pages that require it, etc.)
  • If files exceed $wgMaxImageArea, special support is needed
  • thumbor engine
  • Community RFC

Event Timeline

Jc86035 created this task.Nov 1 2018, 12:32 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 1 2018, 12:32 PM
Jc86035 updated the task description. (Show Details)Nov 1 2018, 12:35 PM
Jc86035 updated the task description. (Show Details)Nov 1 2018, 12:37 PM

I have begun an RfC on Commons for this.

The RfC has ended without a formal closure, and is now archived. There was majority support for all six file types that were options by the end of the RfC, although five users opposed allowing MuseScore uploads mainly due to issues with the file format.

I asked for assistance on how to proceed at the administrators' noticeboard, but no one has responded. If that gets archived then I'll probably just make a commit to mediawiki-config to add all of the file types and see what happens.

Legoktm added subscribers: Ebe123, TheDJ, Legoktm.

The RfC has ended without a formal closure, and is now archived. There was majority support for all six file types that were options by the end of the RfC, although five users opposed allowing MuseScore uploads mainly due to issues with the file format.

I'm not sure adding 5 different file types for the same type of content is the best idea, but okay...

I asked for assistance on how to proceed at the administrators' noticeboard, but no one has responded. If that gets archived then I'll probably just make a commit to mediawiki-config to add all of the file types and see what happens.

Someone will probably need to add support for each file type in MediaWiki first, possibly in the Score extension. @Ebe123 or @TheDJ might be able to advise on how to proceed.

Also, given that these formats all seem to be text based, is there a reason we should be treating them as files, and not as editable wiki pages?

TheDJ added a comment.EditedJan 16 2019, 9:13 PM

A while ago I wrote https://www.mediawiki.org/wiki/Manual:Adding_support_for_new_filetypes specifically so that I dont have to repeat all this every single time someone asks for a fileformat ;)

For communication/tracking, I suggest filing separate subtasks for each specific format. Each ticket can answer all the questions in a list in the description of the task.

And then find developers. Which is probably (as always) the biggest problem to solve.

TheDJ added a comment.Jan 16 2019, 9:17 PM

Also, given that these formats all seem to be text based, is there a reason we should be treating them as files, and not as editable wiki pages?

As far as I know, there is no support in any part of the media engine to use revisions table instead of the image table.

Jc86035 added a comment.EditedJan 17 2019, 6:29 AM

I'm not sure adding 5 different file types for the same type of content is the best idea, but okay...

This is mainly because all of the formats are slightly incompatible with each other when importing/exporting, and most users are unlikely to prefer using more than one. We already host both OGGs and MP3s, as well as both TIFFs and PNGs.

Someone will probably need to add support for each file type in MediaWiki first, possibly in the Score extension.

Not necessarily. To my knowledge MP3s couldn’t be played for years until support was added in the Score extension, so it’s not a deal breaker (for me, at least) if it’s not possible to render or play any of the files.

text based

SVG is XML, but SVG files can still be uploaded to Commons. I also doubt that it would be possible to textually edit compressed MuseScore/MusicXML files.

I've removed site-requests tag until this is completely actionable.

Jc86035 updated the task description. (Show Details)Jan 17 2019, 12:40 PM
Jc86035 updated the task description. (Show Details)Jan 17 2019, 12:51 PM
Jc86035 updated the task description. (Show Details)
Jc86035 updated the task description. (Show Details)
Jc86035 updated the task description. (Show Details)Jan 17 2019, 2:05 PM
Jc86035 added a comment.EditedJan 17 2019, 2:07 PM

@TheDJ, I've added a checklist at the top. I am not aware of anything that isn't already posted here and in the RfC, so it should be possible for others to add checklists for the subtasks if desired.

The interactive concerns would probably only be relevant if anything is implemented with Verovio; I anticipate using MuseScore's existing export functions would be much easier than adding interactive playback for it.

What does "Your thumbnail (esp. when interactive) should be usable [...] for screenreader users" mean?

Jc86035 updated the task description. (Show Details)Jan 17 2019, 2:21 PM
Jc86035 updated the task description. (Show Details)Jan 17 2019, 2:24 PM
4nn1l2 added a subscriber: 4nn1l2.Jan 17 2019, 2:44 PM
TheDJ added a subscriber: Jc3s5h.Jan 18 2019, 9:11 AM

@Jc3s5h thank you for taking the steps, it is very valuable. As you note, adding thumb nailing support is a lot more steps than a simple upload ;)

"Your thumbnail (esp. when interactive) should be usable [...] for screenreader users" mean?

It means that anything you render in a wiki page should be accessible and understandable for screenreader users. At the very minimum that it shouldn't be obtrusive, but especially when you have web players etc.

...Wait, you were replying to me, right?

TheDJ added a comment.Jan 18 2019, 9:33 AM

sigh auto complete ;)

Jc86035 added a comment.EditedJan 18 2019, 9:44 AM

Anyway. Thanks for clarifying that.

On enwiki yesterday I notified HLHJ on their talk page about this, since they don't have a Phabricator account. As noted there, I'm not actually sure whether or not it's necessary to enable thumbnailing before enabling uploads. I don't think it is, since (as I noted in the RfC) MIDI still can't be played through the normal file syntax and has to be passed through an empty <score/>, giving precedent for enabling uploads (2005?) before enabling playback (after 2011). There was also no expectation that thumbnailing would certainly be possible, as far as I'm aware.

It may be desirable to integrate musical notation files into MediaWiki, most likely by allowing their display as thumbnails and possibly allowing audio playback from them. In November I made a Community Wishlist Survey proposal to propose the technical development of such capability. Even though the proposal has not succeeded, it may still be advantageous to host these files on Commons without using them directly in other projects.

Furthermore, allowing uploads first would not necessarily hinder display in pages, since LilyPond and MuseScore can export to PNG/PDF/OGG, and those could be used in lieu of working thumbnails. Once there is sufficient support then those could be replaced with something like [[File:LilyPond.ly|thumb|300px|page=1|vorbis=1]].

HLHJ added a subscriber: HLHJ.Jan 20 2019, 3:15 AM

I take it that by "thumbnail", we mean visual display, and "playback" aural display.

I participated in the RfC discussion. I think there were a few unresolved issues there, and I'm sorry it lapsed; my fault too.

One big one: Plaine & Easie Code is used as a standard in library catalogues. There are over a million music incipits available under CC-BY in this format, which seems a decent argument for inclusion.

Perhaps an even bigger one; as @Legoktm said, this is a long list of formats, and it might be good to prioritize them. Here is my unilateral attempt, based in part on a severly underinformed assessment of how hard each thing will be. Comment and crit welcome:

  1. Just uploading LilyPond (and ABC) would give a lot of functionality, even with no display ability. I'd favour being able to edit them online as markup, but no idea how hard that would be. It would be great to be able to transclude them into, say, Wikipedia articles, but even cut-and-paste would be usable.
  1. MusicXML upload, and MusicXML to Lilypond conversion would be a good next step. There exists copyleft software to do it. MusicXML is an established format, a W3C recommendation, and the most widely used interchange format. It had the broadest support at the RfC on music formats.
  1. Braille music. Gently disagreeing with part of @Jc86035's much earlier comment; there is also a visual web display for the Braille format, and it is really useful if one is sighted. Braille integration seems to be a discrete task, and technically looks as though it could be fairly easy, with a helpful dev who has already written all the (GPL) software components and is interested in integrating them. Details on subtask. There are also conversion utils, so this can be seen as an accessibility interface, not just a format.
  1. Integration of the Verovio score renderer (LGPLv3 license). It will render (sight and sound, example) essentially everything proposed, except Lilypond (done), Braille, and MuseScore (whose inclusion I opposed). Verovio can visualize MusicXML (used by everyone) , MEI (a format used extensively by academics for PD works), Plaine & Easie (librarians in catalogues), ABC, and **kern (still figuring that one out). Display-less uploading of these formats could, of course, be done earlier.

I feel as if some of the other RfC participants might have more useful contributions here...

there is also a visual web display for the Braille format, and it is really useful if one is sighted

I was presuming that those who are sighted would prefer to use the other formats. I've never used Braille music, so I'm probably wrong here. I think if the dev is willing to implement it then that's great.

I still don't think it's fair to exclude MuseScore, if only for its larger user base and audience (and the fact that it's still open source). Combining e.g. MuseScore's soundfonts and/or audio generation with Verovio might also be useful, since as far as I can tell Verovio only has one lonely piano soundfont.

Jc86035 added a comment.EditedJan 20 2019, 3:42 AM

I agree that LilyPond would have to be prioritized, since a MediaWiki extension already exists for it. I don't really mind how the formats are prioritized, although I would put MuseScore either before or after MusicXML.

Editing them as markup could be possible from the beginning, since there is a user script on Commons for editing SVGs from the browser window which could be repurposed for LilyPond/ABC. The check to abort the script if the file on the page is not an SVG is one line long, so it would be fairly easy to make it into a LilyPond editor (although the preview function would have to be omitted, since it's SVG-specific). However, I don't think it should be a priority.

HLHJ added a comment.Jan 20 2019, 4:43 AM

I was presuming that those who are sighted would prefer to use the other formats. I've never used Braille music, so I'm probably wrong here. I think if the dev is willing to implement it then that's great.

I believe that sighted transcribers are the main users.

(Is it OK for me to comment here? If not, please let me know)

I think it's important to point out that "MuseScore" files are not really a recognized "open" format. Yes, they are used as the output of an open-source notation engraver, but they are primarily designed as an internal format for the software. AFAIK, there is no guarantee of versioned stability of MuseScore files between releases, so even minor updates to the software could see major changes to the internal format. See:

https://musescore.org/en/node/41106#comment-181251
https://musescore.org/en/node/13837

Jc86035 added a comment.EditedJan 20 2019, 10:35 AM

@Ahankins, yes, you're welcome to comment here.

According to that comment by Werner Schweer at https://musescore.org/en/node/41106#comment-181251, shouldn't LilyPond have similar problems (although perhaps less troublesome) with breaking changes? The MediaWiki extension for LilyPond has to use the actual LilyPond interpreter anyway, so presumably for any MuseScore extension the same would be done with the MuseScore source code (unless the files are converted to a different format first).

I agree it's important to point it out, but it's not really fair to single out MuseScore for using an "internal" format when the only notation software currently supported by MediaWiki could be described accurately by almost the same words.

Ahankins added a comment.EditedJan 20 2019, 10:41 AM

Yes, Lilypond does have similar problems. I personally wouldn't recommend it as a stable format either, but since it is already supported on MediaWiki I didn't see the point of arguing that it should be removed.

In general, "good" notation formats should be independent of any particular software. I could write a fairly long post on the thinking behind why this should be the case (and would be happy to do so if you think it useful) but the gist of it would be that music notation is complicated, and software implementations typically resort to 'hacks' to get things to display correctly, at the expense of accurately encoding and 'representing' the underlying music.

HLHJ added a comment.Jan 20 2019, 7:58 PM

T49528, "Create a VisualEditor plugin tool to add/edit musical scores", was also discussed in the RfC, and should probably be referenced here.

Jc3s5h removed a subscriber: Jc3s5h.Jan 21 2019, 8:10 PM
lpugin added a subscriber: lpugin.Jan 31 2019, 7:34 AM

Note that there was an overwhelming support for allowing these file types during the proposal discussion (request for comment).

Jc86035 added a subscriber: Lydia_Pintscher.EditedFeb 5 2019, 6:30 PM
  • Is there a good reason to favour LilyPond over other music notation formats? [...]

This is the one that can be reused inside the Wikimedia projects and that I have seen outside it as well. Any new one would need a lot of additional development time to support it that I'm currently not really convinced is worth it.

[...]
On the other hand, if it's exceedingly unlikely that anyone will put in the work to create new extensions, then the scope of T208494 should probably be considerably reduced to only allowing Commons uploads of the formats, since thumbnail generation would effectively require four new extensions (as well as work to enable thumbnails for LilyPond files). If that occurs, it would probably affect this task, since it would become possible to upload music notation files to Commons much sooner. It would be possible to instead indefinitely postpone enabling all formats other than LilyPond on Commons, but this would go against the consensus of the Commons RfC.

@TheDJ It's still not clear to me whether there is a need for the thumbnail generation to be enabled before uploads are enabled, and how the need for such is to be determined. Could you clarify this part of the procedure?

If it's determined that there is no need for thumbnails, then the amount of work which has to be done before enabling uploads is significantly reduced. If it's determined that there is a need for thumbnails, then a very likely outcome (as far as I'm aware) is that only LilyPond uploads are eventually enabled, and support for the other formats remains in development hell indefinitely.

If thumbnails aren't generated at first, then the main effect would be that users would be inconvenienced by having to export the files to PDF before being able to use them, and uses of files would need to be replaced upon enabling thumbnails for each format. On the other hand, even this would be an improvement over the current situation (i.e. scores have to be shared either as wikitext or through other websites); and the fact that exporting to PDF is possible for all of the formats – and would probably be done anyway in the MediaWiki thumbnail generation – means that thumbnails are not a necessity for the files to be usable. As such, I think it would be more practical to enable uploads first, although (again) I don't know how the decision is to be made.

AFAIK, there is no guarantee of versioned stability of MuseScore files between releases, so even minor updates to the software could see major changes to the internal format.

There is guaranteed stability in minor and patch releases. This means that the format is rarely changed, and when it is, the new features must be introduced in a backwards-compatible way so that the file can still be opened in older releases (but the new features will be ignored). An example of this happening is when support for "cresc." and "dim." lines was added. This was done in such a way that the lines would be rendered as hairpins in older versions.

Backwards-incompatible changes must only occur when the major version is increased. There is an argument to say that there should never be backwards-incompatible changes, but I would argue that these changes are necessary to allow the software to improve. This is in contrast to MusicXML which goes to great lengths to preserve backwards compatibility, at the cost of preserving some archaic syntax that it would be much better to get rid of. (An example of this is measure-repeats, which must contain both the instruction to repeat the previous measure, as well as the actual content of that measure written out all over again.)

In general, "good" notation formats should be independent of any particular software. I could write a fairly long post on the thinking behind why this should be the case (and would be happy to do so if you think it useful) but the gist of it would be that music notation is complicated, and software implementations typically resort to 'hacks' to get things to display correctly, at the expense of accurately encoding and 'representing' the underlying music.

True, but unfortunately notation programs also use 'hacks' to export to non-native formats. It's a choice of using a native format (like MuseScore format) and getting optimum results in that one program (and sub-optimum results in everything else), or using a non-native format (like MusicXML) and getting sub-optimum results in all programs.

T49528, "Create a VisualEditor plugin tool to add/edit musical scores", was also discussed in the RfC, and should probably be referenced here.

I would love to see this, but I can more or less guarantee it won't happen, or if it does it will be extremely limited in functionality for the foreseeable future. There has been talk about MuseScore creating an in-browser editor, but the talk is always about ways of taking the existing desktop code and running it in the browser, not starting from scratch in Javascript because we know that it is not a realistic proposition. LibreOffice has an online version called Collabora, but again this is just running the desktop program on a server, not a client-side implementation like Google Docs, and sheet music is much more difficult than text.

My personal preference would be for Wikimedia to allow uploading of some or all of these formats, but insist that files are only editable in the format that they were originally uploaded in. There could be an option to download in other formats, but those files would be generated automatically from the original file.

However, there should be some thought given to the fact that there are already other projects creating these files, and that it would probably be better to avoid duplicating effort where possible. Perhaps where scores already exist in another project, Wikimedia should simply link to the other project's file, perhaps storing it's own copy as a backup/mirror in case the other project goes down?

Koavf added a subscriber: Koavf.Feb 6 2019, 12:36 AM

The RfC has ended without a formal closure, and is now archived. There was majority support for all six file types that were options by the end of the RfC, although five users opposed allowing MuseScore uploads mainly due to issues with the file format.

I'm not sure adding 5 different file types for the same type of content is the best idea, but okay...

I have to be honest, I'm confused by this: what harm is there in allowing (e.g.) PNG and GIF and TIFF and JPEG all at once? Yes, those formats have relative strengths and weaknesses but they are all (minus animated GIFs), static images. It's okay to have PDF and DJVU. I just don't see what is *worse* about including a broader array of free file formats. If one is clearly superior (e.g. how SVG is a better format for many static images than any raster option), then won't users generally gravitate toward that? Plus, it's sometimes cumbersome to convert files, so it's a good thing to allow an easy way to upload (e.g.) MP3s rather than require OGG conversion.

The RfC has ended without a formal closure, and is now archived. There was majority support for all six file types that were options by the end of the RfC, although five users opposed allowing MuseScore uploads mainly due to issues with the file format.

I'm not sure adding 5 different file types for the same type of content is the best idea, but okay...

I have to be honest, I'm confused by this: what harm is there in allowing (e.g.) PNG and GIF and TIFF and JPEG all at once?

Each one will need its own work to be integrated into MediaWiki. That means, we'll need to add its own security screening, its own validation, its own rendering, its own playback, its own set of tests, its own interface messages, its own server-side dependency management, and so on. Adding each format costs maybe US$5–10k of staff time, plus maybe $1–2k per year of on-going security issues / fix-ups / testing / hardware growth requirements / etc. (some formats, like SVG of GIF, cost more – because they're less secure, have complex pipeline issues, or other problems). As each of these formats is novel and rarely used (in comparison to PNG), they'll be towards the upper end of these values.

Adding all seven formats at once probably would save some effort, but we're still talking a significant sum to spend on a relatively low-interest niche, which would instead be spent on other things.

I'm not sure adding 5 different file types for the same type of content is the best idea, but okay...

I have to be honest, I'm confused by this: what harm is there in allowing (e.g.) PNG and GIF and TIFF and JPEG all at once? Yes, those formats have relative strengths and weaknesses but they are all (minus animated GIFs), static images. It's okay to have PDF and DJVU. I just don't see what is *worse* about including a broader array of free file formats. If one is clearly superior (e.g. how SVG is a better format for many static images than any raster option), then won't users generally gravitate toward that? Plus, it's sometimes cumbersome to convert files, so it's a good thing to allow an easy way to upload (e.g.) MP3s rather than require OGG conversion.

The cost is that each format needs to have code written for it, plus maintained. We also need to review different external software for each one, and test it during OS upgrades.

PNG is good for some files, GIF is for animated stuff, TIFF is good for bigger files, and JPG is uh, the format that most consumer cameras use? I'm not really an expert tbh, just my general worries about maintenance costs. That said, if people are willing to put in the development effort and help with maintenance and other people who are more experienced in this area think it's a good idea, I'm happy to help however I can.

lpugin added a comment.Feb 6 2019, 7:29 AM

It seems to me that SVG is a good candidate. It is AFAIK the only vector graphic format supported by decently recent browsers. Thumbnails (PNG) can be easily be generated from it if necessary, including client-side with JavaScript and Canvas.

It seems to me that SVG is a good candidate. It is AFAIK the only vector graphic format supported by decently recent browsers. Thumbnails (PNG) can be easily be generated from it if necessary, including client-side with JavaScript and Canvas.

I assumed it would be PDF due to some/most of the formats having page layout settings (and rendering multi-page scores as one thumbnail would probably be undesirable); MediaWiki can render single pages of PDFs as thumbnails. I don't really know how this would work.

Notably, the Score extension currently doesn't use SVG because of librsvg font issues (see T49578).

lpugin added a comment.Feb 6 2019, 2:10 PM

Notably, the Score extension currently doesn't use SVG because of librsvg font issues (see T49578).

OK, this is good to know. So SVG would be fine as long as librsvg is not used in the pipe to produce them.

Ebe123 added a comment.Feb 6 2019, 2:23 PM

@Jc86035: No, librsvg is not really a blocker. It's that the option for SVG layout in Lilypond is in the next version. For PNG fallback, imagemagick could still be used (instead of librsvg)

TheDJ added a comment.Feb 6 2019, 4:06 PM

@TheDJ It's still not clear to me whether there is a need for the thumbnail generation to be enabled before uploads are enabled, and how the need for such is to be determined. Could you clarify this part of the procedure?

Thumbnails, or rather ANY form of transforming the raw file into output other than a browser download of that exact original file is not required, but of course severely limits the usability of such files within a wiki. It basically only allows preservation of the file.

Ankry added a subscriber: Ankry.Apr 13 2019, 7:25 AM