# Allow musical notation to be expressed in Wikidata, with rendering and maybe playbackOpen, Needs TriagePublicActions

Assigned To
None
Authored By
 Jc86035 Jun 26 2018, 4:58 PM
Tokens
"Like" token, awarded by Liuxinyu970226.

# Description

There should be a way of indicating the standard musical notation of something in Wikidata. Currently there is no way to do so in a way which at least matches the functionality of the math datatype (which can render equations). It is already possible to use the string datatype but it is limited to four hundred characters and would not be able to render anything.

Two possible methods to solve this are:

• using LilyPond notation;
• using files hosted on Commons, probably MuseScore files (or MusicXML files).

There is already an Extension:Score (MediaWiki-extensions-Score) for LilyPond, which does rendering and MIDI audio output. Scores using this LilyPond datatype would also probably be limited to four hundred characters. Longer scores in the public domain (or compatibly licensed) would have to use a different solution, although this may be desired. Quoting @Mahir256:

Apologies if I'm drawing too many unneeded parallels to the present status of LaTeX equations, but I think the situations here are very similar. We have migrated quite a bit away from storing images of equations on Commons to simply rendering them on wikis where they are needed (whether on Wikipedia, on Wikisource, and now here), even as we don't store full-fledged mathematical derivations in statements here due to issues with the maximum length of the string datatype. Similarly we have the ability to migrate from storing images of music snippets on Commons to simply rendering them where they are needed (whether on Wikisource, on Wikipedia, and potentially here), and I am sure it will not be feasible to store full scores of pieces as statements here for the same technical reason. As such the use cases I have suggested above are intended to exclude the possibility of storing entire scores, just as entire derivations are excluded from being stored here.

Allowing MuseScore rendering or playback would require the creation of a MediaWiki extension which would basically incorporate MuseScore's (currently existing) command line export functions, likely for PDF rendering and Ogg audio playback. The scores would be hosted on Commons. Quoting @Mahir256:

I have noticed that the introduction of new extensions to Wikimedia sites becomes problematic as there are generally numerous security issues to be resolved and often there needs to be someone to maintain said extensions for use specifically with Wikimedia sites.

I am not sure if all the concerns are applicable, but there would obviously need to be a security review and post-deployment maintenance. My reasoning for not relying on Extension:Score (tied 97 of 214 in 2017 Community Wishlist Survey):

The Score extension is not particularly easy to use due to not having a graphical input option, and has not been updated in almost four years. Furthermore, the rendering is pixelated and PNG instead of SVG; and the image is hard to resize (a paper size has to be specified) and align (a div has to be made to align it and the audio player to anything). In addition, the double bass and possibly some other instruments are not playable.

# Related ObjectsSearch...

OpenNone
ResolvedLydia_Pintscher
Resolved alaa_wmde
Resolved alaa_wmde
Resolved alaa_wmde
ResolvedLydia_Pintscher
Resolved alaa_wmde
OpenNone
Resolved alaa_wmde
OpenNone
OpenNone
Resolved Greta_Doci_WMDE
OpenNone
OpenNone
StalledNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
DuplicateNone
OpenNone

### Event Timeline

Jc86035 created this task.Jun 26 2018, 4:58 PM
Restricted Application added a subscriber: Aklapper. Jun 26 2018, 4:58 PM
Jc86035 renamed this task from Create musical notation datatype to Create musical notation datatype(s), with rendering and maybe playback.Jun 26 2018, 5:00 PM
Jc86035 updated the task description. (Show Details)Jun 26 2018, 5:02 PM
Jc86035 updated the task description. (Show Details)Jun 26 2018, 5:27 PM

The property proposal for "LilyPond notation" currently has no opposes, so it seems likely to be created at some point.

Thanks for filing the ticket!
From my side: in general fine and if we want a new datatype for it then this could be a nice student project for example.

Open question: Do we want this to be on Wikidata as statements? Or on Commons to link to it? The drawback I can see for having it on Wikidata as statements would be length and licensing limitations that might not be great for this type of content. Thoughts?

@Lydia_Pintscher (See the larger of the excerpts from my proposal on Wikidata:Project chat that Jc86035 included in the task description.) We initially stored population data as individual values on an item, but later began storing it as links to Commons tabular data (properties 1082 and 4179 respectively). We also store isolated mathematical expressions, not full derivations or proofs, as LaTeX snippets, and it would not surprise me if someone came around and proposed a property for links to derivations and proofs hosted on Commons (whether as raw LaTeX or otherwise). It would therefore make sense to initially store as statements musical snippets that fit in the 400-character limit for strings, and then later once a MuseScore extension for MediaWiki comes to fruition (or if Extension:Score is significantly revamped) link to full sheet music on Commons. The specific examples on the property proposal that Jc86035 created should not cause copyright problems, and I personally would not have a problem if we need to restrict such a property's use on items for copyrighted music.

@Jc86035 Regarding the points in your excerpt from the survey back in November/December:

• There is at least one graphical editor of LilyPond music; I am not certain whether this can or should be made into an extension.
• The extension has not been updated in a while, but at least @Ebe123 is around if we have more questions about it.
• SVG output is certainly possible with LilyPond; it requires an extra flag but I am not entirely sure what else may be needed for proper rendering.
• SVG output might solve the resizing and alignment issues, though I am also not entirely sure what else may be needed for this as well.
• Wouldn't the problem of unplayable instruments come down to a need for better soundfonts (either for Fluidsynth or TiMidity++)? It would certainly help to know what soundfonts are used at present.

Also @tstarling who may be able to answer more of the concerns presented in this task.

Ebe123 updated the task description. (Show Details)

Sorry, I was not aware of this task until I got pinged.

Few corrections:
@Jc86035: We do have the most recent stable version of LilyPond; they haven't had such a release in 5 years... I do not think you are being fair to the progress that is being done in the extension, especially that your comment was made when I was beginning to become more active in this extension. We are working on fixing the issues that you pointed out, with these tasks for the points you bring up: T49578: Score should output SVG, T143500: Add all instruments to Extension:Score (please provide an example if its still broken), and T49528: Create a VisualEditor plugin tool to add/edit musical scores. Check the work-board to see what we have planned :)

I understand the frustration of not having a proper GUI for writing LilyPond, and the task is daunting. I am inclined to agree for the case to render MusicXML (of which the ability to do so is tracked with T183642: Support MusicXML to Lilypond Conversion). I am not in favour of having a whole new extension based on MuseScore, which does have its share of format (.mscx/z) problems. That format is very fragile, and does not feature the scope of what is possible through LilyPond, which has broad support for all kinds of notation that MuseScore struggles with. Your assessment that LilyPond is "indisputably inferior" is false.

Other remedies included OMR (not natively available in MuseScore BTW), which is good; however, there is no open-source (or probably even closed-source) application that does it well enough to be useful in my view. MIDI is very hairy because it is not really a representation of sheet music; it can just be converted (somewhat) into that form.

For use in Wikidata, to get around this size limitation, we can use the extension in the way it is mostly used on Wikipedia: display notable snippets/motifs of a piece. My counter-proposal is rather than storing something like the whole score of The Planets, we can store the motifs (and it's not limited to one:

\relative c { \clef bass \time 5/4 \key c \major \tempo "Allegro" \tuplet 3/2 { g8_\markup { \dynamic p \italic "col ligno" } g g } g4 g g8 g g4 \bar ":|]" }

The parallels drawn between \TeX and LilyPond are quite accurate, and @Mahir256's proposal seems to be echoing my sentiment. I apologise for the lack of updates, but we're waiting for the next release of LilyPond for SVG output (had to make changes upstream; it was not as simple as an extra flag sadly) and the extension does need quite a lot done, though time is not available. :)

We do have the most recent stable version of LilyPond; they haven't had such a release in 5 years...

Oh, I actually didn't realize that when I was writing up the wishlist proposal. Sorry about that. I thought it had been updated at some point, possibly because it had been a while since I had downloaded and used LilyPond as an application, and it was updated during the time that I used it.

indisputably inferior

Yes, this is quite unfair in retrospect. I did have to remove some things attributable to incorrect memory from the wishlist proposal after I'd posted it, because I had essentially put everything I could think of in there to make it sort of actually have a chance to be worked on by Community Tech (although I'm not sure if that would actually have helped in the end). I think part of the problem was that I was comparing the LilyPond features available in Extension:Score directly to both the MuseScore desktop application and the MuseScore closed-source online player.

It's also because I was frustrated with trying to add snippets into articles; on w:en:Double bass, I think it took me about an hour to figure out that the paper size had to be changed to resize the image, and I had to set the instrument to something else because the audio was coming out as piano.

I still think it should be possible, at some point, to be able to link to full scores in their original source format; it would be nice to at least support hosting .mscz/x and/or .ly files on Commons. (Is it common for full scores to be written with LilyPond? I must admit I wasn't actually aware that people would transcribe orchestral scores in LilyPond.)

Can't really think of a proper way for partitions to "re-flow" to fit its allotted space. Your way of changing the paper size (or to use the LilyPond command \break) would still be needed. All the instruments that are supported in LilyPond are listed here. The bug T199356: Contrabass MIDI instrument is unusable was created for the missing instrument.

@Mahir256, I hope my replies and linked tasks follows the points you have made. My response to your first point may not have been clear, but I'd see the GUI being added to VisualEditor, but I'll have to study more about both it and Denemo to see its applicability. Huge project though.

Please direct these bug reports and feature requests into their own (new) tasks (with the MediaWiki-extensions-Score project tag) so that this task can focus on the integration of the extension within Wikidata.

Thank you for your comments @Ebe123; I will see if I can get a MediaWiki instance running on my laptop so I can examine the extension myself. I have noted your points regarding new bug reports.

In related developments, we now have a LilyPond notation property. Here's hoping that it is put to good use.

Jc86035 updated the task description. (Show Details)Oct 30 2018, 6:54 PM

(I have made a minor edit to the task description to correct both my stated point of view and factual inaccuracies.)

I have submitted another Wishlist proposal for this year, which if implemented would partially or entirely resolve this task.