Change 370209 had a related patch set uploaded (by Ebe123; owner: Ebe123):
[mediawiki/extensions/Score@master] Output SVG in scores

https://gerrit.wikimedia.org/r/370209

A caveat that may prove important is the treatment of text (like in tempo markings) in SVG. There are some outstanding bugs, such as issue 5174 and issue 3778 (each of them relating to text handling, specifically in tempo markings) that make SVG output less good, although it's very minor.

Further on text, the conversion, using MediaWiki's rasterizer leaves us with the problem described in T36947: Incorrect text positioning in SVG rasterization (scale/transform; font-size; kerning).

Eh, no. We still must wait for it to be released in their 2.20 (or 2.22 if
there's bad luck) stable, and still have to implement it in the MediaWiki-extensions-Score
extension. Re-opening.

YAY

After this blocker, there is also the problem with librsvg, which renders text in SVG scores very badly. As an example:

In T49578#3857903 back in 2017, @Ebe123 wrote:

The new -dcrop option will be in the next stable release (2.20), which may be in the next few months.

Debian 10 (buster?) backports has Lilypond 2.22.0, so perhaps this should depend on a task to update Lilypond? Lilypond itself is now at 2.22.1, with some worthwhile improvements including I think a new -svg command-line option.

Thanks for all y'all do ❣

In T49578#3861166, @Ebe123 wrote:

After this blocker, there is also the problem with librsvg, which renders text in SVG scores very badly. As an example:

Updating librsvg is another block here, notwithstanding further text improvements needed in lilypond, such as with skylines, but that is much less severe.

Because most Score operations now run in a container, it's a lot more straightforward for us to use a newer librsvg without affecting all other SVG rendering.

The most recent comment in T36947 said "librsvg 2.51.2 (and librsvg 2.50.6, once backported)" - could you retry it with one of those versions to see if it's fixed now?

@Legoktm T36947 is very system-dependend and differ between OS and on different scalings. For https://commons.wikimedia.org/wiki/File:O_Canada_Lilypond.svg you just need to upscale the fonts as done in https://commons.wikimedia.org/wiki/File:O_Canada_Lilypond_Workaround.svg, or described at https://en.wikipedia.org/wiki/Wikipedia:SVG_help#bad_letter-alignment_on_small_font-size.

Rendering of

rust-librsvg2.50.6	resvg	Inkscape	batik

librsvg must be packed for debian stretch/buster, since librsvg is on stretch on 2.40 and on buster on 2.44, see https://packages.debian.org/search?keywords=librsvg2 .

resvg, Inkscape and batik are portable or have portable Images, so they do not depend on system-libaries. Therefore it is imho much easier to change svg-renderer (see T40010 ) than to package rust-librsvg2.50.6 (imho depends on cairo, glib, gtk) . I personally prefer to switch to resvg, see svg-benchmarks .

Thank you for the comparison @JoKalliauer, I think we could probably get resvg into the Score/Shellbox container, that said...do we actually need a separate SVG renderer? Could we continue to have LilyPond generate PNGs in addition to SVG? And display to users the SVG with PNG fallback?

Is there anything I can do to help this along? I'm a PHP dev with a little time on my hands currently, but it seems as if this is more to do with aligning SVG rendering libs on container OSes? Let me know if I can help in some way. Is this changeset still useful, or has it been superceded by more recent work?

Message from Han-Wen Nienhuys, August 2021 to lilypond-devel:

* SVG.

  The current SVG backend is glacially slow, and has suffered from
  rendering discrepancies. I propose we retire it ASAP to be
  replaced by Cairo. The only feature missing is the 'svg-woff option;
  not sure how important that is? (CC'ing Jan who implemented this.)

    [hanwen@fedora lilypond]$ time lilypond --svg
input/regression/les-nereides.ly
    GNU LilyPond 2.23.4
    ..
    real 0m1.662s
    [hanwen@fedora lilypond]$ time lilypond  --svg -dbackend=cairo
input/regression/les-nereides.ly
    GNU LilyPond 2.23.4
    ....
    real 0m0.728s

  The Cairo SVG files are larger, but that is because they also embed
  the fonts used for texts, making the rendering exactly equal to the
  PDF/PNG.

I would suggest not using the old SVG backend since it is rarely used and effectively deprecated. It doesn't embed fonts, so rendering depends on the browser. The well-trodden path is to treat LilyPond as a PostScript generator.

The Cairo backend could be used. It is currently disabled by default, but at least if we use it and find bugs, there might be some interest from upstream in fixing them. It has had 16 commits this year.

In other words, the LilyPond backends are:

• PostScript: stable and used by just about everyone
• SVG: a deprecated curiosity
• Cairo: the bleeding edge future, can output to SVG or PNG

In T49578#8155665, @Jonathanischoice wrote:

Is there anything I can do to help this along? I'm a PHP dev with a little time on my hands currently, but it seems as if this is more to do with aligning SVG rendering libs on container OSes? Let me know if I can help in some way. Is this changeset still useful, or has it been superceded by more recent work?

There's plenty to do in PHP. I would suggest installing MediaWiki, Score and a locally compiled LilyPond with Cairo enabled. Then investigate the possibility of using the Cairo backend to generate both SVG and PNG. There is not much in commit 370209 that would stay the same, so it might be easier to start from scratch.

Noting here the upstream progress on including the Cairo backend in Lilypond by default: merge request 1610, and lilypond-devel discussion thread.

I have a sneaking suspicion that unless we are really still wanting to support really old unsupported browsers, like Internet Explorer 11, we really don't need a PNG fallback. SVG images are supported by basically all of the things, and even IE 11 is passable as long as the SVG is not animated:
https://caniuse.com/?search=svg

Comments on this would be welcome, because if we don't need PNG then we can circumvent using --ps and depending on GhostScript. Update: see also T5593

I have got a small initial commit with SVG support working nicely, using the upcoming December release of Lilypond 2.24 (currently testing on 2.23.82 with libcairo). I am using the -dno-use-paper-size-for-page option (if $wgScoreTrim` is true) to do what all the existing trim and size detection code is doing (since -dcrop does not quite work as we need). If phabricator will allow me, I'll try pushing a feature branch to Gerrit, otherwise I'll stash it on GitLab somewhere for someone to have a look at.

I am currently working on getting the multi-page output working as before. I have no idea if the test suite is passing yet, so it's still early days :)

I can't seem to be able to push to Gerrit; in the meantime, the very basic diff is displayed here on GitLab. This should work as long as you have a 2.24 RC version of Lilypond; I have been testing with the 2.23.82 release.

Here's a 1.35 instance with the 1.35 SVG patch installed:
https://rel1-35.home.jon.geek.nz/wiki/Score_examples

Compare, the current PNG output, here:
https://wiki.jon.geek.nz/Score_examples

I can't seem to be able to push to Gerrit

@Jonathanischoice: Thanks for taking a look at the code! Could you please elaborate on the problem in a support forum? Usually developer access should allow you to submit proposed code changes as a Git branch into Gerrit; if you don't want or cannot set up Git/Gerrit, you can also use the Gerrit Patch Uploader. Thanks again!

@Aklapper - I get this, and I'm not sure where/who to post or contact:

remote: error: branch refs/heads/generate-svg:
remote: You need 'Create' rights to create new references.
remote: User: jonathanischoice
remote: Contact an administrator to fix the permissions
remote: Processing changes: refs: 1, done    
To ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Score
 ! [remote rejected] generate-svg -> generate-svg (prohibited by Gerrit: not permitted: create)
error: failed to push some refs to 'ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Score'

@Jonathanischoice you should use git-review to upload changes. It is not possible to push to refs/heads/*, you need to push to refs/for/* to create a change for review, which git-review takes care of.

Ah yes... I remember now. It's been years since I used gerrit... oh, and I need to rebase to master, which will take a while because change

Change 862408 had a related patch set uploaded (by Jonathanischoice; author: Jonathanischoice):

[mediawiki/extensions/Score@master] Get SVG output working for cropped, non-paginated input, T49578

https://gerrit.wikimedia.org/r/862408

This is still a WIP:

• after the rebase to master, I followed my nose and removed the shell trim code that uses convert, and thus also the ImageMagick dependency altogether.
• I'm still working on paginated output, which is also trimmed correctly in both the SVG and PNG, but it is not currently adding the SVG page images.
• I haven't even looked at the tests yet, so I guess they'll be failing; so much for TDD :)
• I suspect that just using --png instead of deriving them from --ps will also make a bunch more code redundant, and even remove the GhostScript dependency, but I haven't tried that yet.

@Jonathanischoice just an fyi. You don't need to delete a translation key from every language dictionary. If you remove it from en and qqq, the rest will be taken care of automatically. https://www.mediawiki.org/wiki/Help:System_message#Removing_existing_messages

Thanks @TheDJ !

I've resolved several things, and pushed an update to Gerrit. I've also found that we don't even need the --ps output and GhostScript any more, because we can fetch the same page size information from the <svg> attributes.

Change 863576 had a related patch set uploaded (by Jonathanischoice; author: Jonathanischoice):

[mediawiki/extensions/Score@master] Get page dimensions from SVG instead of PostScript, T49578

https://gerrit.wikimedia.org/r/863576

I've pushed two commits ready for review - @TheDJ and @tstarling. Still not sure what the procedure is here; it's been ages since I've had to use Gerrit. Test instance here.
Also: is it worth smooshing them together into one commit?

Change 863576 abandoned by Jonathanischoice:

[mediawiki/extensions/Score@master] Get page dimensions from SVG instead of PostScript, T49578

Reason:

Have squashed this into the first commit, see T49578

https://gerrit.wikimedia.org/r/863576

Ok, so it turns out we don't even need to get the page size at all, because that was only being used by GhostScript to generate the PNG from the PS output, which we're not doing in this patch. Have pushed an all-in-one commit to Gerrit. I can't see how to easily add a comment to a Gerrit MR, hence the chattiness here (sorry?)

https://gerrit.wikimedia.org/r/862408

Change 862408 merged by jenkins-bot:

[mediawiki/extensions/Score@master] Use LilyPond with libcairo to generate SVG and PNG directly, T49578

https://gerrit.wikimedia.org/r/862408

Hooray :) Now I have a thought about what to do with the existing PNG renders that are being used on pages until they are next edited. Is it possible to create some sort of maintenance task that can process the image store in the background, generating any missing SVG files using the preserved LilyPond file?

Change 868532 had a related patch set uploaded (by Reedy; author: Jonathanischoice):

[mediawiki/extensions/Score@REL1_39] Use LilyPond with libcairo to generate SVG and PNG directly, T49578

https://gerrit.wikimedia.org/r/868532

Change 868532 merged by jenkins-bot:

[mediawiki/extensions/Score@REL1_39] Use LilyPond with libcairo to generate SVG and PNG directly, T49578

https://gerrit.wikimedia.org/r/868532

Change 370209 abandoned by Ebe123:

[mediawiki/extensions/Score@master] Output SVG in scores

Reason:

Sure

https://gerrit.wikimedia.org/r/370209

Noting here that LilyPond 2.24.0 was released 16 December 2022 - changes. Shall we boldly create an "Update LilyPond in Shellbox container to 2.24" task, similar to the previous one?

This was merged over 2 months ago, what happens next? I'd like to get T134455 advanced so we can get on with SVG Score output somehow.

So you want to move this to WMF production ?

I see at least two follow up tasks for that:

1. Update the shellbox container with a new version of lilypond
   1. This seems complicated by the fact that last time @Legoktm tried to update, the updated package for lilypond wasn't in the version of Debian used yet, and making their own version seemed a challenge.
2. Make a deploy plan
   1. Find a WMF deployer willing to assist you
   2. Deploy the new version of the shellbox container and make sure there are no regressions
   3. Modify the WMF configuration for the beta cluster to use the new config via a patch on the operations repo
   4. Setup a page in beta cluster with various score usages to verify everything works as expected
   5. Modify the WMF configuration to use the new option for deploy group 0
   6. Setup a page on test.wikipedia.org with various score usages to verify everything works as expected
   7. Make an announcement in Tech-news via User-notice
   8. Modify the WMF configuration to use the new option for deploy group 1
   9. Modify the WMF configuration to use the new option for deploy group 2

also..

I'd like to get T134455 advanced so we can get on with SVG Score output somehow.

I'm not sure what these two things have to do with eachother. The file thumbnailing architecture is fully separate from the Score images isn't it ?

You'll have to forgive my total lack of knowledge of Wikipedia internals and toolchains since I'm not an employee, I'm just a Wikipedia editor with an overinflated sense of enthusiasm and can code in PHP, so I offered to help with the Score extension. I'm not sure I'd be much use for most of that?

In T49578#8511298, @Jonathanischoice wrote:

Noting here that LilyPond 2.24.0 was released 16 December 2022 - changes. Shall we boldly create an "Update LilyPond in Shellbox container to 2.24" task, similar to the previous one?

Yes please. TheDJ is right that last time it was a mess and I gave up, but it's certainly worth another shot. Worst case we just have to wait a bit longer until we can use the bookworm 2.24 package.

Surely we can just pull the official upstream binary tarball from GitLab, extract it somewhere, and point LocalSettings.php variables at it? No Debian package required.

In T49578#8708353, @Jonathanischoice wrote:

Surely we can just pull the official upstream binary tarball from GitLab, extract it somewhere, and point LocalSettings.php variables at it? No Debian package required.

Our standard practice is to prefer using Debian packages when possible as it reduces the work we need to do and lets us take advantage of Debian's security support, etc. (also a growing number of Wikimedia devs are also Debian contributors, including myself, so it's a reciprocal relationship).

lilypond is deployed in a container (current config: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/libs/Shellbox/+/refs/heads/master/.pipeline/blubber.yaml#84), so we have a good amount of flexibility in building it ourselves, but last time I tried there were weird complications related to needing a specific guile version that Debian had dropped or something.

(This is not to rule out using an upstream binary, I think we do that in a few other places that I can't remember off the top of my head, I just think that should be a last resort.)

Fair enough. there seems to be promising recent activity on 1026200 anyway

Given the patch on T208578, should we adjust Score's behaviour around serving SVG? So, rather than use srcset (see T134455) to provide SVG with a PNG fallback, we look for the new $wgSVGNativeRendering setting and just serve SVG directly. We'll still need Lilypond to generate PNG, but it is only served for older versions of Mediawiki (using the current srcset behaviour) or where the SVG generated by Lilypond exceeds the new $wgSVGNativeRenderingSizeLimit setting. Another possibility (useful for older versions of Mediawiki) is to detect the presence of the NativeSVGHandler extension and serve SVG directly if enabled. Thoughts?

Lilypond packages are now built with 2.24 in bookworm and trixie (and sid).

@Legoktm @TheDJ - is this likely to happen soon? I'm not sure I can help further other than annoy you by pinging... :-/

... have I gone mad, or have the Debian packages of lilypond 2.24 left out libcairo support? Is this some sort of torture?
Update: I have submitted a Debian bug regarding missing libcairo support in the lilypond 2.24 package.