Page MenuHomePhabricator

Extension:Score / Lilypond is disabled on all wikis
Open, HighPublic

Description

Due to an ongoing security issue, Score/Lilypond have been disabled on Wikimedia wikis for the time being.

This task serves as the public tracking for this issue

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Has this been fixed? It looks like the errors are gone now and Score is working again.

Reedy changed the status of subtask Restricted Task from Open to Stalled.Jul 6 2020, 3:29 PM
Reedy added a comment.Jul 6 2020, 4:07 PM

Has this been fixed? It looks like the errors are gone now and Score is working again.

No, any that are viewable are because the output is already generated and saved. Any new or modified scores (which don't match a pre-saved render) will not work. So you should be able to copy a working render to another page, but not create a new one

Well, some output did not work two days ago, but it does now. Caching problems, I guess, different users are reporting different errors (and the error category is also just shown in some cases). I will keep the warning then as long as this task is open.

Reedy added a comment.Jul 6 2020, 4:39 PM

Well, some output did not work two days ago, but it does now. Caching problems, I guess, different users are reporting different errors (and the error category is also just shown in some cases). I will keep the warning then as long as this task is open.

Yes, a change was made to fix it where they should be fine to be displayed anyway

Mentioned in SAL (#wikimedia-operations) [2020-07-04T02:28:09Z] <reedy@deploy1001> Synchronized php-1.35.0-wmf.39/extensions/Score/includes/Score.php: Short circuit lilypond version check to allow usage of cached files T257066 (duration: 00m 55s)

DMacks added a subscriber: DMacks.Jul 6 2020, 9:51 PM
MBH added a subscriber: MBH.Jul 7 2020, 1:30 PM
ArielGlenn added a subtask: Restricted Task.Jul 8 2020, 6:50 AM
Tgr added a subscriber: Tgr.Jul 10 2020, 3:23 PM

Have we made an effort to reach out to non-Wikimedia MediaWiki users? Given the severity, warning them well before details of the issue become public seems prudent.

Have we made an effort to reach out to non-Wikimedia MediaWiki users? Given the severity, warning them well before details of the issue become public seems prudent.

I'm not seeing anything within the incident doc indicating this was done, though @Dsharpe could confirm for sure. Looks like there's not quite 50 non-Wikimedia wikis currently using Score according to wikiapiary.

Change 612274 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove lilypond for now

https://gerrit.wikimedia.org/r/612274

Change 612274 merged by Muehlenhoff:
[operations/puppet@production] Remove lilypond for now

https://gerrit.wikimedia.org/r/612274

Mentioned in SAL (#wikimedia-operations) [2020-07-13T14:17:01Z] <moritzm> removing lilypond from production T257066

mb added a comment.Jul 21 2020, 4:30 AM

This was announced to be fixed by 6 July 2020. It's now been 3 weeks. Is there an ETA for a fix?

Reedy added a comment.Jul 21 2020, 1:14 PM
In T257066#6321576, @mb wrote:

This was announced to be fixed by 6 July 2020. It's now been 3 weeks. Is there an ETA for a fix?

No it wasn't.

An issue is being diagnosed involving this extension, and it will likely remain down until at least Monday, 6 July 2020. We took the functionality down out of an abundance of caution after being made aware of a potential problem. More to come...

And specifically

will likely remain down until at least Monday, 6 July 2020

ie, this won't be fixed before. Not that it will be fixed on that date.

It will probably be re-enabled in safe mode early next week. Hopefully Monday my time (i.e. Sunday night US time). I don't know how long it will take to restore it in unsafe mode, probably another couple of weeks.

If you're a user, please let us know how much you need this feature, to help us prioritise the work on it.

tstarling closed subtask Restricted Task as Resolved.Jul 29 2020, 2:34 AM
tstarling closed this task as Resolved.Jul 29 2020, 11:49 PM
tstarling claimed this task.

Will there be any disclosure of the issue so 3rd party sites that followed suit in disabling it know what's safe?

Will there be any disclosure of the issue so 3rd party sites that followed suit in disabling it know what's safe?

Yes. I planned on putting out an announcement a couple of days ago, but it turns out to be more complicated than expected. There will definitely be an email to mediawiki-announce at some point, and some of the private tasks will be made public.

Yes. I planned on putting out an announcement a couple of days ago, but it turns out to be more complicated than expected. There will definitely be an email to mediawiki-announce at some point, and some of the private tasks will be made public.

Thanks, I'll keep an eye out!

tstarling reopened this task as Open.Jul 31 2020, 12:48 AM
tstarling reopened subtask Restricted Task as Open.

It's disabled again, since I found a new vulnerability.

tstarling closed this task as Resolved.Aug 4 2020, 8:39 AM
tstarling closed subtask Restricted Task as Resolved.

We still can't announce anything since we're waiting for vendor security releases. Third party sites should leave lilypond execution disabled.

Joe reopened subtask Restricted Task as Open.Aug 13 2020, 1:01 PM
wkandek closed subtask Restricted Task as Resolved.Aug 18 2020, 1:55 PM
Pppery added a subscriber: Pppery.Aug 19 2020, 3:06 PM
kaldari reopened this task as Open.Aug 28 2020, 1:38 AM
kaldari added a subscriber: kaldari.

Reopening per T257091

Alejabo lowered the priority of this task from High to Medium.Oct 6 2020, 11:40 PM
Ladsgroup raised the priority of this task from Medium to High.Oct 6 2020, 11:46 PM
Ladsgroup added a subscriber: Ladsgroup.

Do not change priorities without any context.

It is 3 months since the extension was disabled. When will any news be announced?

The disabling of these score tags appears to have some negative interaction with the rest of the page. See this revision:

https://en.wikipedia.org/w/index.php?title=Circle_of_fifths&oldid=981481037#Chromatic_circle

In the section following the (disabled) score tags, there is some math rendering that shows as:

'"UNIQ--postMath-00000001-QINU"'

instead of the proper fancy Z with a 12 after it. See this revision, where the only change is commenting out the score tags entirely:

https://en.wikipedia.org/w/index.php?title=Circle_of_fifths&type=revision&diff=984700416&oldid=981481037

Please change the method of disabling the score tags such that it does not affect the rest of the page.

@patilise - Because of the nature of the problem, we unfortunately can't share many details right now. We consider this issue high priority and are actively working to resolve it, but the problem has turned out to be more complicated than initially expected. We are still hoping to be able to re-enable the Score extension in safe mode once a couple more bugs are fixed and we have completed a security audit of Score and Lilypond. Unfortunately, some features will not work under safe mode, so even that will only be a partial solution. (Some of the features that are disabled under safe mode are listed in the description of T174413.)

[My personal take, which is only an educated guess, is that it may be a while before Score is back up and running. Communities that rely on it may want to consider replacing score notations with images and/or audio files in the meantime. Tim Starling or other developers may have more specific information.]

Since a couple of days, saving any article that calls the score extension leads to the score becoming invisible (without having the score itself changed). So the workaround of reading pre-rendered files from the cache seems no longer to be working. It is really annoying that the problem is still not solved after such a long time of waiting. There are currently 1,241 pages calling score on en:wikipedia, 748 on de:wikipedia which are endangered of losing the proper display of the contained scores.

Matlin added a subscriber: Matlin.Oct 27 2020, 9:15 AM

Since a couple of days, saving any article that calls the score extension leads to the score becoming invisible (without having the score itself changed). So the workaround of reading pre-rendered files from the cache seems no longer to be working. It is really annoying that the problem is still not solved after such a long time of waiting. There are currently 1,241 pages calling score on en:wikipedia, 748 on de:wikipedia which are endangered of losing the proper display of the contained scores.

And 2,161 on en:Wikisource. And 1,254 pages marked as waiting for a score to be added. In addition, we've got several books that are stalled while waiting for this update.

Ankry added a subscriber: Ankry.Nov 15 2020, 3:38 PM

Since a couple of days, saving any article that calls the score extension leads to the score becoming invisible (without having the score itself changed). So the workaround of reading pre-rendered files from the cache seems no longer to be working. It is really annoying that the problem is still not solved after such a long time of waiting. There are currently 1,241 pages calling score on en:wikipedia, 748 on de:wikipedia which are endangered of losing the proper display of the contained scores.

And 2,161 on en:Wikisource. And 1,254 pages marked as waiting for a score to be added. In addition, we've got several books that are stalled while waiting for this update.

And 4275 in frwikisource and 1891 in plwikisource. Wikisources are highly struck by this problem.
In plwikisource we have a group of users focused on music transcription and few music transcription projects started a year or two ago. Have we to loose them?
Any hint about a workaround? A userspace workaround?

Just few examples of such projects:
https://pl.wikisource.org/wiki/Wiki%C5%BAr%C3%B3d%C5%82a:Wikiprojekt_Moniuszko_2019
https://pl.wikisource.org/wiki/Autor:Ignacy_Komorowski
https://pl.wikisource.org/wiki/Pie%C5%9Bni_Ludu_Polskiego_w_G%C3%B3rnym_Szl%C4%85sku
https://pl.wikisource.org/wiki/%C5%9Apiewnik_dla_dzieci
https://pl.wikisource.org/wiki/Kantyczki_(Miarka)
https://pl.wikisource.org/wiki/Indeks:%C5%9Apiewnik_ko%C5%9Bcielny_katolicki_(T._Flasza,_1930).djvu

Stemby added a subscriber: Stemby.Nov 17 2020, 9:08 AM
Noe added a subscriber: Noe.Nov 22 2020, 2:45 PM

Hi,

I'm one among a presumably significant group of people around the world trying to learn music during isolation, and I've come across this issue in Wikipedia. I wonder what is the status, since there doesn't seem to be a lot of updates lately.

From my limited technical knowledge about this, I am thinking that if the LilyPond security is very difficult to overcome, anyone with access to all the Wikipedia source (understood as the source anyone can read in the edit page of each article) could run lilypond on an offline machine with no sensitive info and generate an image for each <score> tag, and another simple script could then substitute which <score> tag with an appropriate <img> tag. A special attribute could be added so all these cases can easily be reverted to something else if the <score> issue is solved in the future.

Ankry added a comment.EditedNov 29 2020, 8:22 PM

Hi,

I'm one among a presumably significant group of people around the world trying to learn music during isolation, and I've come across this issue in Wikipedia. I wonder what is the status, since there doesn't seem to be a lot of updates lately.

From my limited technical knowledge about this, I am thinking that if the LilyPond security is very difficult to overcome, anyone with access to all the Wikipedia source (understood as the source anyone can read in the edit page of each article) could run lilypond on an offline machine with no sensitive info and generate an image for each <score> tag, and another simple script could then substitute which <score> tag with an appropriate <img> tag. A special attribute could be added so all these cases can easily be reverted to something else if the <score> issue is solved in the future.

This is not so simple. In Wikisource we convert music scores from images to make them editable and proofread the music scores. So users needs to be able to edit them some way. How are they expected to edit the resulting images if their role is just converting images into an editable form?

See eg. this page:
https://en.wikisource.org/wiki/Page%3AChopin_Nocturnes_Op_9_Kistner_First_Edition_1833.djvu/6
transcluded here:
https://en.wikisource.org/wiki/Chopin_Nocturnes_Opus_9/Number_2

The users' tools were taken away from them and I wonder how long they will wait for getting them back.

This is not so simple. In Wikisource we convert music scores from images to make them editable and proofread the music scores. So users needs to be able to edit them some way. How are they expected to edit the resulting images if their role is just converting images into an editable form?

So the editable format is lilypond? The second link is an ogg file. Maybe you could move to midi or musicxml as editable format?

Anyway, my proposed solution was just meant as a quick and dirty hack so Wikipedia end users for the time being can at least see the scores that are currently in lilypond. These scores are usually small examples anyway, not full pieces, but they are important for end users trying to learn from Wikipedia. Once the lilypond security issue is solved, it could be reverted back to <src> (the src could actually be kept as a comment or deactivated somehow). Some scores in Wikipedia are already in image format, so I guess it wouldn't be that bad (for end users).

Ankry added a comment.Nov 30 2020, 4:06 PM

This is not so simple. In Wikisource we convert music scores from images to make them editable and proofread the music scores. So users needs to be able to edit them some way. How are they expected to edit the resulting images if their role is just converting images into an editable form?

So the editable format is lilypond?

Yes. And the image should not replace it. The lilypond code should be still available in some way in the newest page version.
So, if it is updated, the image may also be updated.

The second link is an ogg file. Maybe you could move to midi or musicxml as editable format?

Anyway, my proposed solution was just meant as a quick and dirty hack so Wikipedia end users for the time being can at least see the scores that are currently in lilypond. These scores are usually small examples anyway, not full pieces, but they are important for end users trying to learn from Wikipedia. Once the lilypond security issue is solved, it could be reverted back to <src> (the src could actually be kept as a comment or deactivated somehow). Some scores in Wikipedia are already in image format, so I guess it wouldn't be that bad (for end users).

Making a dirty hack is better than doing nothing. If lilypond cannot be executed on WMF servers, it may be executed on an external server, in specially prepared secured, chrooted environment, and the image uploaded/updated by a bot.

But at the moment, I see no easy way to access a wiki-parsed lilypond code. It is not accessible. Consider {{#tag:score|...}} usage.

Reedy added a comment.Nov 30 2020, 4:09 PM

If lilypond cannot be executed on WMF servers, it may be executed on an external server, in specially prepared secured, chrooted environment

This is what T260330: RFC: PHP microservice for containerized shell execution is designed to do. But these things take time.

Assuming the problem is the CVE, Debian believes it was fixed in a more recent version, but the reply on the mailing list doesn't exactly instill me with confidence. I think we do need to decide in relatively short order whether the issues have been addressed and we're going to stick with lilypond, or whether it will never be secure enough for our purposes and we need to start transitioning to something else like MusicXML files.

Reedy added a comment.Mon, Dec 28, 8:11 PM

Assuming the problem is the CVE, Debian believes it was fixed in a more recent version, but the reply on the mailing list doesn't exactly instill me with confidence. I think we do need to decide in relatively short order whether the issues have been addressed and we're going to stick with lilypond, or whether it will never be secure enough for our purposes and we need to start transitioning to something else like MusicXML files.

The current proposed (at least for WMF purposes) solution is:

If lilypond cannot be executed on WMF servers, it may be executed on an external server, in specially prepared secured, chrooted environment

This is what T260330: RFC: PHP microservice for containerized shell execution is designed to do. But these things take time.

tstarling, do you know why it currently works so long as the vorbis=1 parameter is removed? Is this intentional or not?

Also, I need to know what "time" means here, because the above fact means that it's possible to restore functionality to the wiki, but at the cost of possibly changing a large number of pages. If it's going to take months for a fix to be in place, that would be reasonable, but not if it's going to be patched next week. Can I have a three-point estimate at least?

Reedy added a comment.Tue, Dec 29, 4:38 PM

You'll have to ask the people working on it. You probably won't get an answer till next week.

It's certainly possible to restore functionality in various ways, but that doesn't mean it's necessarily going to happen just because it might be quicker. I doubt SRE are likely to want to deploy other packages (that are completely unknown at this point) for similar functionality, never mind the code changes needed to the Score extension.

Assuming the problem is the CVE, Debian believes it was fixed in a more recent version, but the reply on the mailing list doesn't exactly instill me with confidence.

From the same thread:

I discussed a plan for rectifying it with Han-Wen, and suggested that we
could contribute funding towards fixing it. However, I was not able to get
approval for funding it. So the task remains open for volunteers to
address. Of course, it is difficult to recruit volunteers when it is a
private security issue.

Pity.

Ayack added a subscriber: Ayack.Mon, Jan 4, 10:35 AM

Change 609467 merged by jenkins-bot:
[mediawiki/extensions/Score@master] Add description to config variables

https://gerrit.wikimedia.org/r/609467