Page MenuHomePhabricator

Extension:Score / Lilypond is disabled on all wikis
Closed, ResolvedPublic

Description

Due to an ongoing security issue, Score/Lilypond have been disabled on Wikimedia wikis for the time being.

This task serves as the public tracking for this issue

Related Objects

StatusSubtypeAssignedTask
Resolvedtstarling

Event Timeline

CDanis created this task.Jul 3 2020, 4:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 3 2020, 4:50 PM
Reedy added projects: Security, Security-Team.
Reedy updated the task description. (Show Details)
Majavah added a subscriber: Majavah.Jul 3 2020, 4:54 PM
Ebe123 triaged this task as High priority.Jul 3 2020, 4:55 PM
Ebe123 added a subscriber: Ebe123.

To be more precise, the error is:

Could not execute LilyPond: /dev/null is not an executable file. Make sure $wgScoreLilyPond is set correctly.

This could be a puppet change which changed the firejail command to something that doesn't work anymore.

Reedy added a subscriber: Reedy.Jul 3 2020, 4:56 PM

To be more precise, the error is:

Could not execute LilyPond: /dev/null is not an executable file. Make sure $wgScoreLilyPond is set correctly.

This could be a puppet change which changed the firejail command to something that doesn't work anymore.

$wgScoreLilyPond has been explicitly set to /dev/null

CDanis added a comment.Jul 3 2020, 4:56 PM

To be more precise, the error is:

Could not execute LilyPond: /dev/null is not an executable file. Make sure $wgScoreLilyPond is set correctly.

This could be a puppet change which changed the firejail command to something that doesn't work anymore.

This is a deliberate change.

Change 609467 had a related patch set uploaded (by Ebe123; owner: Ebe123):
[mediawiki/extensions/Score@master] Clarify use of $wgScoreSafeMode

https://gerrit.wikimedia.org/r/609467

Bugreporter raised the priority of this task from High to Unbreak Now!.Jul 3 2020, 6:25 PM
Dsharpe added a subscriber: Dsharpe.Jul 3 2020, 8:01 PM

An issue is being diagnosed involving this extension, and it will likely remain down until at least Monday, 6 July 2020. We took the functionality down out of an abundance of caution after being made aware of a potential problem. More to come...

DannyS712 lowered the priority of this task from Unbreak Now! to High.Jul 3 2020, 9:42 PM

Move back to high - this is just the public tracking task, the actual bug may be unbreak now but this isn't

mb added a subscriber: mb.Jul 4 2020, 12:43 AM
Krinkle added a subtask: Restricted Task.Jul 4 2020, 1:25 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-04T02:28:09Z] <reedy@deploy1001> Synchronized php-1.35.0-wmf.39/extensions/Score/includes/Score.php: Short circuit lilypond version check to allow usage of cached files T257066 (duration: 00m 55s)

Raymond added a subscriber: Raymond.Jul 4 2020, 9:00 AM
Base added a subscriber: Base.Jul 6 2020, 5:16 AM
Reedy moved this task from Incoming to In Progress on the Security-Team board.Jul 6 2020, 3:27 PM

Has this been fixed? It looks like the errors are gone now and Score is working again.

Reedy changed the status of subtask Restricted Task from Open to Stalled.Jul 6 2020, 3:29 PM
Reedy added a comment.Jul 6 2020, 4:07 PM

Has this been fixed? It looks like the errors are gone now and Score is working again.

No, any that are viewable are because the output is already generated and saved. Any new or modified scores (which don't match a pre-saved render) will not work. So you should be able to copy a working render to another page, but not create a new one

Well, some output did not work two days ago, but it does now. Caching problems, I guess, different users are reporting different errors (and the error category is also just shown in some cases). I will keep the warning then as long as this task is open.

Reedy added a comment.Jul 6 2020, 4:39 PM

Well, some output did not work two days ago, but it does now. Caching problems, I guess, different users are reporting different errors (and the error category is also just shown in some cases). I will keep the warning then as long as this task is open.

Yes, a change was made to fix it where they should be fine to be displayed anyway

Mentioned in SAL (#wikimedia-operations) [2020-07-04T02:28:09Z] <reedy@deploy1001> Synchronized php-1.35.0-wmf.39/extensions/Score/includes/Score.php: Short circuit lilypond version check to allow usage of cached files T257066 (duration: 00m 55s)

DMacks added a subscriber: DMacks.Jul 6 2020, 9:51 PM
MBH added a subscriber: MBH.Jul 7 2020, 1:30 PM
ArielGlenn added a subtask: Restricted Task.Jul 8 2020, 6:50 AM
Tgr added a subscriber: Tgr.Jul 10 2020, 3:23 PM

Have we made an effort to reach out to non-Wikimedia MediaWiki users? Given the severity, warning them well before details of the issue become public seems prudent.

Have we made an effort to reach out to non-Wikimedia MediaWiki users? Given the severity, warning them well before details of the issue become public seems prudent.

I'm not seeing anything within the incident doc indicating this was done, though @Dsharpe could confirm for sure. Looks like there's not quite 50 non-Wikimedia wikis currently using Score according to wikiapiary.

Change 612274 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove lilypond for now

https://gerrit.wikimedia.org/r/612274

Change 612274 merged by Muehlenhoff:
[operations/puppet@production] Remove lilypond for now

https://gerrit.wikimedia.org/r/612274

Mentioned in SAL (#wikimedia-operations) [2020-07-13T14:17:01Z] <moritzm> removing lilypond from production T257066

mb added a comment.Tue, Jul 21, 4:30 AM

This was announced to be fixed by 6 July 2020. It's now been 3 weeks. Is there an ETA for a fix?

Reedy added a comment.Tue, Jul 21, 1:14 PM
In T257066#6321576, @mb wrote:

This was announced to be fixed by 6 July 2020. It's now been 3 weeks. Is there an ETA for a fix?

No it wasn't.

An issue is being diagnosed involving this extension, and it will likely remain down until at least Monday, 6 July 2020. We took the functionality down out of an abundance of caution after being made aware of a potential problem. More to come...

And specifically

will likely remain down until at least Monday, 6 July 2020

ie, this won't be fixed before. Not that it will be fixed on that date.

It will probably be re-enabled in safe mode early next week. Hopefully Monday my time (i.e. Sunday night US time). I don't know how long it will take to restore it in unsafe mode, probably another couple of weeks.

If you're a user, please let us know how much you need this feature, to help us prioritise the work on it.

tstarling closed subtask Restricted Task as Resolved.Wed, Jul 29, 2:34 AM
tstarling closed this task as Resolved.Wed, Jul 29, 11:49 PM
tstarling claimed this task.

Will there be any disclosure of the issue so 3rd party sites that followed suit in disabling it know what's safe?

Will there be any disclosure of the issue so 3rd party sites that followed suit in disabling it know what's safe?

Yes. I planned on putting out an announcement a couple of days ago, but it turns out to be more complicated than expected. There will definitely be an email to mediawiki-announce at some point, and some of the private tasks will be made public.

Yes. I planned on putting out an announcement a couple of days ago, but it turns out to be more complicated than expected. There will definitely be an email to mediawiki-announce at some point, and some of the private tasks will be made public.

Thanks, I'll keep an eye out!

tstarling reopened this task as Open.Fri, Jul 31, 12:48 AM
tstarling reopened subtask Restricted Task as Open.

It's disabled again, since I found a new vulnerability.

tstarling closed this task as Resolved.Tue, Aug 4, 8:39 AM
tstarling closed subtask Restricted Task as Resolved.

We still can't announce anything since we're waiting for vendor security releases. Third party sites should leave lilypond execution disabled.