Page MenuHomePhabricator

Unexpected syntax highlight error
Closed, ResolvedPublicBUG REPORT

Description

Expected behavior

The following code sample renders with syntax highlighting on a wiki with SyntaxHighlight enabled.

<syntaxhighlight lang="json">
{
  "key": "value"
}
</syntaxhighlight>

Observed behavior

On this page on the API Portal, this sample is not highlighted and the page is marked as having a syntax highlight error.

Event Timeline

DannyS712 added a subscriber: DannyS712.

It seems to affect any input that is not already cached? Eg https://api.wikimedia.org/wiki/API_reference/Core/Media_files/File_object has json that is highlighted, but changing anything in that json results in it no longer working when previewing the edit

Can reproduce on mediawiki.org - copying a snippet of highlighted code from, eg, https://www.mediawiki.org/wiki/Extension:SyntaxHighlight, works, but changing anything in that input results in it no longer working (see https://www.mediawiki.org/w/index.php?title=User:DannyS712/sandbox&oldid=4402108 for a demo - the top is copied from existing wikitext that works, and changing a single character results in it not working anymore)

Legoktm triaged this task as Unbreak Now! priority.Feb 11 2021, 6:33 AM
Legoktm added subscribers: tstarling, BPirkle, Legoktm.

Probably caused by the Shellbox rollout

Error running '/usr/bin/firejail' '--quiet' '--profile=/srv/mediawiki/php-1.36.0-wmf.30/includes/shell/firejail.profile' '--whitelist=/srv/mediawiki/php-1.36.0-wmf.30/vendor/wikimedia/shellbox/src/Command/limit.sh' '--blacklist=/srv/mediawiki/php-1.36.0-wmf.30/LocalSettings.php' '--noroot' '--seccomp' '--private-dev' '--net=none' -- /bin/bash '/srv/mediawiki/php-1.36.0-wmf.30/vendor/wikimedia/shellbox/src/Command/limit.sh' ''\''/srv/mediawiki/php-1.36.0-wmf.30/extensions/SyntaxHighlight_GeSHi/includes/../pygments/pygmentize'\'' '\''-l'\'' '\''sparql'\'' '\''-f'\'' '\''html'\'' '\''-O'\'' '\''cssclass=mw-highlight,encoding=utf-8'\''' 'SB_INCLUDE_STDERR=;SB_CPU_LIMIT=50; SB_CGROUP='\''/sys/fs/cgroup/memory/mediawiki/job'\''; SB_MEM_LIMIT=1073741824; SB_FILE_SIZE_LIMIT=536870912; SB_WALL_CLOCK_LIMIT=180; SB_USE_LOG_PIPE=yes': mkdir: cannot create directory '/sys/fs/cgroup/memory/mediawiki/job/5': Permission denied
limit.sh: failed to create the cgroup.
/bin/bash: /srv/mediawiki/php-1.36.0-wmf.30/extensions/SyntaxHighlight_GeSHi/includes/../pygments/pygmentize: No such file or directory

The useful part of the stacktrace is:

from /srv/mediawiki/php-1.36.0-wmf.30/vendor/wikimedia/shellbox/src/Command/UnboxedExecutor.php(407)
#0 /srv/mediawiki/php-1.36.0-wmf.30/vendor/wikimedia/shellbox/src/Command/UnboxedCommand.php(29): Shellbox\Command\UnboxedExecutor->execute(MediaWiki\Shell\Command)
#1 /srv/mediawiki/php-1.36.0-wmf.30/includes/shell/Command.php(227): Shellbox\Command\UnboxedCommand->execute()
#2 /srv/mediawiki/php-1.36.0-wmf.30/extensions/SyntaxHighlight_GeSHi/includes/SyntaxHighlight.php(301): MediaWiki\Shell\Command->execute()
#3 /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/objectcache/wancache/WANObjectCache.php(1565): SyntaxHighlight::{closure}(boolean, integer, array, NULL, array)
#4 /srv/mediawiki/php-1.36.0-wmf.30/includes/libs/objectcache/wancache/WANObjectCache.php(1392): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#5 /srv/mediawiki/php-1.36.0-wmf.30/extensions/SyntaxHighlight_GeSHi/includes/SyntaxHighlight.php(310): WANObjectCache->getWithSetCallback(string, integer, Closure)
#6 /srv/mediawiki/php-1.36.0-wmf.30/extensions/SyntaxHighlight_GeSHi/includes/SyntaxHighlight.php(352): SyntaxHighlight::highlightInner(string, string, array)
#7 /srv/mediawiki/php-1.36.0-wmf.30/extensions/SyntaxHighlight_GeSHi/includes/SyntaxHighlight.php(131): SyntaxHighlight::highlight(string, string, array, Parser)
#8 /srv/mediawiki/php-1.36.0-wmf.30/includes/parser/Parser.php(3929): SyntaxHighlight::parserHook(string, array, Parser, PPTemplateFrame_Hash)

I double checked, the pygmentize file does exist:

legoktm@mw1337:~$ file /srv/mediawiki/php-1.36.0-wmf.30/extensions/SyntaxHighlight_GeSHi/includes/../pygments/pygmentize
/srv/mediawiki/php-1.36.0-wmf.30/extensions/SyntaxHighlight_GeSHi/includes/../pygments/pygmentize: Zip archive data, K\003\004\024

On the first error message, this seems seems important but is easy to miss:

mkdir: cannot create directory '/sys/fs/cgroup/memory/mediawiki/job/5': Permission denied

On the first error message, this seems seems important but is easy to miss:

mkdir: cannot create directory '/sys/fs/cgroup/memory/mediawiki/job/5': Permission denied

I think it's all related...if I disable the cgroup part, it still fails that it can't find pygmentize:

legoktm@mwmaint1002:~$ sudo -u www-data '/usr/bin/firejail' '--profile=/srv/mediawiki/php-1.36.0-wmf.30/includes/shell/firejail.profile' '--whitelist=/srv/mediawiki/php-1.36.0-wmf.30/vendor/wikimedia/shellbox/src/Command/limit.sh' '--blacklist=/srv/mediawiki/php-1.36.0-wmf.30/LocalSettings.php' '--noroot' '--private-dev' '--net=none' -- /bin/bash '/srv/mediawiki/php-1.36.0-wmf.30/vendor/wikimedia/shellbox/src/Command/limit.sh' ''\''/srv/mediawiki/php-1.36.0-wmf.30/extensions/SyntaxHighlight_GeSHi/includes/../pygments/pygmentize'\'' '\''-l'\'' '\''sparql'\'' '\''-f'\'' '\''html'\'' '\''-O'\'' '\''cssclass=mw-highlight,encoding=utf-8'\''' 'SB_INCLUDE_STDERR=;SB_CPU_LIMIT=50; SB_MEM_LIMIT=1073741824; SB_FILE_SIZE_LIMIT=536870912; SB_WALL_CLOCK_LIMIT=180; SB_USE_LOG_PIPE=yes'
Reading profile /srv/mediawiki/php-1.36.0-wmf.30/includes/shell/firejail.profile
Reading profile /etc/firejail/mediawiki.local
Parent pid 19794, child pid 19795
Child process initialized
/bin/bash: /srv/mediawiki/php-1.36.0-wmf.30/extensions/SyntaxHighlight_GeSHi/includes/../pygments/pygmentize: No such file or directory

(I removed --quiet from firejail)

OK, the problem is '--whitelist=/srv/mediawiki/php-1.36.0-wmf.30/vendor/wikimedia/shellbox/src/Command/limit.sh'. Removing that fixes the cgroup and no such file or directory issue. I need to re-read the firejail docs, but I bet moving limit.sh from includes/ to vendor/ broke it.

And SyntaxHighlight is the only command with the executable in the MediaWiki tree IIRC, so other stuff isn't affected.

As I wrote in T182486: Command::whitelistPaths() with firejail doesn't work exactly as expected:

When firejail gets --whitelist=/srv/mediawiki/core/includes/shell/limit.sh, it will hide everything in /srv except for the whitelisted file. Except it leaves anything outside of /srv fully accessible.

The pre-shellbox FirejailCommand implementation was:

		if ( $this->whitelistedPaths ) {
			// Always whitelist limit.sh
			$cmd[] = '--whitelist=' . __DIR__ . '/limit.sh';
			foreach ( $this->whitelistedPaths as $whitelistedPath ) {
				$cmd[] = "--whitelist={$whitelistedPath}";
			}
		}

So we never actually whitelisted limit.sh. The Shellbox implementation unconditionally whitelists it, which it shouldn't do.

Change 663384 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/libs/Shellbox@master] Don't unconditionally allowPath( 'limit.sh' )

https://gerrit.wikimedia.org/r/663384

Change 663448 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/vendor@master] wikimedia/shellbox: Don't unconditionally allowPath( 'limit.sh' )

https://gerrit.wikimedia.org/r/663448

Change 663388 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/vendor@wmf/1.36.0-wmf.30] wikimedia/shellbox: Don't unconditionally allowPath( 'limit.sh' )

https://gerrit.wikimedia.org/r/663388

Change 663384 merged by jenkins-bot:
[mediawiki/libs/Shellbox@master] Don't unconditionally allowPath( 'limit.sh' )

https://gerrit.wikimedia.org/r/663384

Change 663388 merged by jenkins-bot:
[mediawiki/vendor@wmf/1.36.0-wmf.30] wikimedia/shellbox: Don't unconditionally allowPath( 'limit.sh' )

https://gerrit.wikimedia.org/r/663388

Mentioned in SAL (#wikimedia-operations) [2021-02-11T08:11:06Z] <legoktm@deploy1001> Synchronized php-1.36.0-wmf.30/vendor/wikimedia/shellbox/src/Command/BashWrapper.php: wikimedia/shellbox: Don't unconditionally allowPath( 'limit.sh' ) - T274474 (duration: 01m 32s)

Legoktm added subscribers: Suzukaze-c, Erutuon.

This should be fixed in production, I just need to do a Shellbox release and fix in master now.

T274476 might be a duplicate.

Yep, just tested the page and now it works.

Note that you'll need to ?action=purge any page that is currently displaying the non-highlighted syntax.

Change 663520 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/vendor@master] Update wikimedia/shellbox to 1.0.3

https://gerrit.wikimedia.org/r/663520

Change 663521 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/core@master] Update wikimedia/shellbox to 1.0.3

https://gerrit.wikimedia.org/r/663521

Change 663448 abandoned by Legoktm:
[mediawiki/vendor@master] wikimedia/shellbox: Don't unconditionally allowPath( 'limit.sh' )

Reason:
Superseded by I0740696953b828b295318433388a28d868384f02
for master.

https://gerrit.wikimedia.org/r/663448

Change 663520 merged by ArielGlenn:
[mediawiki/vendor@master] Update wikimedia/shellbox to 1.0.3

https://gerrit.wikimedia.org/r/663520

Change 663521 merged by jenkins-bot:
[mediawiki/core@master] Update wikimedia/shellbox to 1.0.3

https://gerrit.wikimedia.org/r/663521

On the first error message, this seems seems important but is easy to miss:

mkdir: cannot create directory '/sys/fs/cgroup/memory/mediawiki/job/5': Permission denied

I think it's all related...if I disable the cgroup part, it still fails that it can't find pygmentize:

legoktm@mwmaint1002:~$ sudo -u www-data '/usr/bin/firejail' '--profile=/srv/mediawiki/php-1.36.0-wmf.30/includes/shell/firejail.profile' '--whitelist=/srv/mediawiki/php-1.36.0-wmf.30/vendor/wikimedia/shellbox/src/Command/limit.sh' '--blacklist=/srv/mediawiki/php-1.36.0-wmf.30/LocalSettings.php' '--noroot' '--private-dev' '--net=none' -- /bin/bash '/srv/mediawiki/php-1.36.0-wmf.30/vendor/wikimedia/shellbox/src/Command/limit.sh' ''\''/srv/mediawiki/php-1.36.0-wmf.30/extensions/SyntaxHighlight_GeSHi/includes/../pygments/pygmentize'\'' '\''-l'\'' '\''sparql'\'' '\''-f'\'' '\''html'\'' '\''-O'\'' '\''cssclass=mw-highlight,encoding=utf-8'\''' 'SB_INCLUDE_STDERR=;SB_CPU_LIMIT=50; SB_MEM_LIMIT=1073741824; SB_FILE_SIZE_LIMIT=536870912; SB_WALL_CLOCK_LIMIT=180; SB_USE_LOG_PIPE=yes'
Reading profile /srv/mediawiki/php-1.36.0-wmf.30/includes/shell/firejail.profile
Reading profile /etc/firejail/mediawiki.local
Parent pid 19794, child pid 19795
Child process initialized
/bin/bash: /srv/mediawiki/php-1.36.0-wmf.30/extensions/SyntaxHighlight_GeSHi/includes/../pygments/pygmentize: No such file or directory

(I removed --quiet from firejail)

The cgroup failure seems to be entirely different, filed as T274942: "mkdir: cannot create directory '/sys/fs/cgroup/memory/mediawiki/job/5': Permission denied" "limit.sh: failed to create the cgroup.".