Error
- mwversion: 1.36.0-wmf.32
- reqId: YEDoxzaaHZ-C392vEKPVPgAAAQo
error
Error running '/usr/bin/firejail' [XXX] '……/pygmentize'\' [XXX] mkdir: cannot create directory '/sys/fs/cgroup/memory/mediawiki/job/5': Permission denied limit.sh: failed to create the cgroup.
exception.trace
from /srv/mediawiki/php-1.36.0-wmf.32/vendor/wikimedia/shellbox/src/Command/UnboxedExecutor.php(407)
#0 /srv/mediawiki/php-1.36.0-wmf.32/vendor/wikimedia/shellbox/src/Command/UnboxedCommand.php(29): Shellbox\Command\UnboxedExecutor->execute(MediaWiki\Shell\Command)
#1 /srv/mediawiki/php-1.36.0-wmf.32/includes/shell/Command.php(227): Shellbox\Command\UnboxedCommand->execute()
#2 /srv/mediawiki/php-1.36.0-wmf.32/extensions/SyntaxHighlight_GeSHi/includes/SyntaxHighlight.php(301): MediaWiki\Shell\Command->execute()
#3 /srv/mediawiki/php-1.36.0-wmf.32/includes/libs/objectcache/wancache/WANObjectCache.php(1604): SyntaxHighlight::{closure}(boolean, integer, array, NULL, array)
#4 /srv/mediawiki/php-1.36.0-wmf.32/includes/libs/objectcache/wancache/WANObjectCache.php(1432): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#5 /srv/mediawiki/php-1.36.0-wmf.32/extensions/SyntaxHighlight_GeSHi/includes/SyntaxHighlight.php(310): WANObjectCache->getWithSetCallback(string, integer, Closure)
#6 /srv/mediawiki/php-1.36.0-wmf.32/extensions/SyntaxHighlight_GeSHi/includes/SyntaxHighlight.php(352): SyntaxHighlight::highlightInner(string, string, array)
#7 /srv/mediawiki/php-1.36.0-wmf.32/extensions/SyntaxHighlight_GeSHi/includes/SyntaxHighlight.php(131): SyntaxHighlight::highlight(string, string, array, Parser)
#8 /srv/mediawiki/php-1.36.0-wmf.32/includes/parser/Parser.php(3969): SyntaxHighlight::parserHook(string, array, Parser, PPFrame_Hash)
#9 /srv/mediawiki/php-1.36.0-wmf.32/includes/parser/PPFrame_Hash.php(341): Parser->extensionSubstitution(array, PPFrame_Hash)
#10 /srv/mediawiki/php-1.36.0-wmf.32/includes/parser/Parser.php(2880): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#11 /srv/mediawiki/php-1.36.0-wmf.32/includes/parser/Parser.php(1560): Parser->replaceVariables(string)
#12 /srv/mediawiki/php-1.36.0-wmf.32/includes/parser/Parser.php(653): Parser->internalParse(string)
#13 /srv/mediawiki/php-1.36.0-wmf.32/includes/content/WikitextContent.php(375): Parser->parse(string, Title, ParserOptions, boolean, boolean, integer)
#14 /srv/mediawiki/php-1.36.0-wmf.32/includes/content/AbstractContent.php(591): WikitextContent->fillParserOutput(Title, integer, ParserOptions, boolean, ParserOutput)
#15 /srv/mediawiki/php-1.36.0-wmf.32/includes/Revision/RenderedRevision.php(266): AbstractContent->getParserOutput(Title, integer, ParserOptions, boolean)
#16 /srv/mediawiki/php-1.36.0-wmf.32/includes/Revision/RenderedRevision.php(235): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean)
#17 /srv/mediawiki/php-1.36.0-wmf.32/includes/Revision/RevisionRenderer.php(215): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array)
#18 /srv/mediawiki/php-1.36.0-wmf.32/includes/Revision/RevisionRenderer.php(152): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#19 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#20 /srv/mediawiki/php-1.36.0-wmf.32/includes/Revision/RenderedRevision.php(197): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array)
#21 /srv/mediawiki/php-1.36.0-wmf.32/includes/poolcounter/PoolWorkArticleView.php(137): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#22 /srv/mediawiki/php-1.36.0-wmf.32/includes/poolcounter/PoolCounterWork.php(162): PoolWorkArticleView->doWork()
#23 /srv/mediawiki/php-1.36.0-wmf.32/includes/page/ParserOutputAccess.php(260): PoolCounterWork->execute()
#24 /srv/mediawiki/php-1.36.0-wmf.32/includes/page/Article.php(762): MediaWiki\Page\ParserOutputAccess->getParserOutput(WikiPage, ParserOptions, MediaWiki\Revision\RevisionStoreRecord, integer)
#25 /srv/mediawiki/php-1.36.0-wmf.32/includes/page/Article.php(575): Article->generateContentOutput(User, ParserOptions, integer, OutputPage, array)
#26 /srv/mediawiki/php-1.36.0-wmf.32/includes/actions/ViewAction.php(74): Article->view()
#27 /srv/mediawiki/php-1.36.0-wmf.32/includes/MediaWiki.php(532): ViewAction->show()
#28 /srv/mediawiki/php-1.36.0-wmf.32/includes/MediaWiki.php(316): MediaWiki->performAction(Article, Title)
#29 /srv/mediawiki/php-1.36.0-wmf.32/includes/MediaWiki.php(944): MediaWiki->performRequest()
#30 /srv/mediawiki/php-1.36.0-wmf.32/includes/MediaWiki.php(548): MediaWiki->main()
#31 /srv/mediawiki/php-1.36.0-wmf.32/index.php(53): MediaWiki->run()
#32 /srv/mediawiki/php-1.36.0-wmf.32/index.php(46): wfIndexMain()
#33 /srv/mediawiki/w/index.php(3): require(string)
#34 {main}See also https://logstash.wikimedia.org/goto/755d7bab5da0f7d16044a902d39c9a40.
Impact
Either a fatal error that makes articles uncacheable and unavailable at times, or (ideally, I think is the case) it falls back to rendering without syntax highlighting.
Having said that, looking at various pages that use SyntaxHighlight, those seem to work. So it might be an intermittent issue, yet quite common on a whole. Maybe we're benefiting from Memcached covering the gap for now with stale values being re-newed?
Notes
Seems to have started on 10 Feb 2021 around 15:00 and then ramped up to 100K/hour at 11 Feb 00:00 and has remained around that level since then.
Seems to be distributed over servers equally, including in groups appserver, api_appserver, jobrunner, and parsoid.