Page MenuHomePhabricator

Request timeout in LuaError::getScriptTraceHtml() leads to shared memory corruption, segfault flood
Open, MediumPublicPRODUCTION ERROR

Description

A timeout led to a flood of PHP errors and segfaults on parse1007, which ended when php-fpm automatically restarted after 800ms.

Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\RequestTimeout\RequestTimeoutException: The maximum execution time of {limit} seconds was exceeded
exception.trace
from /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/request-timeout/src/Detail/ExcimerTimerWrapper.php(97)
#0 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/request-timeout/src/Detail/ExcimerTimerWrapper.php(72): Wikimedia\RequestTimeout\Detail\ExcimerTimerWrapper->onTimeout(integer)
#1 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Preprocessor_Hash.php(495): Wikimedia\RequestTimeout\Detail\ExcimerTimerWrapper->Wikimedia\RequestTimeout\Detail\{closure}(integer)
#2 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Preprocessor_Hash.php(137): Preprocessor_Hash->buildDomTreeArrayFromText(string, integer)
#3 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Parser.php(2889): Preprocessor_Hash->preprocessToObj(string, integer)
#4 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Parser.php(4220): Parser->preprocessToDom(string)
#5 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Parser.php(1625): Parser->finalizeHeadings(string, string, boolean)
#6 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Parser.php(700): Parser->internalParse(string)
#7 /srv/mediawiki/php-1.41.0-wmf.17/includes/language/MessageCache.php(1484): Parser->parse(string, MediaWiki\Title\Title, ParserOptions, boolean)
#8 /srv/mediawiki/php-1.41.0-wmf.17/includes/language/Message.php(1428): MessageCache->parse(string, MediaWiki\Title\Title, boolean, boolean, LanguageEn)
#9 /srv/mediawiki/php-1.41.0-wmf.17/includes/language/Message.php(1005): Message->parseText(string)
#10 /srv/mediawiki/php-1.41.0-wmf.17/includes/language/Message.php(1043): Message->format(string)
#11 /srv/mediawiki/php-1.41.0-wmf.17/extensions/Scribunto/includes/Engines/LuaCommon/LuaError.php(85): Message->parse()
#12 /srv/mediawiki/php-1.41.0-wmf.17/extensions/Scribunto/includes/Hooks.php(160): MediaWiki\Extension\Scribunto\Engines\LuaCommon\LuaError->getScriptTraceHtml(array)
#13 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Parser.php(3414): MediaWiki\Extension\Scribunto\Hooks::invokeHook(Parser, PPTemplateFrame_Hash, array)
#14 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Parser.php(3099): Parser->callParserFunction(PPTemplateFrame_Hash, string, array)
#15 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/PPFrame_Hash.php(274): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#16 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Parser.php(3290): PPFrame_Hash->expand(PPNode_Hash_Tree)
#17 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/PPFrame_Hash.php(274): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#18 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Parser.php(3290): PPFrame_Hash->expand(PPNode_Hash_Tree)
#19 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/PPFrame_Hash.php(274): Parser->braceSubstitution(array, PPFrame_Hash)
#20 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Parser.php(2932): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#21 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Parsoid/Config/DataAccess.php(388): Parser->replaceVariables(string, PPFrame_Hash)
#22 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wikitext/Wikitext.php(43): MediaWiki\Parser\Parsoid\Config\DataAccess->preprocessWikitext(MediaWiki\Parser\Parsoid\Config\PageConfig, ParserOutput, string)
#23 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TT/TemplateHandler.php(1035): Wikimedia\Parsoid\Wikitext\Wikitext::preprocess(Wikimedia\Parsoid\Config\Env, string)
#24 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TT/TemplateHandler.php(1078): Wikimedia\Parsoid\Wt2Html\TT\TemplateHandler->expandTemplate(Wikimedia\Parsoid\Wt2Html\TT\TemplateEncapsulator)
#25 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TT/TemplateHandler.php(1123): Wikimedia\Parsoid\Wt2Html\TT\TemplateHandler->onTemplate(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)
#26 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TT/TokenHandler.php(150): Wikimedia\Parsoid\Wt2Html\TT\TemplateHandler->onTag(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)
#27 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(132): Wikimedia\Parsoid\Wt2Html\TT\TokenHandler->process(array)
#28 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(185): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->processChunk(array)
#29 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(140): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->process(array, array)
#30 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Utils/PipelineUtils.php(106): Wikimedia\Parsoid\Wt2Html\ParserPipeline->parse(array, array)
#31 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/Frame.php(142): Wikimedia\Parsoid\Utils\PipelineUtils::processContentInPipeline(Wikimedia\Parsoid\Config\Env, Wikimedia\Parsoid\Wt2Html\Frame, array, array)
#32 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TT/AttributeTransformManager.php(71): Wikimedia\Parsoid\Wt2Html\Frame->expand(array, array)
#33 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TT/AttributeExpander.php(643): Wikimedia\Parsoid\Wt2Html\TT\AttributeTransformManager->process(array)
#34 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TT/AttributeExpander.php(698): Wikimedia\Parsoid\Wt2Html\TT\AttributeExpander->processComplexAttributes(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)
#35 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TT/TokenHandler.php(159): Wikimedia\Parsoid\Wt2Html\TT\AttributeExpander->onAny(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)
#36 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(132): Wikimedia\Parsoid\Wt2Html\TT\TokenHandler->process(array)
#37 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(185): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->processChunk(array)
#38 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(140): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->process(array, array)
#39 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Utils/PipelineUtils.php(106): Wikimedia\Parsoid\Wt2Html\ParserPipeline->parse(string, array)
#40 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Ext/ParsoidExtensionAPI.php(397): Wikimedia\Parsoid\Utils\PipelineUtils::processContentInPipeline(Wikimedia\Parsoid\Config\Env, Wikimedia\Parsoid\Wt2Html\Frame, string, array)
#41 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Ext/ParsoidExtensionAPI.php(975): Wikimedia\Parsoid\Ext\ParsoidExtensionAPI->wikitextToDOM(string, array, boolean)
#42 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Ext/Gallery/Gallery.php(136): Wikimedia\Parsoid\Ext\ParsoidExtensionAPI->renderMedia(string, array, NULL, boolean, boolean)
#43 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Ext/Gallery/Gallery.php(212): Wikimedia\Parsoid\Ext\Gallery\Gallery::pLine(Wikimedia\Parsoid\Ext\ParsoidExtensionAPI, string, integer, Wikimedia\Parsoid\Ext\Gallery\Opts)
#44 [internal function]: Wikimedia\Parsoid\Ext\Gallery\Gallery->Wikimedia\Parsoid\Ext\Gallery\{closure}(array)
#45 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Ext/Gallery/Gallery.php(210): array_map(Closure, array)
#46 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TT/ExtensionHandler.php(94): Wikimedia\Parsoid\Ext\Gallery\Gallery->sourceToDom(Wikimedia\Parsoid\Ext\ParsoidExtensionAPI, string, array)
#47 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TT/ExtensionHandler.php(279): Wikimedia\Parsoid\Wt2Html\TT\ExtensionHandler->onExtension(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)
#48 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TT/TokenHandler.php(150): Wikimedia\Parsoid\Wt2Html\TT\ExtensionHandler->onTag(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)
#49 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(132): Wikimedia\Parsoid\Wt2Html\TT\TokenHandler->process(array)
#50 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(195): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->processChunk(array)
#51 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(193): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->processChunkily(string, array)
#52 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/TreeBuilder/TreeBuilderStage.php(497): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->processChunkily(string, array)
#53 [internal function]: Wikimedia\Parsoid\Wt2Html\TreeBuilder\TreeBuilderStage->processChunkily(string, array)
#54 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/DOMPostProcessor.php(952): Generator->current()
#55 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(180): Wikimedia\Parsoid\Wt2Html\DOMPostProcessor->processChunkily(string, array)
#56 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipelineFactory.php(299): Wikimedia\Parsoid\Wt2Html\ParserPipeline->parseChunkily(string, array)
#57 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Wikitext/ContentModelHandler.php(130): Wikimedia\Parsoid\Wt2Html\ParserPipelineFactory->parse(string)
#58 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Parsoid.php(174): Wikimedia\Parsoid\Wikitext\ContentModelHandler->toDOM(Wikimedia\Parsoid\Ext\ParsoidExtensionAPI)
#59 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Parsoid.php(216): Wikimedia\Parsoid\Parsoid->parseWikitext(MediaWiki\Parser\Parsoid\Config\PageConfig, ParserOutput, array)
#60 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Parsoid/ParsoidOutputAccess.php(298): Wikimedia\Parsoid\Parsoid->wikitext2html(MediaWiki\Parser\Parsoid\Config\PageConfig, array, NULL, ParserOutput)
#61 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Parsoid/ParsoidOutputAccess.php(469): MediaWiki\Parser\Parsoid\ParsoidOutputAccess->parseInternal(MediaWiki\Parser\Parsoid\Config\PageConfig, array)
#62 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Parsoid/ParsoidOutputAccess.php(244): MediaWiki\Parser\Parsoid\ParsoidOutputAccess->parse(MediaWiki\Page\PageStoreRecord, ParserOptions, array, MediaWiki\Revision\RevisionStoreRecord)
#63 /srv/mediawiki/php-1.41.0-wmf.17/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(742): MediaWiki\Parser\Parsoid\ParsoidOutputAccess->getParserOutput(MediaWiki\Page\PageStoreRecord, ParserOptions, MediaWiki\Revision\RevisionStoreRecord, integer)
#64 /srv/mediawiki/php-1.41.0-wmf.17/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(569): MediaWiki\Rest\Handler\Helper\HtmlOutputRendererHelper->getParserOutputInternal(ParserOptions)
#65 /srv/mediawiki/php-1.41.0-wmf.17/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(665): MediaWiki\Rest\Handler\Helper\HtmlOutputRendererHelper->getParserOutput()
#66 /srv/mediawiki/php-1.41.0-wmf.17/includes/Rest/Handler/ParsoidHandler.php(909): MediaWiki\Rest\Handler\Helper\HtmlOutputRendererHelper->getPageBundle()
#67 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/extension/src/Rest/Handler/PageHandler.php(92): MediaWiki\Rest\Handler\ParsoidHandler->wt2html(MediaWiki\Parser\Parsoid\Config\PageConfig, array)
#68 /srv/mediawiki/php-1.41.0-wmf.17/includes/Rest/Router.php(517): MWParsoid\Rest\Handler\PageHandler->execute()
#69 /srv/mediawiki/php-1.41.0-wmf.17/includes/Rest/Router.php(422): MediaWiki\Rest\Router->executeHandler(MWParsoid\Rest\Handler\PageHandler)
#70 /srv/mediawiki/php-1.41.0-wmf.17/includes/Rest/EntryPoint.php(195): MediaWiki\Rest\Router->execute(MediaWiki\Rest\RequestFromGlobals)
#71 /srv/mediawiki/php-1.41.0-wmf.17/includes/Rest/EntryPoint.php(135): MediaWiki\Rest\EntryPoint->execute()
#72 /srv/mediawiki/php-1.41.0-wmf.17/rest.php(31): MediaWiki\Rest\EntryPoint::main()
#73 /srv/mediawiki/w/rest.php(3): require(string)
#74 {main}

Other related errors

MessageCount
[{reqId}] {exception_url} PHP Warning: DOMElement::setAttributeNS() expects parameter 3 to be string, object given203
(104)Connection reset by peer: AH01075: Error dispatching request to :45
AH01067: Failed to read FastCGI header45
[WARNING] [pool www-7.4] child %d exited on signal 11 (SIGSEGV) after %g seconds from start40
[error] Unexpected structure when adding media info.24
[WARNING] failed processes threshold (40 in 60 sec) is reached, initiating reload5

Details

Request URL
https://commons.wikimedia.org/w/rest.php/commons.wikimedia.org/v3/page/pagebundle/User%3ABernd_Schwabe_in_Hannover%2Fgallery/779634554

Event Timeline

Arlolra moved this task from Inbox to Vaguely Related on the RemexHtml board.
Arlolra added a subscriber: tstarling.

If I'm reading this correctly,

#6 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/remex-html/src/TreeBuilder/BeforeHtml.php(33): Wikimedia\RemexHtml\TreeBuilder\TreeBuilder->insertElement(string, Wikimedia\RemexHtml\Tokenizer\PlainAttributes, boolean, integer, integer)

is,

$this->builder->insertElement( 'html', new PlainAttributes,	false,
				$sourceStart, 0 );

However, the throw from,

#1 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/remex-html/src/DOM/DOMBuilder.php(259): DOMElement->setAttributeNS(NULL, string, Wikimedia\RemexHtml\Tokenizer\Attribute)

is happening in a loop of those new PlainAttributes,

foreach ( $element->attrs->getObjects() as $attr ) {

which, you know, there shouldn't be any. That smells like memory corruption.

Looking earlier in the logs, there's an [error] Unexpected structure when adding media info.,
https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2023.07.16?id=j7WzXokBzmHF6YkdaP2q

Before that though, on the same host (parse1007) there's a Wikimedia\RequestTimeout\RequestTimeoutException,
https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-deploy-1-7.0.0-1-2023.07.16?id=-xyzXokBuj8l-JjZaUxD

The trace there indicates it was also during rendering of a gallery,

#39 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Utils/PipelineUtils.php(106): Wikimedia\Parsoid\Wt2Html\ParserPipeline->parse(string, array)
#40 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Ext/ParsoidExtensionAPI.php(397): Wikimedia\Parsoid\Utils\PipelineUtils::processContentInPipeline(Wikimedia\Parsoid\Config\Env, Wikimedia\Parsoid\Wt2Html\Frame, string, array)
#41 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Ext/ParsoidExtensionAPI.php(975): Wikimedia\Parsoid\Ext\ParsoidExtensionAPI->wikitextToDOM(string, array, boolean)
#42 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Ext/Gallery/Gallery.php(136): Wikimedia\Parsoid\Ext\ParsoidExtensionAPI->renderMedia(string, array, NULL, boolean, boolean)
#43 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Ext/Gallery/Gallery.php(212): Wikimedia\Parsoid\Ext\Gallery\Gallery::pLine(Wikimedia\Parsoid\Ext\ParsoidExtensionAPI, string, integer, Wikimedia\Parsoid\Ext\Gallery\Opts)
#44 [internal function]: Wikimedia\Parsoid\Ext\Gallery\Gallery->Wikimedia\Parsoid\Ext\Gallery\{closure}(array)
#45 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/parsoid/src/Ext/Gallery/Gallery.php(210): array_map(Closure, array)

Similar to last time (T293568#7483468), this implicates Excimer

from /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/request-timeout/src/Detail/ExcimerTimerWrapper.php(97)
#0 /srv/mediawiki/php-1.41.0-wmf.17/vendor/wikimedia/request-timeout/src/Detail/ExcimerTimerWrapper.php(72): Wikimedia\RequestTimeout\Detail\ExcimerTimerWrapper->onTimeout(integer)
#1 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Preprocessor_Hash.php(495): Wikimedia\RequestTimeout\Detail\ExcimerTimerWrapper->Wikimedia\RequestTimeout\Detail\{closure}(integer)
#2 /srv/mediawiki/php-1.41.0-wmf.17/includes/parser/Preprocessor_Hash.php(137): Preprocessor_Hash->buildDomTreeArrayFromText(string, integer)

/cc @tstarling

Arlolra triaged this task as Medium priority.
Arlolra added a subscriber: Arlolra.

Most likely opcache corruption. Errors started only 1ms after the request timeout at 12:36:03.013, and then there was a flood of segfaults over the next few hundred milliseconds, finishing with "[WARNING] failed processes threshold (40 in 60 sec) is reached, initiating reload" at 12:36:03.800.

It's unlikely the subsequent requests will help us to isolate the bug. I'll update the task description to show details of the timeout request.

The only slightly unusual thing in the timeout stack trace is that it's in a catch block (frame #12), catching a LuaError.

If there was a timeout in Lua, it would have thrown a LuaSandboxTimeoutError, which would have been caught and rethrown as a ScribuntoException. But we see it formatting a LuaError which suggests that there was some other error from Lua.

It's in a Lua module which was invoked from a template which was in a gallery. The page was [[commons:User:Bernd Schwabe in Hannover/gallery]].

tstarling renamed this task from PHP Warning: DOMElement::setAttributeNS() expects parameter 3 to be string, object given to Request timeout in LuaError::getScriptTraceHtml() leads to shared memory corruption, segfault flood.Aug 25 2023, 2:01 AM
tstarling claimed this task.
tstarling changed Request URL from https://commons.wikimedia.org/w/rest.php/commons.wikimedia.org/v3/page/pagebundle/File%3ABrezina_-_Brunelli(...)_-_NARA_-_108875456_(page_1133).jpg/783692778 to https://commons.wikimedia.org/w/rest.php/commons.wikimedia.org/v3/page/pagebundle/User%3ABernd_Schwabe_in_Hannover%2Fgallery/779634554.
tstarling updated the task description. (Show Details)