Page MenuHomePhabricator

504 Gateway Timeout: httpReason\":\"upstreamrequesttimeout in action api
Open, Needs TriagePublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

  • Query the following revision URLs in the api

What happens?:
A 504 Gateway Timeout:{\"httpCode\":504,\"httpReason\":\"upstreamrequesttimeout\"}" error is returned

What should have happened instead?:

Retrieve the data

Other information (browser name/version, screenshots, etc.):

I believe the error is similar to the one mentioned in https://phabricator.wikimedia.org/T410007

Event Timeline

On https://zh.wikipedia.org/w/rest.php/v1/revision/90601530/html, I see "The maximum execution time of 60 seconds was exceeded" log entries. Here is some error info from the exception logged on the timeout. Maybe isDiffMarker() is slow, or maybe something broader and the timeout happened to occur in that method.

  • mwversion: 1.46.0-wmf.5
  • timestamp: 2025-12-17T17:56:50.523Z
  • phpversion: 8.3.26
  • reqId: 0b9ed7c7-57e1-4485-b340-54f0ff05910b
  • Find reqId in Logstash
normalized_message
[{reqId}] {exception_url}   Wikimedia\RequestTimeout\RequestTimeoutException: The maximum execution time of {limit} seconds was exceeded
FrameLocationCall
from/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/request-timeout/src/Detail/ExcimerTimerWrapper.php(130)
#0/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/request-timeout/src/Detail/ExcimerTimerWrapper.php(105)Wikimedia\RequestTimeout\Detail\ExcimerTimerWrapper->onTimeout(int)
#1/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Html2Wt/DiffUtils.php(226)Wikimedia\RequestTimeout\Detail\ExcimerTimerWrapper->Wikimedia\RequestTimeout\Detail\{closure}(int)
#2/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Utils/DiffDOMUtils.php(184)Wikimedia\Parsoid\Html2Wt\DiffUtils::isDiffMarker(Wikimedia\Parsoid\DOM\Element)
#3/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Utils/WTUtils.php(198)Wikimedia\Parsoid\Utils\DiffDOMUtils::previousNonDeletedSibling(Wikimedia\Parsoid\DOM\Element)
#4/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/DOM/Handlers/HandleLinkNeighbours.php(141)Wikimedia\Parsoid\Utils\WTUtils::findFirstEncapsulationWrapperNode(Wikimedia\Parsoid\DOM\Element)
#5/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Utils/DOMTraverser.php(143)Wikimedia\Parsoid\Wt2Html\DOM\Handlers\HandleLinkNeighbours::handler(Wikimedia\Parsoid\DOM\Element, Wikimedia\Parsoid\Utils\DTState)
#6/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Utils/DOMTraverser.php(226)Wikimedia\Parsoid\Utils\DOMTraverser->callHandlers(Wikimedia\Parsoid\DOM\Element, MediaWiki\Parser\Parsoid\Config\SiteConfig, Wikimedia\Parsoid\Utils\DTState)
#7/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Utils/DOMTraverser.php(240)Wikimedia\Parsoid\Utils\DOMTraverser->traverseInternal(bool, MediaWiki\Parser\Parsoid\Config\SiteConfig, Wikimedia\Parsoid\DOM\Element, Wikimedia\Parsoid\Utils\DTState)
#8/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Utils/DOMTraverser.php(182)Wikimedia\Parsoid\Utils\DOMTraverser->traverseInternal(bool, MediaWiki\Parser\Parsoid\Config\SiteConfig, Wikimedia\Parsoid\DOM\DocumentFragment, Wikimedia\Parsoid\Utils\DTState)
#9/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/DOM/Processors/DOMPPTraverser.php(42)Wikimedia\Parsoid\Utils\DOMTraverser->traverse(MediaWiki\Parser\Parsoid\Config\SiteConfig, Wikimedia\Parsoid\DOM\DocumentFragment, Wikimedia\Parsoid\Utils\DTState)
#10/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/DOMProcessorPipeline.php(131)Wikimedia\Parsoid\Wt2Html\DOM\Processors\DOMPPTraverser->run(Wikimedia\Parsoid\Config\Env, Wikimedia\Parsoid\DOM\DocumentFragment, array, bool)
#11/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/DOMProcessorPipeline.php(175)Wikimedia\Parsoid\Wt2Html\DOMProcessorPipeline->doPostProcess(Wikimedia\Parsoid\DOM\DocumentFragment)
#12/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(127)Wikimedia\Parsoid\Wt2Html\DOMProcessorPipeline->process(Wikimedia\Parsoid\DOM\DocumentFragment, array)
#13/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Utils/PipelineUtils.php(138)Wikimedia\Parsoid\Wt2Html\ParserPipeline->parse(array, array)
#14/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Utils/PipelineUtils.php(461)Wikimedia\Parsoid\Utils\PipelineUtils::processContentInPipeline(Wikimedia\Parsoid\Config\Env, Wikimedia\Parsoid\Wt2Html\PageConfigFrame, array, array)
#15/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Utils/PipelineUtils.php(520)Wikimedia\Parsoid\Utils\PipelineUtils::expandAttrValueToDOM(Wikimedia\Parsoid\Config\Env, Wikimedia\Parsoid\Wt2Html\PageConfigFrame, array, bool, bool)
#16/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/TT/AttributeExpander.php(615)Wikimedia\Parsoid\Utils\PipelineUtils::expandAttrValuesToDOM(Wikimedia\Parsoid\Config\Env, Wikimedia\Parsoid\Wt2Html\PageConfigFrame, array, bool, bool)
#17/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/TT/AttributeExpander.php(671)Wikimedia\Parsoid\Wt2Html\TT\AttributeExpander->buildExpandedAttrs(Wikimedia\Parsoid\Tokens\SelfclosingTagTk, array)
#18/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/TT/AttributeExpander.php(743)Wikimedia\Parsoid\Wt2Html\TT\AttributeExpander->processComplexAttributes(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)
#19/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/TT/UniversalTokenHandler.php(62)Wikimedia\Parsoid\Wt2Html\TT\AttributeExpander->onAny(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)
#20/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/TokenHandlerPipeline.php(116)Wikimedia\Parsoid\Wt2Html\TT\UniversalTokenHandler->process(array)
#21/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/TokenHandlerPipeline.php(177)Wikimedia\Parsoid\Wt2Html\TokenHandlerPipeline->processChunk(array)
#22/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(159)Wikimedia\Parsoid\Wt2Html\TokenHandlerPipeline->processChunkily(array, array)
#23/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(158)Wikimedia\Parsoid\Wt2Html\ParserPipeline->processPipelineStages(array, string, array)
#24/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(158)Wikimedia\Parsoid\Wt2Html\ParserPipeline->processPipelineStages(array, string, array)
#25/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(158)Wikimedia\Parsoid\Wt2Html\ParserPipeline->processPipelineStages(array, string, array)
#26/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(181)Wikimedia\Parsoid\Wt2Html\ParserPipeline->processPipelineStages(array, string, array)
#27/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipelineFactory.php(617)Wikimedia\Parsoid\Wt2Html\ParserPipeline->parseChunkily(string, array)
#28/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Wikitext/ContentModelHandler.php(185)Wikimedia\Parsoid\Wt2Html\ParserPipelineFactory->parse(string)
#29/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Parsoid.php(205)Wikimedia\Parsoid\Wikitext\ContentModelHandler->toDOM(Wikimedia\Parsoid\Ext\ParsoidExtensionAPI, null)
#30/srv/mediawiki/php-1.46.0-wmf.5/vendor/wikimedia/parsoid/src/Parsoid.php(271)Wikimedia\Parsoid\Parsoid->parseWikitext(MediaWiki\Parser\Parsoid\Config\PageConfig, MediaWiki\Parser\ParserOutput, array, null)
#31/srv/mediawiki/php-1.46.0-wmf.5/includes/Parser/Parsoid/ParsoidParser.php(153)Wikimedia\Parsoid\Parsoid->wikitext2html(MediaWiki\Parser\Parsoid\Config\PageConfig, array, null, MediaWiki\Parser\ParserOutput)
#32/srv/mediawiki/php-1.46.0-wmf.5/includes/Parser/Parsoid/ParsoidParser.php(286)MediaWiki\Parser\Parsoid\ParsoidParser->genParserOutput(MediaWiki\Parser\Parsoid\Config\PageConfig, MediaWiki\Parser\ParserOptions, null)
#33/srv/mediawiki/php-1.46.0-wmf.5/includes/Content/WikitextContentHandler.php(375)MediaWiki\Parser\Parsoid\ParsoidParser->parse(string, MediaWiki\Title\Title, MediaWiki\Parser\ParserOptions, bool, bool, int, null)
#34/srv/mediawiki/php-1.46.0-wmf.5/includes/Content/ContentHandler.php(1574)MediaWiki\Content\WikitextContentHandler->fillParserOutput(MediaWiki\Content\WikitextContent, MediaWiki\Content\Renderer\ContentParseParams, MediaWiki\Parser\ParserOutput)
#35/srv/mediawiki/php-1.46.0-wmf.5/includes/Content/Renderer/ContentRenderer.php(67)MediaWiki\Content\ContentHandler->getParserOutput(MediaWiki\Content\WikitextContent, MediaWiki\Content\Renderer\ContentParseParams)
#36/srv/mediawiki/php-1.46.0-wmf.5/includes/Revision/RenderedRevision.php(246)MediaWiki\Content\Renderer\ContentRenderer->getParserOutput(MediaWiki\Content\WikitextContent, MediaWiki\Page\PageIdentityValue, MediaWiki\Revision\RevisionStoreRecord, MediaWiki\Parser\ParserOptions, array)
#37/srv/mediawiki/php-1.46.0-wmf.5/includes/Revision/RenderedRevision.php(219)MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(MediaWiki\Content\WikitextContent, array)
#38/srv/mediawiki/php-1.46.0-wmf.5/includes/Revision/RevisionRenderer.php(225)MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array)
#39/srv/mediawiki/php-1.46.0-wmf.5/includes/Revision/RevisionRenderer.php(158)MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, MediaWiki\Parser\ParserOptions, array)
#40/srv/mediawiki/php-1.46.0-wmf.5/includes/Revision/RenderedRevision.php(182)MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#41/srv/mediawiki/php-1.46.0-wmf.5/includes/Page/ParserOutputAccess.php(591)MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#42/srv/mediawiki/php-1.46.0-wmf.5/includes/Page/ParserOutputAccess.php(679)MediaWiki\Page\ParserOutputAccess->renderRevision(MediaWiki\Page\PageStoreRecord, MediaWiki\Parser\ParserOptions, MediaWiki\Revision\RevisionStoreRecord, array)
#43/srv/mediawiki/php-1.46.0-wmf.5/includes/PoolCounter/PoolCounterWorkViaCallback.php(68)MediaWiki\Page\ParserOutputAccess->MediaWiki\Page\{closure}()
#44/srv/mediawiki/php-1.46.0-wmf.5/includes/PoolCounter/PoolCounterWork.php(159)MediaWiki\PoolCounter\PoolCounterWorkViaCallback->doWork()
#45/srv/mediawiki/php-1.46.0-wmf.5/includes/Page/ParserOutputAccess.php(494)MediaWiki\PoolCounter\PoolCounterWork->execute()
#46/srv/mediawiki/php-1.46.0-wmf.5/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(921)MediaWiki\Page\ParserOutputAccess->getParserOutput(MediaWiki\Page\PageStoreRecord, MediaWiki\Parser\ParserOptions, MediaWiki\Revision\RevisionStoreRecord, array)
#47/srv/mediawiki/php-1.46.0-wmf.5/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(650)MediaWiki\Rest\Handler\Helper\HtmlOutputRendererHelper->getParserOutputInternal()
#48/srv/mediawiki/php-1.46.0-wmf.5/includes/Rest/Handler/Helper/HtmlOutputRendererHelper.php(473)MediaWiki\Rest\Handler\Helper\HtmlOutputRendererHelper->getParserOutput()
#49/srv/mediawiki/php-1.46.0-wmf.5/includes/Rest/Handler/RevisionHTMLHandler.php(75)MediaWiki\Rest\Handler\Helper\HtmlOutputRendererHelper->getHtml()
#50/srv/mediawiki/php-1.46.0-wmf.5/includes/Rest/SimpleHandler.php(41)MediaWiki\Rest\Handler\RevisionHTMLHandler->run(int)
#51/srv/mediawiki/php-1.46.0-wmf.5/includes/Rest/Module/Module.php(472)MediaWiki\Rest\SimpleHandler->execute()
#52/srv/mediawiki/php-1.46.0-wmf.5/includes/Rest/Module/Module.php(301)MediaWiki\Rest\Module\Module->executeHandler(MediaWiki\Rest\Handler\RevisionHTMLHandler)
#53/srv/mediawiki/php-1.46.0-wmf.5/includes/Rest/Router.php(485)MediaWiki\Rest\Module\Module->execute(string, MediaWiki\Rest\RequestFromGlobals)
#54/srv/mediawiki/php-1.46.0-wmf.5/includes/Rest/Router.php(444)MediaWiki\Rest\Router->doExecute(string, MediaWiki\Rest\RequestFromGlobals)
#55/srv/mediawiki/php-1.46.0-wmf.5/includes/Rest/EntryPoint.php(207)MediaWiki\Rest\Router->execute(MediaWiki\Rest\RequestFromGlobals)
#56/srv/mediawiki/php-1.46.0-wmf.5/includes/MediaWikiEntryPoint.php(184)MediaWiki\Rest\EntryPoint->execute()
#57/srv/mediawiki/php-1.46.0-wmf.5/rest.php(25)MediaWiki\MediaWikiEntryPoint->run()
#58/srv/mediawiki/w/rest.php(3)require(string)
#59{main}

The time felt, off so I timed it, and it is off (15 vs 60 seconds)...

time curl -as 'https://zh.wikipedia.org/w/rest.php/v1/revision/90601530/html'
{"httpCode":504,"httpReason":"upstream request timeout"}

real    0m15.305s
user    0m0.086s
sys     0m0.024s

The time felt, off so I timed it, and it is off (15 vs 60 seconds)...

time curl -as 'https://zh.wikipedia.org/w/rest.php/v1/revision/90601530/html'
{"httpCode":504,"httpReason":"upstream request timeout"}

real    0m15.305s
user    0m0.086s
sys     0m0.024s

An upstream layer (presumably Envoy) is timing out after 15 seconds but the PHP request continues to run. After 60 seconds, PHP throws a RequestTimeoutException.

For example, I sent such a request with curl at 04:58:10. It returned an error after 15 seconds but it took until 04:59:11 for the exception log entry to appear.

The response has a server: envoy header.

$ date --utc ; time curl -I https://zh.wikipedia.org/w/rest.php/v1/revision/90601530/html 
Mon 12 Jan 2026 04:58:10 UTC
HTTP/2 504 
content-length: 57
content-type: application/json
date: Mon, 12 Jan 2026 04:58:25 GMT
server: envoy
age: 16
x-cache: cp5023 miss, cp5023 miss
x-cache-status: miss
server-timing: cache;desc="miss", host;desc="cp5023"
strict-transport-security: max-age=106384710; includeSubDomains; preload
report-to: { "group": "wm_nel", "max_age": 604800, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }
nel: { "report_to": "wm_nel", "max_age": 604800, "failure_fraction": 0.05, "success_fraction": 0.0}
set-cookie: WMF-Last-Access=12-Jan-2026;Path=/;HttpOnly;secure;Expires=Fri, 13 Feb 2026 00:00:00 GMT
set-cookie: WMF-Last-Access-Global=12-Jan-2026;Path=/;Domain=.wikipedia.org;HttpOnly;secure;Expires=Fri, 13 Feb 2026 00:00:00 GMT
x-client-ip: 2403:5809:37b3:0:8cd:c498:9ac:b05c
set-cookie: GeoIP=AU:NSW:Sydney:-33.83:151.08:v4; Path=/; secure; Domain=.wikipedia.org
set-cookie: NetworkProbeLimit=0.001;Path=/;Secure;SameSite=None;Max-Age=3600
set-cookie: WMF-Uniq=7v_I90s-XPlzjVmCaU66RwLmAAAAAFvdBGugFuXfzuA6VuTqmRfwpuQ5NCeV24lc;Domain=.wikipedia.org;Path=/;HttpOnly;secure;SameSite=None;Expires=Tue, 12 Jan 2027 00:00:00 GMT
server-timing: WMF-Uniq;fy25-26-we-4-2-hcaptcha-editing=control-2;
x-request-id: 74ae7b12-eac0-44eb-be2f-df50da98d534


real    0m15.591s
user    0m0.081s
sys     0m0.014s

Another instance of the error seen today

"fail_reason": "wmf api call returned status not 200, 206 or 302 for url https://zh.wikipedia.org/w/rest.php/v1/revision/91120893/html? with error 504 Gateway Timeout:{\"httpCode\":504,\"httpReason\":\"upstreamrequesttimeout\"}",