upstream request timeout, http-status 504 in the API
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	Wurgl
	Nov 13 2025, 8:59 AM

Description

Steps to replicate the issue (include links if applicable):

wget -O - "https://de.wikipedia.org/w/api.php?format=json&formatversion=2&action=query&list=search&srsearch=insource%3Ahttps%20insource%3A%2F%5C%5Bhttps%3A%5C%2F%5C%2F%5B%5E%20%5C%5D%5D%2A%27%27%2F&srnamespace=0&srlimit=max&srinfo=&srprop="

What happens?:
--2025-11-13 07:39:49-- https://de.wikipedia.org/w/api.php?format=json&formatversion=2&action=query&list=search&srsearch=insource%3Ahttps%20insource%3A%2F%5C%5Bhttps%3A%5C%2F%5C%2F%5B%5E%20%5C%5D%5D%2A%27%27%2F&srnamespace=0&srlimit=max&srinfo=&srprop=
Resolving de.wikipedia.org (de.wikipedia.org)... 2620:0:861:ed1a::1, 208.80.154.224
Connecting to de.wikipedia.org (de.wikipedia.org)|2620:0:861:ed1a::1|:443... connected.
HTTP request sent, awaiting response... 504 Gateway Timeout
Retrying.

What should have happened instead?:
Retrieve some data

This happens since the 11th of November, did not see the problem before. And it happens with various different search patterns from my home machine (not logged in) and from within the toolforge cloud when logged in with my bot account.

In my PHP-Code I get this answer: {"httpReason":"upstream request timeout","httpCode":504} in addition to status 504

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	rest-gateway: set 53s timeout for action API	operations/deployment-charts	master	+2 -1

Customize query in gerrit

Related Objects

Mentioned In: T412225: 504 Gateway Timeout: httpReason\":\"upstreamrequesttimeout in action api
T410758: Timeouts searching for terms and regular expressions too low
T403212: Support \r, \n, \t, and \uNNNN in insource and intitle queries
T408223: Action API via rest-gateway production rollout
Mentioned Here: T406607: [Hypothesis] 5.2.2b: Action API Rerouting
T408223: Action API via rest-gateway production rollout

Event Timeline

Wurgl created this task.Nov 13 2025, 8:59 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 13 2025, 8:59 AM

Aklapper renamed this task from upstream request timeout, http-status 504 in the API to upstream request timeout, http-status 504 in the API.Nov 13 2025, 9:44 AM

Aklapper updated the task description. (Show Details)

Aklapper added a project: MediaWiki-Action-API.Nov 13 2025, 9:48 AM

Maintenance_bot added a project: MW-Interfaces-Team.Nov 13 2025, 10:30 AM

The problem can be reproduced on the API-Sandox

german:
https://de.wikipedia.org/wiki/Spezial:API-Spielwiese#action=query&format=json&list=search&pageids=1&formatversion=2&srsearch=insource%3Ahttps%20insource%3A%2F%5C%5Bhttps%3A%5C%2F%5C%2F%5B%5E%20%5C%5D%5D*''%2F&srnamespace=0

englisch:
https://en.wikipedia.org/wiki/Special:ApiSandbox#action=query&format=json&list=search&formatversion=2&srsearch=insource%3Ahttps%20insource%3A%2F%5C%5Bhttps%3A%5C%2F%5C%2F%5B%5E%20%5C%5D%5D*''%2F

Some http error shows up, no data.

I have replicated this for the wget and the API Sandbox German. API Sandbox English returns the data. @Wurgl Is Sandbox English broken as well from your end..?

Atieno triaged this task as Medium priority.Nov 14 2025, 10:17 AM

Atieno moved this task from Incoming (Needs Triage) to Bugs & Chores on the MW-Interfaces-Team board.

Yesterday, the english sandbox returned an error. Today it works.

The error on deWP appears always after 15 seconds. Maybe on enWP this time limit is set to a larger number?

doctaxon subscribed.Nov 15 2025, 5:21 PM

Sometimes it's an 503 error instead of 504 error.

One of my more important bot scripts is working with the API Search query and is still needed for important functions in dewiki every hour with any complaints if it's not working. That's why you should allow me to raise the priority to High.

Thank you

@doctaxon: Do you plan to work on fixing this task, as you increased the priority of this task?

Let me know how to point out something important to someone or someone's board other ways than priorities ... I don't know about or am missing it on Phabricator

Pppery added a project: CirrusSearch.Nov 16 2025, 2:34 AM

Restricted Application added a project: Discovery-Search. · View Herald TranscriptNov 16 2025, 2:34 AM

It smells, that the problem is not the search logic, since in https://de.wikipedia.org/wiki/Special:search you can search for insource:https insource:/\[https:\/\/[^ \]]*/ and there are results. The search take ~24 seconds. The same search in the API-Sandbox or with wget fails always after 15,x seconds.

So the reason for this 504 status seems to be in some frontend code which is used in the API but not in Special:search.

@dcausse : it looks like it's something with these merges: https://gerrit.wikimedia.org/r/q/project:mediawiki/extensions/CirrusSearch

It looks like you have insight into the code changes. Can you help with this bug here?

Indeed, the internal timeout should be 50s to allow the regex to run. It is possible that something changed in the request flow that there is now a component failing earlier than the allowed 50s.

In T410007#11377698, @dcausse wrote:

... there is now a component failing earlier than the allowed 50s.

How we can find out the component failing?

• dcausse mentioned this in T408223: Action API via rest-gateway production rollout.Nov 17 2025, 9:29 AM

In T410007#11377822, @doctaxon wrote:

In T410007#11377698, @dcausse wrote:

... there is now a component failing earlier than the allowed 50s.

How we can find out the component failing?

we're looking into it, we'll get some explanation and a fix hopefully soon.

Change #1206205 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] rest-gateway: set 53s timeout for action API

https://gerrit.wikimedia.org/r/1206205

gerritbot added a project: Patch-For-Review.Nov 17 2025, 10:59 AM

Change #1206205 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: set 53s timeout for action API

https://gerrit.wikimedia.org/r/1206205

This issue was happening as a result of the migration of the action API to a common gateway within WMF infrastructure (work ticket: T408223, higher level reasoning/tracking: T406607). We're currently undergoing a slow rollout of wikis by group with the exception of enwiki, which means that all wikis are currently behind the gateway, along with 10% of requests for enwiki. The gateway by default itself imposes a timeout of 15 seconds, which was causing the issue seen here. We've since raised the timeout and the queries in this ticket are now succeeding. Apologies for the disruption.

Maintenance_bot removed a project: Patch-For-Review.Nov 17 2025, 11:31 AM

pfischer edited projects, added Discovery-Search (2025.10.20 - 2025.12.31); removed Discovery-Search.Nov 17 2025, 1:09 PM