Page MenuHomePhabricator
Create Task
Maniphest T398448

HTTP 503 errors trying to reach Wikipedia: 2025-07-02 s4 overload
Closed, ResolvedPublicBUG REPORT
Actions

Description

Steps to replicate the issue (include links if applicable):

  • Try to edit any ruwiki page.
  • Observe 503 error.

This happens since a few minutes ago. It's also noticeable on Wikimedia Status:

image.png (711×869 px, 51 KB)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
ApiQueryCategoryMembers: Use correct index for categorylinksmediawiki/corewmf/1.45.0-wmf.8+5 -1
ApiQueryCategoryMembers: Use correct index for categorylinksmediawiki/coreREL1_44+5 -1
ApiQueryCategoryMembers: Use correct index for categorylinksmediawiki/coremaster+5 -1
Customize query in gerrit

Related Objects

Event Timeline

Wellverywell created this task.Jul 2 2025, 2:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 2 2025, 2:02 PM
Willy1018 subscribed.Jul 2 2025, 2:10 PM

AWB seems to be affected. I can't make any edits through it now.

Draceane subscribed.Jul 2 2025, 2:11 PM

I have noticed Error 429 while running MassRename JS on Commons + autocomplete in the searchbox isn't working.

Aklapper renamed this task from Cannot edit Wikipedia to HTTP 503 errors trying to reach Wikipedia.Jul 2 2025, 2:29 PM
Aklapper edited projects, added SRE; removed Editing-team.
Don-vip awarded a token.Jul 2 2025, 2:36 PM
Aklapper renamed this task from HTTP 503 errors trying to reach Wikipedia to HTTP 503 errors trying to reach Wikipedia: 2025-07-02 s4 overload.Jul 2 2025, 2:52 PM
Aklapper mentioned this in T398450: End user reported "Error: 503, Backend fetch failed" contacting an unspecified Beta Cluster hosted wiki.
gerritbot added a comment.Jul 2 2025, 3:29 PM

Change #1165923 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/core@master] ApiQueryCategoryMembers: Use correct index for categorylinks

https://gerrit.wikimedia.org/r/1165923

gerritbot added a project: Patch-For-Review.Jul 2 2025, 3:29 PM
Zabe subscribed.Jul 2 2025, 3:31 PM
Clement_Goubert subscribed.Jul 2 2025, 4:33 PM

For the record, this is this incident https://www.wikimediastatus.net/incidents/57jsxtn7hlvf

Clement_Goubert triaged this task as Medium priority.Jul 2 2025, 4:37 PM

Incident is resolved, setting medium priority for follow-up.

gerritbot added a comment.Jul 2 2025, 5:11 PM

Change #1165923 merged by jenkins-bot:

[mediawiki/core@master] ApiQueryCategoryMembers: Use correct index for categorylinks

https://gerrit.wikimedia.org/r/1165923

Maintenance_bot removed a project: Patch-For-Review.Jul 2 2025, 5:31 PM
ReleaseTaggerBot added a project: MW-1.45-notes (1.45.0-wmf.9; 2025-07-08).Jul 2 2025, 6:00 PM
Aklapper added a project: Wikimedia-Incident.Jul 2 2025, 6:18 PM
gerritbot added a comment.Jul 2 2025, 9:24 PM

Change #1165996 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/core@wmf/1.45.0-wmf.8] ApiQueryCategoryMembers: Use correct index for categorylinks

https://gerrit.wikimedia.org/r/1165996

gerritbot added a project: Patch-For-Review.Jul 2 2025, 9:24 PM
gerritbot added a comment.Jul 2 2025, 9:26 PM

Change #1165997 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/core@REL1_44] ApiQueryCategoryMembers: Use correct index for categorylinks

https://gerrit.wikimedia.org/r/1165997

gerritbot added a comment.Jul 2 2025, 9:51 PM

Change #1165997 merged by jenkins-bot:

[mediawiki/core@REL1_44] ApiQueryCategoryMembers: Use correct index for categorylinks

https://gerrit.wikimedia.org/r/1165997

ReleaseTaggerBot added a project: MW-1.44-notes.Jul 2 2025, 10:00 PM
gerritbot added a comment.Jul 2 2025, 10:16 PM

Change #1165996 merged by jenkins-bot:

[mediawiki/core@wmf/1.45.0-wmf.8] ApiQueryCategoryMembers: Use correct index for categorylinks

https://gerrit.wikimedia.org/r/1165996

Stashbot added a comment.Jul 2 2025, 10:17 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-02T22:17:47Z] <zabe@deploy1003> Started scap sync-world: Backport for [[gerrit:1165996|ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]]

Stashbot mentioned this in T385890: Add support for read new for categorylinks migration.Jul 2 2025, 10:17 PM
Stashbot added a comment.Jul 2 2025, 10:19 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-02T22:19:56Z] <zabe@deploy1003> zabe: Backport for [[gerrit:1165996|ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Jul 2 2025, 10:27 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-02T22:27:00Z] <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1165996|ApiQueryCategoryMembers: Use correct index for categorylinks (T385890 T398448)]] (duration: 09m 12s)

Maintenance_bot removed a project: Patch-For-Review.Jul 2 2025, 10:30 PM
ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.8; 2025-07-01); removed MW-1.45-notes (1.45.0-wmf.9; 2025-07-08).Jul 2 2025, 11:00 PM
Clement_Goubert added projects: serviceops, DBA.Jul 3 2025, 1:30 PM
Ladsgroup mentioned this in T398693: Make sure query killer is installed on all replicas.Jul 4 2025, 10:25 AM
Ladsgroup mentioned this in T398692: Investigate circuit breaker for s4 incident.
Ladsgroup closed this task as Resolved.Jul 4 2025, 10:52 AM
Ladsgroup moved this task from Triage to Done on the DBA board.
Ladsgroup subscribed.

I filed T398693: Make sure query killer is installed on all replicas and T398692: Investigate circuit breaker for s4 incident as follow ups. Closing this. Feel free to add more follow ups if I missed something.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits