Page MenuHomePhabricator
Create Task
Maniphest T388646

Fatal DBUnexpectedError: "Database servers in cluster26 are overloaded."
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

I repeatedly get errors while reading files and pages on Commons. Now from https://commons.wikimedia.org/wiki/Commons:Copyright_rules_by_territory/Mexico

[3748e17c-3153-4612-b8a0-ee2f1ed6756a] 2025-03-12 11:15:57: Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError"
Chrome console
Failed to load resource: the server responded with a status of 500 ()
Logstash record
[3748e17c-3153-4612-b8a0-ee2f1ed6756a
Wikimedia\Rdbms\DBUnexpectedError: Database servers in cluster26 are overloaded. In order to protect application servers, the circuit breaking to databases of this section have been activated. Please try again a few seconds.

Related Objects

Event Timeline

Yann created this task.Mar 12 2025, 11:20 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 12 2025, 11:20 AM
matmarex subscribed.Mar 12 2025, 11:30 AM

There has been a brief outage starting about 30 minutes ago, still ongoing. Looking at https://www.wikimediastatus.net:

image.png (211×850 px, 16 KB)

Also reported at https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#c-HandsomeFella-20250312112100-Technical_error

jijiki changed the task status from Open to In Progress.Mar 12 2025, 11:35 AM
jijiki triaged this task as Unbreak Now! priority.
jijiki subscribed.Mar 12 2025, 12:41 PM

This is an ongoing incident, we have been looking into it

jcrespo mentioned this in T388659: MWExceptionRenderer not adding styles for message boxes (will appear unstyled in future MediaWiki version).Mar 12 2025, 12:43 PM
zdev merged a task: T388668: Internal error while trying to edit a Meta wiki page.Mar 12 2025, 1:39 PM
zdev subscribed.
jijiki closed this task as Resolved.Mar 12 2025, 1:55 PM
jijiki claimed this task.

Things are operational, thanks for reporting!

jcrespo reopened this task as Open.Mar 13 2025, 12:08 PM
jcrespo subscribed.

Reopening.

Yann added a comment.Mar 13 2025, 12:20 PM

it happened again now with https://commons.wikimedia.org/w/index.php?title=File:Le_247%C3%A8me_r%C3%A9giment_d%27infanterie_et_son_porte-drapeau,_le_servannais_Alexandre_Miniac,_durant_la_Grande_Guerre..jpg&action=submit

[eb4a7b76-e0ab-4a1c-9899-d1682f15831f] 2025-03-13 12:19:50: Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError"
jcrespo added a comment.Mar 13 2025, 12:24 PM

We are aware and people is working on it, see www.wikimediastatus.org

AntiCompositeNumber merged a task: T388776: DBUnexpectedError when attempting to view certain deleted diffs.Mar 13 2025, 12:47 PM
AntiCompositeNumber subscribed.
thcipriani moved this task from Untriaged to Mar 2025 on the Wikimedia-production-error board.Mar 13 2025, 3:38 PM
zdev added a comment.Mar 14 2025, 1:27 PM

Could it be related to T382699: Page Deletion Fatal exception of type "Wikimedia\Rdbms\DBQueryError" due to lock timeout and its subtask T385809: Protect page actions with advisory locks?

Aklapper added a comment.Mar 14 2025, 1:55 PM

Again: We are aware, people are working on it. Guesses won't help. :)

Volans closed this task as Resolved.Mar 14 2025, 3:24 PM
Volans subscribed.

The root cause of the two incidents that happened when rolling out a new configuration has been found and resolved. An incident report will be available soon. A brief timeline of the two incidents can be found in our status page (March 12th, March 13th).

matmarex mentioned this in T388807: Uncaught "Error: Call to a member function replace() on null" (expecting ReplacementArray, in LanguageConverter).Mar 14 2025, 7:01 PM
matmarex mentioned this in T389001: "LogicException: Process cache for 'en' should be set by now." during session initialization.Mar 18 2025, 1:03 AM
Tgr mentioned this in T389182: startupregistrystats-testwiki fails to run on php-1.44.0-wmf.21.Mar 18 2025, 12:57 PM
LD mentioned this in T390510: Fatal DBUnexpectedError: "Database servers in extension1 are overloaded".Mar 30 2025, 10:40 PM
Krinkle renamed this task from Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError" to Fatal DBUnexpectedError: "Database servers in cluster26 are overloaded.".Mar 31 2025, 3:03 PM
Krinkle updated the task description. (Show Details)
Krinkle moved this task from Mar 2025 to Jan-Mar 2025 on the Wikimedia-production-error board.Apr 1 2025, 2:13 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits