Page MenuHomePhabricator
Create Task
Maniphest T378076

Parsercache issues in codfw causing large-scale outage
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

https://www.wikimediastatus.net/incidents/b406lmnx5s57

I repeatedly got while trying to access https://commons.wikimedia.org/wiki/Commons:Featured_picture_candidates/candidate_list and https://commons.wikimedia.org/wiki/Commons:Administrators%27_noticeboard/Blocks_and_protections (no edit, just reading):

Our servers are currently under maintenance or experiencing a technical problem. Please try again in a few minutes.
See the error message at the bottom of this page for more information.

Request from 78.242.214.114 via cp6013.drmrs.wmnet, ATS/9.2.5
Error: 502, Broken pipe at 2024-10-24 10:56:45 GMT

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
ParserCache: Set connect and recieve timeoutsoperations/mediawiki-configmaster+2 -0
changeprop: parsoidCachePrewarm: halve concurrencyoperations/deployment-chartsmaster+1 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedLadsgroupT378068 pc1017 crashed
 ResolvedPRODUCTION ERRORNoneT378076 Parsercache issues in codfw causing large-scale outage

Event Timeline

Yann created this task.Oct 24 2024, 11:06 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 24 2024, 11:06 AM
Yann updated the task description. (Show Details)Oct 24 2024, 11:06 AM
Yann added a comment.Oct 24 2024, 11:11 AM

And now, while editing https://commons.wikimedia.org/w/index.php?title=Commons:Featured_picture_candidates/File:Calaveras_of_day_of_the_dead_in_mexico.jpg&action=submit

Request from 89.248.174.2 via cp3067.esams.wmnet, ATS/9.2.5
Error: 502, Broken pipe at 2024-10-24 11:09:38 GMT

https://commons.wikimedia.org/w/index.php?title=Commons:Featured_picture_candidates/File:Narrentag-Oberndorf_2024-Nachtumzug-07985.jpg&action=submit

Original error: upstream connect error or disconnect/reset before headers. reset reason: connection termination
Wellverywell subscribed.Oct 24 2024, 11:15 AM

This happens on any page on any wiki trying to open it while logged-in.

Samwalton9-WMF subscribed.Oct 24 2024, 11:32 AM

https://www.wikimediastatus.net/incidents/b406lmnx5s57

CDanis renamed this task from Error: 502, Broken pipe via cp6013.drmrs.wmnet to Parsercache issues in codfw causing large-scale outage.Oct 24 2024, 1:10 PM
CDanis triaged this task as High priority.
CDanis updated the task description. (Show Details)
CDanis added projects: DBA, SRE.
gerritbot added a comment.Oct 24 2024, 1:10 PM

Change #1082778 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/deployment-charts@master] changeprop: parsoidCachePrewarm: halve concurrency

https://gerrit.wikimedia.org/r/1082778

gerritbot added a project: Patch-For-Review.Oct 24 2024, 1:10 PM

Change #1082778 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop: parsoidCachePrewarm: halve concurrency

https://gerrit.wikimedia.org/r/1082778

BTullis subscribed.Oct 24 2024, 1:22 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 24 2024, 1:30 PM
ABran-WMF moved this task from Triage to Refine on the DBA board.Oct 25 2024, 5:28 AM
ABran-WMF added a parent task: T378068: pc1017 crashed.Oct 25 2024, 5:31 AM
ABran-WMF subscribed.
In T378076#10258297, @Samwalton9-WMF wrote:

https://www.wikimediastatus.net/incidents/b406lmnx5s57

this is indeed a side effect of the technical issue we faced yesterday

thcipriani moved this task from Untriaged to Oct 2024 on the Wikimedia-production-error board.Oct 25 2024, 2:52 PM
jcrespo mentioned this in T378385: Spike in JobQueue job backlog time (500ms -> 4-8 minutes).Oct 28 2024, 5:16 PM
jcrespo subscribed.EditedOct 28 2024, 5:21 PM

I linked the above task: T378385: Spike in JobQueue job backlog time (500ms -> 4-8 minutes), as mitigations may have slowed down jobqueue processing. We may need to have a look, as while almost surely was a consequence of the reduced concurrency, it could show some interesting signal, too- as potentially the bottleneck/increased load could have started before the incident.

Jdforrester-WMF subscribed.Oct 28 2024, 5:29 PM
Novem_Linguae subscribed.Oct 28 2024, 7:27 PM
ABran-WMF mentioned this in T378068: pc1017 crashed.Oct 29 2024, 12:39 PM
Ladsgroup closed this task as Resolved.Nov 21 2024, 4:59 PM
Ladsgroup moved this task from Refine to Done on the DBA board.
Ladsgroup subscribed.

The issue is over (and has been over for weeks now), we are changing how parsercache works drastically which would reduce the impact of such issues and outages. It doesn't make sense to keep this ticket open.

gerritbot added a comment.Jan 4 2025, 2:33 AM

Change #1108141 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] ParserCache: Set connect and recieve timeouts

https://gerrit.wikimedia.org/r/1108141

gerritbot added a project: Patch-For-Review.Jan 4 2025, 2:33 AM
Samwalton9-WMF unsubscribed.Jan 6 2025, 10:51 AM
gerritbot added a comment.Jan 6 2025, 12:40 PM

Change #1108141 merged by jenkins-bot:

[operations/mediawiki-config@master] ParserCache: Set connect and recieve timeouts

https://gerrit.wikimedia.org/r/1108141

Stashbot added a comment.Jan 6 2025, 12:40 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-06T12:40:55Z] <ladsgroup@deploy2002> Started scap sync-world: Backport for [[gerrit:1108141|ParserCache: Set connect and recieve timeouts (T378076 T373037)]]

Stashbot mentioned this in T373037: Make ParserCache more like a ring.Jan 6 2025, 12:40 PM
Stashbot added a comment.Jan 6 2025, 12:46 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-06T12:46:34Z] <ladsgroup@deploy2002> ladsgroup: Backport for [[gerrit:1108141|ParserCache: Set connect and recieve timeouts (T378076 T373037)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Jan 6 2025, 12:54 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-06T12:54:34Z] <ladsgroup@deploy2002> Finished scap sync-world: Backport for [[gerrit:1108141|ParserCache: Set connect and recieve timeouts (T378076 T373037)]] (duration: 13m 39s)

Maintenance_bot removed a project: Patch-For-Review.Jan 6 2025, 1:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits