upstream connect error or disconnect/reset before headers. reset reason: overflow
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	alaa
	Feb 10 2022, 7:33 PM

Description

Update: 2022-03-10:

Please read:

If you ended up searching and finding this task because of a

upstream connect error or disconnect/reset before headers. reset reason: overflow

error message that you received, please know that this is a symptom of a number of different problems that can appear lower in our tech stack. It is not a cause in itself and doesn't help in pinpointing what is happening. It only tells us something is probably happening. Chances are that the SRE team is probably well aware already and are coordinating in IRC in the #wikimedia-operations channel to investigate the actual problem.

Original task text - 2022-02-10

I'm facing this error (a lot of times) when I'm trying to reach English Wikipedia page(s).

Started on 10 February 2022, 19:29 UTC

upstream connect error or disconnect/reset before headers. reset reason: overflow

Related Objects
Search...

		Status	Subtype	Assigned	Task
					Restricted Task
		Resolved		akosiaris	T301505 upstream connect error or disconnect/reset before headers. reset reason: overflow

Event Timeline

alaa created this task.Feb 10 2022, 7:33 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 10 2022, 7:33 PM

iamrunner subscribed.Feb 10 2022, 7:33 PM

alaa added a project: Wikimedia-Incident.Feb 10 2022, 7:34 PM

It's intermittent for me, but I can reproduce.

alaa updated the task description. (Show Details)Feb 10 2022, 7:35 PM

Zabe subscribed.Feb 10 2022, 7:35 PM

Same for me; but was going slow for some minues prior.

alaa updated the task description. (Show Details)Feb 10 2022, 7:36 PM

Hello everyone, this is a known incident and it is being worked on by Wikimedia Foundation staff. Thank you for your patience.

MBH subscribed.Feb 10 2022, 7:37 PM

ppelberg subscribed.Feb 10 2022, 7:42 PM

Ladsgroup added a project: SRE.Feb 10 2022, 7:44 PM

Ladsgroup subscribed.

Daimona subscribed.Feb 10 2022, 7:49 PM

NguoiDungKhongDinhDanh subscribed.Feb 10 2022, 7:54 PM

Ladsgroup added a parent task: Restricted Task.Feb 10 2022, 8:00 PM

We think this is resolved now. If you still can't access the wikis, please let us know.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptFeb 10 2022, 8:04 PM

We think this is resolved now. If you still can't access the wikis, please let us know.

Maintenance_bot moved this task from Incoming to Done on the User-Ladsgroup board.Feb 10 2022, 8:15 PM

iamrunner unsubscribed.Feb 10 2022, 9:14 PM

Not resolved yet.

Saw it again at 18:45, 11 February 2022 (UTC) when trying to save an edit to https://commons.wikimedia.org/wiki/Commons:Administrators/Inactivity_section/Feb-Mar_2022.

I encountered it again just now.

NguoiDungKhongDinhDanh added a comment.Feb 11 2022, 6:48 PM

This comment was removed by NguoiDungKhongDinhDanh.

TheresNoTime subscribed.Feb 11 2022, 6:50 PM

I believe that one took only five minutes and Wikipedia has been accessible after that. So I close this again. Thank you for understanding.

This issue is still happening.

RLazarus claimed this task.Feb 17 2022, 2:18 AM

Thibaut120094 subscribed.Feb 24 2022, 2:43 AM

Stang subscribed.Feb 24 2022, 3:12 AM

This is happening again to me and others at enwiki. Reported at https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Thursday?

Izno subscribed.Feb 24 2022, 3:37 AM

Thanks for letting us know! We did indeed have this issue again for a few minutes earlier (intermittently between 02:36 and 03:00 UTC) but things are back to normal now. Sorry for the inconvenience, and more permanent solutions are in progress to keep this from happening again.

• MZMcBride triaged this task as High priority.Feb 25 2022, 2:04 AM

Happened again between 12:25 and 12:28 UTC but things are back to normal now.

Func merged a task: T302664: Site back 503.Feb 26 2022, 1:08 PM

Func added a subscriber: Jonathan5566.

Where is documentation of this issue? The returns make think it is not resolved.

Hi! This resurfaced during the weekend. It is not a single issue (despite appearances), rather the message "upstream connect error or disconnect/reset before headers. reset reason: overflow" that we see can be a symptom of various causes. It is not resolved yet and has a chance of reappearing over the next few days/weeks while we work on what we hope will be a set of more permanent solutions.

In T301505#7742137, @akosiaris wrote:

Hi! This resurfaced during the weekend. It is not a single issue (despite appearances), rather the message "upstream connect error or disconnect/reset before headers. reset reason: overflow" that we see can be a symptom of various causes. It is not resolved yet and has a chance of reappearing over the next few days/weeks while we work on what we hope will be a set of more permanent solutions.

Hi. :-) Thank you for the update and clarification. I just encountered this error again just now at https://en.wikipedia.org/wiki/Special:Contributions/OldEagles.

We had a occurrence of this a couple of hours ago, will post more details soon.

In T301505#7766564, @akosiaris wrote:

We had a occurrence of this a couple of hours ago, will post more details soon.

As promised: https://wikitech.wikimedia.org/wiki/Incident_documentation/2022-03-10_MediaWiki_availability

Reading through the incident doc makes it obvious that the error pasted here is a symptom (and one in fact multiple layers away from the actual causes of the issue) and not the reason for this. The same error can be emitted due to other reasons as well and is not limited to the database layer malfunctioning.

I am going to alter the text of the task to reflect the above and lower priority due to that.

akosiaris lowered the priority of this task from High to Low.Mar 10 2022, 10:44 AM

akosiaris updated the task description. (Show Details)

Lomrjyo subscribed.Mar 12 2022, 12:45 PM

PorkchopGMX subscribed.Mar 14 2022, 7:02 PM

StevenSun subscribed.Mar 27 2022, 2:13 PM

akosiaris mentioned this in T307647: 2022-05-05 Wikimedia full site outage.May 5 2022, 12:38 PM

Philosopher subscribed.Jun 12 2022, 8:55 AM

William_Avery subscribed.Jun 12 2022, 9:09 AM

0xDeadbeef subscribed.Jun 13 2022, 3:51 AM

Liz subscribed.Jun 13 2022, 4:18 AM

akosiaris mentioned this in T310450: fawiki user reports getting 503 errors with message "upstream connect error or disconnect before headers".Jun 27 2022, 4:36 PM

CX_Zoom subscribed.Sep 15 2022, 8:48 PM

In general, shouldn't phabricator tickets be one ticket = one cause? This one seems like it may be one ticket = many causes since it's such a generic error message. I've intermittently gotten this error message half a dozen times over the last year and I always assumed it was random-ish ops stuff that was the cause.

Perhaps this should be closed?

It was recently linked at enwiki village pump technical for an incident today, but I would suspect the original cause of this months-old ticket is unrelated.

Hi!

In T301505#8240830, @Novem_Linguae wrote:

In general, shouldn't phabricator tickets be one ticket = one cause? This one seems like it may be one ticket = many causes since it's such a generic error message. I've intermittently gotten this error message half a dozen times over the last year and I always assumed it was random-ish ops stuff that was the cause.

While there is some good sense in trying to have 1 task tracking 1 issue/symptom, that's not always possible. 1 ticket tracking 1 cause is rarely doable as issues/symptom might (and often do) have >1 causes. This holds true for this task as well. There are multiple causes for this generic error message to appear. You don't need to assume though it is random-ish ops stuff. When we are able, we publish Incident Reports under https://wikitech.wikimedia.org/wiki/Incident_status.

Perhaps this should be closed?

The only reason it stays open is so that people can find it easily and don't end up creating new ones (it does appear to be working) when this issue/symptom appears.

It was recently linked at enwiki village pump technical for an incident today, but I would suspect the original cause of this months-old ticket is unrelated.

Your suspicion would be probably correct, but we are leaving this open for when this issue appears, not the causes of it back then.

MBH unsubscribed.Oct 6 2022, 3:55 PM

I just received one trying to look up https://en.wiktionary.org/wiki/stuff. Not reproducible it seems.

In T301505#8507048, @AlexisJazz wrote:

I just received one trying to look up https://en.wiktionary.org/wiki/stuff. Not reproducible it seems.

See T326495: 2023-01-08 Wikimedia Connectivity Issues

@RLazarus: Removing task assignee as this open task has been assigned for more than two years - see the email sent to all task assignees on 2024-04-15.
Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome! :)
If this task has been resolved in the meantime, or should not be worked on by anybody ("declined"), please update its task status via "Add Action… 🡒 Change Status".
Also see https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator. Thanks!

@akosiaris I would be inclined to close this task or would you prefer to leave it open as a pointer for users searching for the error message?

In T301505#9807768, @LSobanski wrote:

@akosiaris I would be inclined to close this task or would you prefer to leave it open as a pointer for users searching for the error message?

It's 1.5 years later and we haven't had much of this happening lately. Apparently the actions we 've taken elsewhere in the infrastructure (requestctl et al) make the actual causes that were making this error appear to users not happen anymore. I 'll resolve and hopefully we won't have to reopen it.

Stang unsubscribed.Thu, Oct 24, 3:35 AM

upstream connect error or disconnect/reset before headers. reset reason: overflowClosed, ResolvedPublicActions