Page MenuHomePhabricator

https://idp.wikimedia.org/ down
Closed, ResolvedPublic

Description

I am trying to access some services like Icinga and Orchestrator and I keep getting the following error from https://idp.wikimedia.org/:

upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure

How to reproduce:
https://orchestrator.wikimedia.org/web/cluster/alias/s1
https://icinga.wikimedia.org/

Event Timeline

It seems to be normal now, are you still having problems?

joanna_borun lowered the priority of this task from High to Low.Jul 12 2025, 7:49 PM

IDP seems to be back to normal. Decreasing the priority and leaving the investigation for Monday.

wikimedia-sre
<marostegui> idp is down?
<marostegui> I get icinga or orchestrator with: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure
<marostegui> https://idp.wikimedia.org/ is down yep
<marostegui> https://phabricator.wikimedia.org/T399356 created this
<moritzm> marostegui: I've restarted Tomcat and it's working again, I'll have a closer log at the logs on Monday
<marostegui> moritzm: thank you <3
wikimedia-operations
<moritzm> !log restarted Tomcat on idp1004
<+stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
Marostegui assigned this task to MoritzMuehlenhoff.

I am not sure there was a follow up on this but after it was restarted, it was all working fine for a week so I am going to close this as fixed.