Page MenuHomePhabricator

Error: 503, Backend fetch failed
Closed, ResolvedPublic

Description

Mediawiki.org, enwiki and zhwiki can not access, there may be more wikis that cannot be accessed.
Error message:

Request from 120.229.35.146 via cp4030 cp4030, Varnish XID 344497377
Error: 503, Backend fetch failed at Fri, 25 May 2018 09:25:00 GMT

Screenshot:

Event Timeline

Restricted Application added a project: Operations. · View Herald TranscriptMay 25 2018, 9:28 AM
Restricted Application added subscribers: Cosine02, Aklapper. · View Herald Transcript

Can you retry, it should be solved now or shortly (or may need a refresh of your browser)?

Can you retry, it should be solved now or shortly (or may need a refresh of your browser)?

Yes, I retry but this problem still exists.

It seems that I can visit wikitech but other WMF‘s websites cannot access.

I can visit now, but I don't know what happened.

Not your fault, issues with connectivity on certain regions on the world- a datacenter was disabled to workaround it. Thank you for the quick report, it helped!

ayounsi closed this task as Resolved.May 25 2018, 12:35 PM
ayounsi claimed this task.
ayounsi added a subscriber: ayounsi.

Our San Francisco datacenter is linked to our infrastructure by 2 links.
1 link had a planned maintenance, the other had an outage at the wrong time.
as soon as we noticed the issue, we disabled that datacenter, redirecting the users to a functional datacenter.
The links are back to a healthy state, and will re-enable that datacenter shortly.
We identified short and long term solutions to increase resiliency and reduce the risk of the issue happening again.

Thank you for reporting the issue.

as soon as we noticed the issue, we disabled that datacenter, redirecting the users to a functional datacenter.

In this case, is the disabling action automatic or manual?

Manual.
It would be automatic in an ideal world, but not enough resources to work on that.

Vvjjkkii renamed this task from Error: 503, Backend fetch failed to 0acaaaaaaa.Jul 1 2018, 1:08 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed ayounsi as the assignee of this task.
Vvjjkkii triaged this task as High priority.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
ema renamed this task from 0acaaaaaaa to Error: 503, Backend fetch failed.Jul 2 2018, 9:34 AM
ema closed this task as Resolved.
ema assigned this task to ayounsi.
ema updated the task description. (Show Details)
ema raised the priority of this task from High to Needs Triage.Jul 2 2018, 10:04 AM
ema added subscribers: Aklapper, gerritbot.