Page MenuHomePhabricator

503 Backend fetch failed
Open, NormalPublic

Description

Just got a 503 error or simplewiki:
Request from [snip] via cp1081 cp1081, Varnish XID 1033011297 Error: 503, Backend fetch failed at Thu, 19 Sep 2019 01:13:39 GMT

Event Timeline

Restricted Application added a project: User-DannyS712. · View Herald TranscriptThu, Sep 19, 1:16 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

@DannyS712: When doing what exactly? Viewing? Editing? Moving? Something else?

Viewing a diff, I believe

There is a thread on checkuser-l about this. At least three users intermittently got 503s today / this week when using Special:CheckUser. I don't know if it's related, but I'm assuming what @DannyS712 experienced is also intermittent, which suggests some broader issue. There were no recent changes to the CheckUser extension https://github.com/wikimedia/mediawiki-extensions-CheckUser/commits/master

Zzuuzz added a subscriber: Zzuuzz.Fri, Sep 20, 6:49 AM

I've been getting this intermittently through checkuser since around Thursday afternoon (UTC).

Request from [my ip] via cp1089 cp1089, Varnish XID 777562684
Error: 503, Backend fetch failed at Fri, 20 Sep 2019 06:30:36 GMT

In particular there is one /17 range which is consistently failing, using the shortest duration of one week. I know the results should be there, but they're not going to be particularly huge.

Sometimes the check is logged in the CU log, and sometimes not - mostly not -, but for this range always the error.

BBlack added a subscriber: BBlack.EditedSat, Sep 21, 1:18 AM

[removed - someone linked this during an ongoing incident and I assumed it was fresh. These reports are from days ago and my comment was not relevant]

I haven't seen this error in the last couple of days.
I suggest whatever was affecting checkuser has probably been resolved.

herron triaged this task as Normal priority.Mon, Sep 23, 7:43 PM

There have been more reports of this. Going off of the error logs for XTools, the last burst of 503s from the MediaWiki API happened from around 11:00 to 12:00 UTC on October 1.

Re: CheckUser, there was a recent security patch (T207094, backport to master) which did suffer from some initial performance issues. These issues were fixed around September 23rd and the patch redeployed. So the timing of these issues seems to match up with what's being discussed upon this task. There's also a current, public bug to further improve the security patch in its current state: T234239.