Page MenuHomePhabricator

Beta cluster api.php never responds
Closed, ResolvedPublic

Description

https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page returns promptly but https://en.wikipedia.beta.wmflabs.org/w/api.php times out. This is causing all the Readers Web browser tests to fail. (Sorry, if this is a duplicate and I just couldn't find it!)

In a browser, a maintenance response is eventually served:

Error
Our servers are currently under maintenance or experiencing a technical problem. Please try again in a few minutes.

See the error message at the bottom of this page for more information.

If you report this error to the Wikimedia System Administrators, please include the details below.

Request from 73.252.38.252 via deployment-cache-text04 deployment-cache-text04, Varnish XID 58778062
Error: 503, Backend fetch failed at Wed, 21 Feb 2018 15:08:23 GMT

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 21 2018, 3:09 PM
Niedzielski updated the task description. (Show Details)

Looks like the api is able to render a trivial test or two for me currently: https://en.wikipedia.beta.wmflabs.org/w/api.php?action=query&titles=Anewreferencepage&prop=revisions&rvlimit=max&rvprop=content&list=backlinks&bltitle=Anewreferencepage&bllimit=max

I noticed hhvm on deployment-mediawiki05 flailing trying to read from some pipes and I kicked it.

Reference strace snippet:

[pid 18164] lseek(185, 0, SEEK_CUR <unfinished ...>                                                                                                                       
[pid 18164] <... lseek resumed> )       = -1 ESPIPE (Illegal seek)                                                                                                        
[pid 18162] lseek(228, 18446744073709551615, SEEK_SET <unfinished ...>                                                                                                    
[pid 18164] lseek(185, 18446744073709551615, SEEK_SET <unfinished ...>                                                                                                    
[pid 18162] <... lseek resumed> )       = -1 ESPIPE (Illegal seek)                                                                                                        
[pid 18164] <... lseek resumed> )       = -1 ESPIPE (Illegal seek)

reference lsof snippet:

hhvm    15008 www-data  185w  FIFO               0,10       0t0 35505083 pipe
...
hhvm    15008 www-data  228w  FIFO               0,10       0t0 35511027 pipe

@Niedzielski could you verify that this resolves your issue?

Niedzielski closed this task as Resolved.Feb 21 2018, 5:17 PM
Niedzielski assigned this task to thcipriani.

Fixed! Thank you @thcipriani!