Page MenuHomePhabricator

Problems with multiple tools
Closed, ResolvedPublic

Description

Since yesterday (2015-06-10), there are problems with multiple tools of mine, cause uncertain.

Some tools to not respond, but keep loading idly even plain HTML. Examples:

A webservice2 restart does NOT fix the problem.

Also, I notice yesterday what seemed to be massive delays/timeouts in some of my APIs, though I cannot tell if these were from the webservice part or the database part.

Event Timeline

Magnus raised the priority of this task from to Unbreak Now!.
Magnus updated the task description. (Show Details)
Magnus added a project: Cloud-VPS.
Magnus subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
  • lighttpd-fist is running on tools-webgrid-lighttpd-1401
  • lighttpd-mix-n-match on tools-webgrid-lighttpd-1408
  • lighttpd-wikidata-todo on tools-webgrid-lighttpd-1407

neither of those seems to have an excessive load. SSH on the servers seems to be functional, file system is responding normally, so doesn't seem to be NFS related.

From the webproxy nginx log:

2015/06/11 07:56:03 [error] 9634#0: *8415080 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: xxx.yyy.223.2, server: , request: "GET /magnustools/resources/js/jquery/jquery.cookie.js HTTP/1.1", upstream: "http://10.68.17.195:57219/magnustools/resources/js/jquery/jquery.cookie.js", host: "tools.wmflabs.org", referrer: "https://tools.wmflabs.org/fist/wdfist/index.html"

Curl http://10.68.17.195:57219/magnustools/resources/js/jquery/jquery.cookie.js seems to hang (not time out!!). Looking at the network inspector in chrome also shows that /magnustools/ seems to be the cause.

Restarted magnustools webserver, which seems to have solved the issue.

valhallasw removed projects: Cloud-VPS, Cloud-Services.
valhallasw set Security to None.