Problems with multiple tools
Closed, ResolvedPublic

Description

Since yesterday (2015-06-10), there are problems with multiple tools of mine, cause uncertain.

Some tools to not respond, but keep loading idly even plain HTML. Examples:

A webservice2 restart does NOT fix the problem.

Also, I notice yesterday what seemed to be massive delays/timeouts in some of my APIs, though I cannot tell if these were from the webservice part or the database part.

Magnus created this task.Jun 11 2015, 7:12 AM
Magnus updated the task description. (Show Details)
Magnus raised the priority of this task from to Unbreak Now!.
Magnus added a project: Cloud-VPS.
Magnus added a subscriber: Magnus.
Restricted Application added a project: Cloud-Services. · View Herald TranscriptJun 11 2015, 7:12 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
valhallasw added a subscriber: valhallasw.
  • lighttpd-fist is running on tools-webgrid-lighttpd-1401
  • lighttpd-mix-n-match on tools-webgrid-lighttpd-1408
  • lighttpd-wikidata-todo on tools-webgrid-lighttpd-1407

neither of those seems to have an excessive load. SSH on the servers seems to be functional, file system is responding normally, so doesn't seem to be NFS related.

From the webproxy nginx log:

2015/06/11 07:56:03 [error] 9634#0: *8415080 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: xxx.yyy.223.2, server: , request: "GET /magnustools/resources/js/jquery/jquery.cookie.js HTTP/1.1", upstream: "http://10.68.17.195:57219/magnustools/resources/js/jquery/jquery.cookie.js", host: "tools.wmflabs.org", referrer: "https://tools.wmflabs.org/fist/wdfist/index.html"

Curl http://10.68.17.195:57219/magnustools/resources/js/jquery/jquery.cookie.js seems to hang (not time out!!). Looking at the network inspector in chrome also shows that /magnustools/ seems to be the cause.

Restarted magnustools webserver, which seems to have solved the issue.

valhallasw closed this task as Resolved.Jun 11 2015, 8:05 AM
valhallasw removed projects: Cloud-VPS, Cloud-Services.
valhallasw set Security to None.
Restricted Application added a project: Cloud-Services. · View Herald TranscriptJul 25 2015, 9:08 PM