Page MenuHomePhabricator

tools-webgrid-lighttpd-1206 half-dead, webservices and ssh unaccessible
Closed, ResolvedPublic

Description

Another instance, tools-webgrid-lighttpd-1206 which have problem, my webservice running on it return 404 and can't restart, can't ssh in too.

Event Timeline

Phe created this task.Jan 27 2016, 11:02 AM
Phe raised the priority of this task from to Unbreak Now!.
Phe updated the task description. (Show Details)
Phe added a project: Toolforge.
Phe added a subscriber: Phe.
Restricted Application added a project: Cloud-Services. · View Herald TranscriptJan 27 2016, 11:02 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
tools-webgrid-lighttpd-1206 login: [418800.440736] INFO: task lighttpd:28011 blocked for more than 120 seconds.
[418800.443108] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[418800.444451] INFO: task php-cgi:25487 blocked for more than 120 seconds.
[418800.445209] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[418800.446137] INFO: task php-cgi:31222 blocked for more than 120 seconds.
[418800.446882] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
valhallasw@tools-bastion-01:~$ qhost -h tools-webgrid-lighttpd-1206 -j
HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS
-------------------------------------------------------------------------------
global                  -               -     -       -       -       -       -
tools-webgrid-lighttpd-1206.eqiad.wmflabs lx26-amd64      4  2.42    7.8G    2.1G   23.9G     0.0
   job-ID  prior   name       user         state submit/start at     queue      master ja-task-ID
   ----------------------------------------------------------------------------------------------
   2555130 0.31264 lighttpd-j tools.jurgen r     01/21/2016 20:37:29 webgrid-li MASTER
   2555492 0.31262 lighttpd-w tools.wikiwa r     01/21/2016 20:46:20 webgrid-li MASTER
   2555718 0.31261 lighttpd-e tools.edinc  r     01/21/2016 20:52:15 webgrid-li MASTER
   2555803 0.31261 lighttpd-m tools.multic r     01/21/2016 20:54:50 webgrid-li MASTER
   2555964 0.31260 lighttpd-n tools.nullze r     01/21/2016 20:58:09 webgrid-li MASTER
   2556078 0.31259 lighttpd-u tools.usergr r     01/21/2016 21:08:28 webgrid-li MASTER
   2556250 0.31258 lighttpd-g tools.guc    r     01/21/2016 21:12:10 webgrid-li MASTER
   2556629 0.31257 lighttpd-m tools.matvar r     01/21/2016 21:21:52 webgrid-li MASTER
   2557401 0.31254 lighttpd-m tools.mg-bot r     01/21/2016 21:41:29 webgrid-li MASTER
   2558642 0.31248 lighttpd-c tools.cats-p r     01/21/2016 22:14:15 webgrid-li MASTER
   2559628 0.31244 lighttpd-c tools.cgstat r     01/21/2016 22:40:35 webgrid-li MASTER
   2559905 0.31243 lighttpd-p tools.popula r     01/21/2016 22:48:49 webgrid-li MASTER
   2560032 0.31243 lighttpd-g tools.grilli r     01/21/2016 22:51:58 webgrid-li MASTER
   2560267 0.31241 lighttpd-g tools.german r     01/21/2016 22:59:36 webgrid-li MASTER
   2560677 0.31240 lighttpd-s tools.sbot   r     01/21/2016 23:08:49 webgrid-li MASTER
   2560795 0.31240 lighttpd-a tools.analyt r     01/21/2016 23:11:22 webgrid-li MASTER
   2561042 0.31238 lighttpd-l tools.liange r     01/21/2016 23:18:21 webgrid-li MASTER
   2561313 0.31237 lighttpd-w tools.wmukev r     01/21/2016 23:26:49 webgrid-li MASTER
   2561512 0.31236 lighttpd-w tools.wikitr r     01/21/2016 23:31:21 webgrid-li MASTER
   2561934 0.31234 lighttpd-i tools.icalen r     01/21/2016 23:44:40 webgrid-li MASTER
   2562050 0.31234 lighttpd-g tools.global r     01/21/2016 23:47:26 webgrid-li MASTER
   2562223 0.31233 lighttpd-i tools.incola r     01/21/2016 23:52:24 webgrid-li MASTER
   2624609 0.30799 lighttpd-o tools.osm4wi r     01/23/2016 22:02:04 webgrid-li MASTER
   2643020 0.30596 lighttpd-i tools.imagem r     01/24/2016 19:41:04 webgrid-li MASTER
   2714464 0.30048 lighttpd-r tools.random dr    01/27/2016 06:00:25 webgrid-li MASTER
   2714638 0.30047 lighttpd-w tools.wiwosm r     01/27/2016 06:08:26 webgrid-li MASTER
   2715513 0.30042 lighttpd-j tools.jembot dr    01/27/2016 06:41:33 webgrid-li MASTER
   2715578 0.30041 lighttpd-p tools.phetoo dr    01/27/2016 06:43:42 webgrid-li MASTER
   2720714 0.30010 twitter.sh tools.avicbo t     01/27/2016 11:06:21 webgrid-li MASTER
   2720726 0.30010 random-fea tools.random t     01/27/2016 11:06:21 webgrid-li MASTER
   2720830 0.30010 lighttpd-d tools.dplbot dt    01/27/2016 11:02:54 webgrid-li MASTER
   2720968 0.30009 twitter.sh tools.avicbo t     01/27/2016 11:06:21 webgrid-li MASTER
   2721064 0.30008 sdab_procs tools.dplbot t     01/27/2016 11:05:32 webgrid-li MASTER
   2721079 0.30008 twitter.sh tools.avicbo t     01/27/2016 11:06:21 webgrid-li MASTER
   2721223 0.30007 twitter.sh tools.avicbo t     01/27/2016 11:06:21 webgrid-li MASTER

Restarted the host from wikitech.

I was about to notify a guc outage, I suppose that's the reason of it.

valhallasw closed this task as Resolved.Jan 27 2016, 12:45 PM
valhallasw claimed this task.