Another instance, tools-webgrid-lighttpd-1206 which have problem, my webservice running on it return 404 and can't restart, can't ssh in too.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T124133 NFS overload is causing instances to freeze | |||
| Resolved | valhallasw | T124875 tools-webgrid-lighttpd-1206 half-dead, webservices and ssh unaccessible |
Event Timeline
Comment Actions
tools-webgrid-lighttpd-1206 login: [418800.440736] INFO: task lighttpd:28011 blocked for more than 120 seconds. [418800.443108] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [418800.444451] INFO: task php-cgi:25487 blocked for more than 120 seconds. [418800.445209] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [418800.446137] INFO: task php-cgi:31222 blocked for more than 120 seconds. [418800.446882] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
valhallasw@tools-bastion-01:~$ qhost -h tools-webgrid-lighttpd-1206 -j HOSTNAME ARCH NCPU LOAD MEMTOT MEMUSE SWAPTO SWAPUS ------------------------------------------------------------------------------- global - - - - - - - tools-webgrid-lighttpd-1206.eqiad.wmflabs lx26-amd64 4 2.42 7.8G 2.1G 23.9G 0.0 job-ID prior name user state submit/start at queue master ja-task-ID ---------------------------------------------------------------------------------------------- 2555130 0.31264 lighttpd-j tools.jurgen r 01/21/2016 20:37:29 webgrid-li MASTER 2555492 0.31262 lighttpd-w tools.wikiwa r 01/21/2016 20:46:20 webgrid-li MASTER 2555718 0.31261 lighttpd-e tools.edinc r 01/21/2016 20:52:15 webgrid-li MASTER 2555803 0.31261 lighttpd-m tools.multic r 01/21/2016 20:54:50 webgrid-li MASTER 2555964 0.31260 lighttpd-n tools.nullze r 01/21/2016 20:58:09 webgrid-li MASTER 2556078 0.31259 lighttpd-u tools.usergr r 01/21/2016 21:08:28 webgrid-li MASTER 2556250 0.31258 lighttpd-g tools.guc r 01/21/2016 21:12:10 webgrid-li MASTER 2556629 0.31257 lighttpd-m tools.matvar r 01/21/2016 21:21:52 webgrid-li MASTER 2557401 0.31254 lighttpd-m tools.mg-bot r 01/21/2016 21:41:29 webgrid-li MASTER 2558642 0.31248 lighttpd-c tools.cats-p r 01/21/2016 22:14:15 webgrid-li MASTER 2559628 0.31244 lighttpd-c tools.cgstat r 01/21/2016 22:40:35 webgrid-li MASTER 2559905 0.31243 lighttpd-p tools.popula r 01/21/2016 22:48:49 webgrid-li MASTER 2560032 0.31243 lighttpd-g tools.grilli r 01/21/2016 22:51:58 webgrid-li MASTER 2560267 0.31241 lighttpd-g tools.german r 01/21/2016 22:59:36 webgrid-li MASTER 2560677 0.31240 lighttpd-s tools.sbot r 01/21/2016 23:08:49 webgrid-li MASTER 2560795 0.31240 lighttpd-a tools.analyt r 01/21/2016 23:11:22 webgrid-li MASTER 2561042 0.31238 lighttpd-l tools.liange r 01/21/2016 23:18:21 webgrid-li MASTER 2561313 0.31237 lighttpd-w tools.wmukev r 01/21/2016 23:26:49 webgrid-li MASTER 2561512 0.31236 lighttpd-w tools.wikitr r 01/21/2016 23:31:21 webgrid-li MASTER 2561934 0.31234 lighttpd-i tools.icalen r 01/21/2016 23:44:40 webgrid-li MASTER 2562050 0.31234 lighttpd-g tools.global r 01/21/2016 23:47:26 webgrid-li MASTER 2562223 0.31233 lighttpd-i tools.incola r 01/21/2016 23:52:24 webgrid-li MASTER 2624609 0.30799 lighttpd-o tools.osm4wi r 01/23/2016 22:02:04 webgrid-li MASTER 2643020 0.30596 lighttpd-i tools.imagem r 01/24/2016 19:41:04 webgrid-li MASTER 2714464 0.30048 lighttpd-r tools.random dr 01/27/2016 06:00:25 webgrid-li MASTER 2714638 0.30047 lighttpd-w tools.wiwosm r 01/27/2016 06:08:26 webgrid-li MASTER 2715513 0.30042 lighttpd-j tools.jembot dr 01/27/2016 06:41:33 webgrid-li MASTER 2715578 0.30041 lighttpd-p tools.phetoo dr 01/27/2016 06:43:42 webgrid-li MASTER 2720714 0.30010 twitter.sh tools.avicbo t 01/27/2016 11:06:21 webgrid-li MASTER 2720726 0.30010 random-fea tools.random t 01/27/2016 11:06:21 webgrid-li MASTER 2720830 0.30010 lighttpd-d tools.dplbot dt 01/27/2016 11:02:54 webgrid-li MASTER 2720968 0.30009 twitter.sh tools.avicbo t 01/27/2016 11:06:21 webgrid-li MASTER 2721064 0.30008 sdab_procs tools.dplbot t 01/27/2016 11:05:32 webgrid-li MASTER 2721079 0.30008 twitter.sh tools.avicbo t 01/27/2016 11:06:21 webgrid-li MASTER 2721223 0.30007 twitter.sh tools.avicbo t 01/27/2016 11:06:21 webgrid-li MASTER
Restarted the host from wikitech.