deployment-aqs01.deployment-prep.eqiad.wmflabs is stall. salt respond but ssh commands do not.
Looking at the instance console:
[3828240.260148] INFO: task jbd2/vda3-8:111 blocked for more than 120 seconds. [3828240.263584] Not tainted 3.16.0-4-amd64 #1 [3828240.264122] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Some tasks are hanging on /dev/vda3 ?
Mentioned in SAL [2016-05-11T11:42:49Z] <hashar> rebooting deployment-aqs01 via wikitech T134981
It is back. Puppet is lagged out:
The last Puppet run was at Sat May 7 04:51:29 UTC 2016 (6174 minutes ago).
Puppet log that auto started on instance boot:
Notice: /Stage[main]/Scap/Package[scap]/ensure: ensure changed '3.1.0-1' to '3.2.0-1'
Notice: /Stage[main]/Base::Initramfs/Initramfs::Script[mdadm-sleep]/File[/etc/initramfs-tools/scripts/init-premount/mdadm-sleep]/ensure: defined content as '{md5}e3aea6f9a4dfd00bfc8723132a410d1e'
Info: /Stage[main]/Base::Initramfs/Initramfs::Script[mdadm-sleep]/File[/etc/initramfs-tools/scripts/init-premount/mdadm-sleep]: Scheduling refresh of Exec[update-initramfs]
Notice: /Stage[main]/Initramfs/Exec[update-initramfs]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Cassandra::Metrics/Base::Service_unit[cassandra-metrics-collector]/Service[cassandra-metrics-collector]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Cassandra::Metrics/Base::Service_unit[cassandra-metrics-collector]/Service[cassandra-metrics-collector]: Unscheduling refresh on Service[cassandra-metrics-collector]
Notice: /Stage[main]/Cassandra/Cassandra::Instance[default]/Base::Service_unit[cassandra]/Service[cassandra]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Cassandra/Cassandra::Instance[default]/Base::Service_unit[cassandra]/Service[cassandra]: Unscheduling refresh on Service[cassandra]