Shinken has been blowing up -releng with errors since about 7pm PDT last night.
Intermittent reports of errors all morning.
Shinken has been blowing up -releng with errors since about 7pm PDT last night.
Intermittent reports of errors all morning.
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | None | T97033 Beta cluster intermittent failures | |||
Resolved | hashar | T97130 JobQueueError Redis server error: Could not insert 1 cirrusSearchLinksUpdatePrioritized job(s). |
Could be the cause for T97047: EventLogging schemas are not served properly on beta cluster (which does not seem to be intermittent though).
This problem is still ongoing, although @coren and @Andrew may have found the root cause: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917
After updating the kernel on labvirt1001 and labvirt1002, the following guests may be problem free:
Overnight monitoring of those libvirt guests should tell us whether or not the root cause of the problem has been solved.
From T96905 it seems MySQL/MariaDB is not started on boot and deployment-db1 got rebooted on Thu Apr 23 23:53. I have restated MySQL :-)
This should be all fixed now; I'm not seeing the intermittent VM stalls anymore and all kernels have been upgraded to the fixed kernel.