I noticed a periodic spike in 500s from varnish, turns out piwik quite regularly spews 500s back to clients:
The root cause seems to be exhausted mysql connections, from /var/log/apache2/error.log
[Wed Jan 04 01:43:15.171605 2017] [:error] [pid 22875] [client 10.64.32.97:5493] Error in Piwik (tracker): SQLSTATE[08004] [1040] Too many connections [Wed Jan 04 01:43:15.381398 2017] [:error] [pid 22875] [client 10.64.0.95:32831] Error in Piwik (tracker): SQLSTATE[08004] [1040] Too many connections [Wed Jan 04 01:43:15.421769 2017] [:error] [pid 22875] [client 10.64.0.98:42041] Error in Piwik (tracker): SQLSTATE[08004] [1040] Too many connections
Sometimes also errors similar to this
[Wed Jan 04 02:01:03.905886 2017] [:error] [pid 25107] [client 10.64.32.97:12162] Error in Piwik (tracker): Error query: SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction In query: UPDATE piwik_log_visit SET idvisitor = ?, visit_last_action_time = ?, visit_total_actions = visit_total_actions + 1 , visit_total_events = visit_total_events + 1 , visit_total_time = ?, custom_var_k1 = ?, custom_var_v1 = ?, custom_var_k2 = ?, custom_var_v2 = ?, custom_var_k3 = ?, custom_var_v3 = ? WHERE idsite = ? AND idvisit = ? Parameters: array ( 0 =>