The biggest offender is one of the new jobrunners, so while I'd tend to think it has to do with connection overload, but I think this merits a better investigation for sure.

Should we consider this solved? It has not happened for a month: https://grafana.wikimedia.org/dashboard/db/mysql-aggregated?panelId=10&fullscreen&orgId=1&from=now-30d&to=now

Actually, this got solved for s2 this morning, but now it is happening on s1: https://grafana.wikimedia.org/dashboard/db/mysql-aggregated?panelId=10&fullscreen&orgId=1&from=1521744688622&to=1521820284514&var-dc=eqiad%20prometheus%2Fops&var-group=core&var-shard=All&var-role=All

I am not sure I am understanding your comment, on that graph (and the one I posted) it is mostly 0 errors, no?

We followed up on IRC, but we identified this is a terbium process trying to connect to db1067 and not having the right grants. We are not even sure if that should be running, so we may look at what it is and what it is trying to do and either fix it or kill it.

So by doing a traffic capture, I have seen proxysql trying to connect from terbium to db1067 every 10 seconds, which matches what we saw on the logs:

libmariadb._pid.1572._client_version.2.3.1	_platform.x86_64.program_name.proxysql_monitor
16:33:29.362863 IP 10.64.32.13.57126 > 10.64.48.22.3306: Flags [F.], seq 201, ack 176, win 58, options [nop,nop,], length 0