It all started in mid-January when I started getting XTools has more and more increasingly been getting errors when writing to the user database. This includes 1205 Lock wait timeout exceeded; try restarting transaction and more rarely 2006 MySQL server has gone away. All the queries being ran are very fast (~0.00 sec).
Then I noticed the issue with Event Metrics, which relies heavily on the user database. We started getting connectivity errors, once on February 5th, 12th and 13th.
The issue is usually short-lived, but as of today, since I wrote this task, it has worsened. I've gotten more and more errors, and now a new one: 1040 Too many connections. Currently https://tools.wmflabs.org/copypatrol and https://eventmetrics.wmflabs.org/login is unusable.
XTools receives frequent requests, and Event Metrics has a cron that runs a query every 5 minutes. We get emailed when fatal errors happen, and I've never gotten this connectivity error prior to February 5. This makes me think it's something new. It would seem that it's getting worse by the day, and perhaps about to hit the breaking point?