Change Details

Recently, I've intermittently had connectivity issues when querying user databases: `SQLSTATE[HY000] [2002] Connection refused"` This happened at: * 2019-02-05 03:25 UTC * 2019-02-12 19:42 UTC * 2019-02-13 03:40 UTC The issue was very short-lived. Only a handful of requests were affected. This has happened to XTools (`s51187`), which receives frequent requests, and Event Metrics (`s53003` and `s53713`), which has a cron that runs a query every 5 minutes. We get emailed when fatal errors happen, and I've never gotten this connectivity error prior to February 5. This makes me think it's something new.

It all started in mid-January when I started getting #XTools has more and more increasingly been getting errors when writing to the user database. This includes `1205 Lock wait timeout exceeded; try restarting transaction` and more rarely `2006 MySQL server has gone away`. All the queries being ran are very fast (~0.00 sec). Then I noticed the issue with #event_metrics, which relies heavily on the user database. We started getting connectivity errors, once on February 5th, 12th and 13th. The issue is usually short-lived, but as of today, since I wrote this task, it has worsened. I've gotten more and more errors, and now a new one: `1040 Too many connections`. Currently https://tools.wmflabs.org/copypatrol and https://eventmetrics.wmflabs.org/login is unusable. XTools receives frequent requests, and Event Metrics has a cron that runs a query every 5 minutes. We get emailed when fatal errors happen, and I've never gotten this connectivity error prior to February 5. This makes me think it's something new. It would seem that it's getting worse by the day, and perhaps about to hit the breaking point?

Recently,It all started in mid-January when I started getting #XTools has more and more increasingly been getting errors when writing to the user database. This includes `1205 Lock wait timeout exceeded; try restarting transaction` and more rarely `2006 MySQL server has gone away`. I've intermittently had connectivity issues when querying user databases:All the queries being ran are very fast (~0.00 sec). `SQLSTATE[HY000] [2002] Connection refused"`Then I noticed the issue with #event_metrics, which relies heavily on the user database. We started getting connectivity errors, once on February 5th, 12th and 13th. This happened at: * 2019-02-05 03:25 UTC * 2019-02-12 19:42 UTC * 2019-02-13 03:40 UTCe issue is usually short-lived, but as of today, since I wrote this task, it has worsened. I've gotten more and more errors, and now a new one: `1040 Too many connections`. Currently https://tools.wmflabs.org/copypatrol and https://eventmetrics.wmflabs.org/login is unusable. The issue was very short-lived. Only a handful of XTools receives frequests were affected. This has happened to XTools (`s51187`)nt requests, which receives frequent requests,and Event Metrics has a cron that runs a query every 5 minutes. and Event Metrics (`s53003` and `s53713`)We get emailed when fatal errors happen, which has a croand I've never gotten that runs a query every 5 minutesis connectivity error prior to February 5. We get emailed when fatal errors happen,This makes me think it's something new. and I've never gotten this connectivity error prior to February 5.It would seem that it's getting worse by the day, This makes me think it's somethand perhaps about to hit the breaking new.point?