There were a couple of update errors that may have been caused by this.

Amanda mentioned to Beat at Wikimania that the SDC team might be able to help out on a few of the ISA related issues where the current team needed support.

This aspect of the ISA tool (database connection issues) was identified as the area where the team has the least expertise and where the offered support from the WMF would be most valuable

There has been a lot of instability in the database connections/access to the database. This is particularly pronounced on the Dev instance <isa-dev.toolforge.org> but also occurs on the Live instance <isa.toolforge.org> . We know that insufficient error handling in ISA might account for some of the issues, but the same lack of error handling also makes it hard to trace. We can currently not say if it is due to our implementation, a library or Toolforge itself. There is a worry that migrating from Dev to Live will bring with it increased instability.

@NavinoEvans, @Eugene233 and @Sebastian_Berlin-WMSE can all support by providing log locations and further background as needed.

Heyo @Lokal_Profil ^_^ I talked with @Cparle about this. Cormac did you catch up with the ISA folks?

I noticed in P35377 there are also SSL errors connecting to commons.wikimedia.org:

urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='commons.wikimedia.org', port=443): Read timed out. (read timeout=5)

That points to a more generalized network connectivity issue than a database issue, but hard to say.

I don't see that specific error (2013, 'Lost connection to MySQL server during query') in the isa-dev logs (/data/project/isa-dev/uwsgi.log) but I do see a few occurrences in the isa logs (/data/proejct/isa/uwsgi.log).

The date/time of the errors in the logs don't seem to match any outage of ToolsDB, so it seems to confirm that the tool is sometimes seeing connectivity issues that are independent from the DB itself.

I would monitor if they continue at a similar rate or not, I only see a couple per day this week:

tools.isa@tools-sgebastion-10:~$ grep " - ERROR - " uwsgi.log
2023-10-29 03:19:40,585 - ERROR - Exception on /campaigns/140/stats [GET]
2023-10-29 15:17:34,322 - ERROR - Exception on /user-settings [GET]
2023-11-06 05:51:58,191 - ERROR - Exception on /campaigns/239 [GET]
  File "/data/project/isa/www/python/venv/lib/python3.11/site-packages/pymysql/connections.py", line 758, in _read_pack2023-11-06 05:51:57,977 - ERROR - Exception on /campaigns/239 [GET]
2023-11-06 05:56:21,771 - ERROR - Exception on /campaigns/239 [GET]
[SQL: SELECT contribution.id AS contribution_id, contribution.username AS contribution_username, contribution.user_id AS contribution_user_id, contribution.campaign_id AS contribution_campaign_id, contribution.file AS contribution_file, contribution.edit_type AS contribution_edit_type, contribution.edit_action AS contribution_edit_action, contribution.country AS contribution_country, contribution.depict2023-11-06 05:56:21,771 - ERROR - Exception on /campaigns/239 [GET]
  File "/data/project/isa/www/python/venv/lib/python3.11/site-packages/pymysql/connections.py", line 758, in _read_pack2023-11-06 05:56:21,778 - ERROR - Exception on /campaigns/239 [GET]
2023-11-07 22:27:50,708 - ERROR - Exception on /campaigns/239 [GET]
2023-11-07 22:31:43,667 - ERROR - Exception on /campaigns/94 [GET]
2023-11-08 10:11:14,192 - ERROR - Exception on /campaigns/228 [GET]
2023-11-08 22:36:14,274 - ERROR - Exception on /campaigns/228 [GET]
2023-11-09 10:47:28,172 - ERROR - Exception on /campaigns/228 [GET]

This actually seems like it might be something to do with SQLAlchemy and mysql recycling connections differently https://stackoverflow.com/questions/29755228/sqlalchemy-mysql-lost-connection-to-mysql-server-during-query

@Cparle oh yes, that's very possible. wait_timeout is currently set to 600 (10 minutes) on ToolsDB.

In T318050#9295909, @Abit wrote:

Heyo @Lokal_Profil ^_^ I talked with @Cparle about this. Cormac did you catch up with the ISA folks?

Many thanks! I've pinged the rest of the group to make sure they are aware of the developments here.