Page MenuHomePhabricator
Create Task
Maniphest T241029

EditGroups tool: Unstable connection to SQL database on Toolforge
Open, Needs TriagePublic
Actions

Description

The EditGroups tool listens to the EventStream for Wikidata edits and stores their metadata in its own SQL database.

The listener process is a Python program which runs in Kubernetes. It connects to the SQL database (s53685__editgroups) via Django's ORM and performs the ingestion by batches of edits.

Over the past few weeks, the listener has started to die a lot more frequently, generally because its SQL connection vanishes:

django.db.utils.OperationalError: (2006, 'MySQL server has gone away')

(full stack trace available in the corresponding GitHub issue).

These repeated failures (about one every hour currently) mean that the listener is accumulating lag (currently 7 hours behind).

Has labsdb become more unstable recently? Is there any way to prevent these disconnections? Could perhaps the database be moved to a more reliable host?

Event Timeline

Pintoch created this task.Dec 18 2019, 4:39 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 18 2019, 4:39 AM
JJMC89 edited projects, added Data-Services; removed Wikidata, Toolforge.Dec 18 2019, 4:45 AM
Pintoch renamed this task from EditGroups tool: Unstable connection to SQL database on Toollabs to EditGroups tool: Unstable connection to SQL database on Toolforge.Dec 18 2019, 11:24 AM
bd808 subscribed.Dec 29 2019, 12:03 AM

'MySQL server has gone away' means that the connection has been closed from the server side. This is not uncommon and something that tools should be prepared to deal with. Per the connection handling policy, tools should minimize the amount of time they hold a database connection open, and the server may choose to close idle connections at any time.

Pintoch added a comment.Dec 29 2019, 9:01 AM

OK! This connection should never be idle given that bot edits on Wikidata never stop, so I am still not sure why this happens. It might be due to the specifics of how Django handles these long-running SQL connections.

bd808 added a project: Tools.Jan 4 2020, 8:30 PM
bd808 moved this task from Backlog to ToolsDB on the Data-Services board.Feb 10 2020, 6:06 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL