Page MenuHomePhabricator

tools.wmflabs.org/sigma exceeded 'max_user_connections'
Open, Needs TriagePublic

Description

When attempting to view the request url above, the trace below is experienced.

The key parts seem to be:

pymysql.err.InternalError: (1226, "User 's51469' has exceeded the 'max_user_connections' resource (current value: 10)")

The above exception was the direct cause of the following exception:

pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'enwiki.labsdb'")

Not sure why this is happening but this may have high traffic at the moment due to a mention on https://en.wikipedia.org/wiki/Wikipedia_talk:Arbitration_Committee/Noticeboard#Edgar181_desysopped

Details

Request URL
https://tools.wmflabs.org/sigma/editorinteract.py?users=Edgar181&users=Deli+nk&users=ChemNerd&users=PCock&users=Gnome+de+plume&users=Edgeweyes&users=TimBuck2&users=Slideshow+Bob&users=&users=&startdate=&enddate=&ns=Wikipedia&server=enwiki
Stack Trace

Traceback (most recent call last):

File "/data/project/sigma/www/python/venv/lib/python3.5/site-packages/aiomysql/connection.py", line 503, in _connect
  await self._request_authentication()
File "/data/project/sigma/www/python/venv/lib/python3.5/site-packages/aiomysql/connection.py", line 785, in _request_authentication
  auth_packet = await self._read_packet()
File "/data/project/sigma/www/python/venv/lib/python3.5/site-packages/aiomysql/connection.py", line 593, in _read_packet
  packet.check_error()
File "/data/project/sigma/www/python/venv/lib/python3.5/site-packages/pymysql/protocol.py", line 220, in check_error
  err.raise_mysql_exception(self._data)
File "/data/project/sigma/www/python/venv/lib/python3.5/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
  raise errorclass(errno, errval)

pymysql.err.InternalError: (1226, "User 's51469' has exceeded the 'max_user_connections' resource (current value: 10)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File "./app.py", line 33, in inner
  return func(*a, **kw)
File "./app.py", line 71, in editorinteract_py
  tpl, store = EditorInteract(params)()
File "./editorinteract.py", line 28, in __call__
  loop.run_until_complete(self.do_query(store.server, store.query))
File "/usr/lib/python3.5/asyncio/base_events.py", line 466, in run_until_complete
  return future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
  raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
  result = coro.throw(exc)
File "./editorinteract.py", line 158, in do_query
  user_edits = dict(zip(query.users, await asyncio.gather(*[get_user_edits(u) for u in query.users])))
File "/usr/lib/python3.5/asyncio/futures.py", line 380, in __iter__
  yield self  # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
  future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
  raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
  result = coro.send(None)
File "./editorinteract.py", line 154, in get_user_edits
  async with sql.connect(**connect_info(server)) as conn:
File "/data/project/sigma/www/python/venv/lib/python3.5/site-packages/aiomysql/utils.py", line 48, in __aenter__
  self._obj = await self._coro
File "/data/project/sigma/www/python/venv/lib/python3.5/site-packages/aiomysql/connection.py", line 75, in _connect
  await conn._connect()
File "/data/project/sigma/www/python/venv/lib/python3.5/site-packages/aiomysql/connection.py", line 523, in _connect
  self._host) from e

pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'enwiki.labsdb'")

Event Timeline

RhinosF1 created this task.Dec 6 2019, 6:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 6 2019, 6:29 PM
RhinosF1 edited projects, added Tools; removed Cloud-Services.Dec 6 2019, 6:31 PM

Could T235954#5589121 be related?

RhinosF1 edited projects, added Tools; removed Tool-Database-Queries.

Actually fits better

zhuyifei1999 added a subscriber: zhuyifei1999.

Do you have a minimum reproducing test case?

Bstorm added a subscriber: Bstorm.EditedDec 6 2019, 6:42 PM

User 's51469' has exceeded the 'max_user_connections' resource (current value: 10) All users are limited to 10 connections on the wiki replicas. We are working on setting up a review process for case-by-case exceptions (T236974: Establish a process for increasing a toolforge tool's connections to the wiki replicas), but those are expected to be rare.

Is the tool opening new connections for every hit?

Bstorm added a comment.Dec 6 2019, 6:49 PM

Ahh, looking at the test case, it's using aiomysql. That very much is likely to be opening new connections on any request. That's a really bad idea unless it can be constrained to 10 connections and forced to reuse them.

zhuyifei1999 renamed this task from Unable to connect to MySQL (enwiki.labsdb) to tools.wmflabs.org/sigma exceeded 'max_user_connections'.Dec 6 2019, 6:57 PM
zhuyifei1999 edited projects, added Tools; removed Tool-Database-Queries.

Given https://tools.wmflabs.org/sigma/editorinteract.py?users=Example&users=RhinosF1&users=&startdate=&enddate=&ns=&server=enwiki doesn’t error, my guess is a new connection is opened concurrently for each user it checks.

The failed request is for 13/4 users

Bstorm added a comment.Dec 6 2019, 7:03 PM

Dunno if T235954#5589121 has to do with it. If there is a toolforge tooling issue around it, we would like to fix that. It just appears from the error that async mysql might be opening a lot of connections.

Bstorm added a comment.Dec 6 2019, 7:05 PM

Given https://tools.wmflabs.org/sigma/editorinteract.py?users=Example&users=RhinosF1&users=&startdate=&enddate=&ns=&server=enwiki doesn’t error, my guess is a new connection is opened concurrently for each user it checks.
The failed request is for 13/4 users

That makes sense. @Sigma ^^ That seems to be the fix there if the users it is checking, not the accesses are also made to reuse connections one way or another.

Dunno if T235954#5589121 has to do with it. If there is a toolforge tooling issue around it, we would like to fix that. It just appears from the error that async mysql might be opening a lot of connections.

It was a first thought but I’m edging on no as some stuff work now