Page MenuHomePhabricator

Particular MariaDB queries not working on SWAP
Closed, ResolvedPublic

Description

I'm having trouble running MariaDB queries on SWAP which were functional as of Dec 15th. Here is a minimal test case.

I need help to move forward in identifying and resolving why this test query is working for some SWAP users but is throwing a LookupError: unknown encoding: binary error for me.

test = wmf.mariadb.run("""
SELECT rev_page, SUM(rev_minor_edit)
FROM revision
WHERE rev_page =104584
""", "pawiki")

My pandas and mysql-connector versions:
Name: mysql-connector-python
Version: 8.0.18

Name: pandas
Version: 0.25.3

I tried restarting the server, running the queries on different notebooks, restarting kernels, asking product analysts to run the same queries (it worked for them), and running queries without SUMs or divide statements (worked).

Event Timeline

Iflorez moved this task from Backlog to In Progress on the GLOW board.
Iflorez added a subscriber: Analytics.
Iflorez removed a subscriber: Analytics.

A weird, error, something is wrong when using charset=binary to convert from whatever result mysql gives back into python:

~/venv/lib/python3.5/site-packages/mysql/connector/conversion.py in _DECIMAL_to_python(self, value, desc)
    446         Returns value as a decimal.Decimal.
    447         """
--> 448         val = value.decode(self.charset)
    449         return Decimal(val)
    450 

LookupError: unknown encoding: binary

@Neil_P._Quinn_WMF wmfdata sets this when doing mysql.connect.
https://github.com/neilpquinn/wmfdata/blob/master/wmfdata/mariadb.py#L63-L70

I see the same error that Irene does. Is binary what should be set here?

A weird, error, something is wrong when using charset=binary to convert from whatever result mysql gives back into python:

~/venv/lib/python3.5/site-packages/mysql/connector/conversion.py in _DECIMAL_to_python(self, value, desc)
    446         Returns value as a decimal.Decimal.
    447         """
--> 448         val = value.decode(self.charset)
    449         return Decimal(val)
    450 

LookupError: unknown encoding: binary

@Neil_P._Quinn_WMF wmfdata sets this when doing mysql.connect.
https://github.com/neilpquinn/wmfdata/blob/master/wmfdata/mariadb.py#L63-L70

I see the same error that Irene does. Is binary what should be set here?

Well, it's been that way for a long time, and has been working fine up until now. But I just figured out what caused it to break: it was working for me until I upgraded mysql-connector-python from 8.0.17 to 8.0.18 just now. Now, it throws the same error for me. Very odd since that should be just a patch release. I'll have to look into this more.

nshahquinn-wmf added subscribers: nettrom_WMF, mpopov.

Okay, I think I figured it out! Details at https://github.com/neilpquinn/wmfdata/pull/6. I'll leave this open until it gets merged by @mpopov or @nettrom_WMF.

nshahquinn-wmf moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

@mpopov has merged the fix! @Iflorez, if you encounter this problem again, please reopen the task. However, the version of wmfdata with the fix is currently not working on due to a separate issue (T242683) so you won't be to try out this fix until that is resolved.

In T242448#5801373, @Neil_P._Quinn_WMF wrote:

However, the version of wmfdata with the fix is currently not working on due to a separate issue (T242683) so you won't be to try out this fix until that is resolved.

@Iflorez, that issue has been resolved too, so the latest version of wmfdata should work properly!