Page MenuHomePhabricator
Create Task
Maniphest T196153

Quarry cannot save queries with emojies
Open, Needs TriagePublic
Actions

Description

A query like SELECT '😂'; gets truncated into SELECT ' after it's received by the quarry-internal database.

Doing more investigation after https://gerrit.wikimedia.org/r/#/c/436576/, definitions like VARCHAR(255) BINARY and TEXT BINARY are actually non-binary, but likely with a binary collation. To store the emojies we could change the database charset to utf8mb4 or binary, but:

  • the former causes an error in creating the user table ERROR 1071 (42000) at line 3: Specified key was too long; max key length is 767 bytes,
  • the latter makes sqlalchemy return python binary str instead of unicode, causing UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 8: ordinal not in range(128)

We could investigate using https://stackoverflow.com/a/43403017, but that probably needs some upgrades to our database server.

CC T194691 T192698

Related Objects

Event Timeline

zhuyifei1999 created this task.Jun 1 2018, 11:06 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 1 2018, 11:06 AM
Aklapper added projects: Quarry, I18n.Jun 1 2018, 1:33 PM
zhuyifei1999 updated the task description. (Show Details)Jun 1 2018, 3:14 PM
zhuyifei1999 added a subscriber: Framawiki.
• Vvjjkkii renamed this task from Quarry cannot save queries with emojies to mubaaaaaaa.Jul 1 2018, 1:07 AM
• Vvjjkkii triaged this task as High priority.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from mubaaaaaaa to Quarry cannot save queries with emojies.Jul 2 2018, 1:52 AM
CommunityTechBot raised the priority of this task from High to Needs Triage.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
CommunityTechBot added a subscriber: Aklapper.
zhuyifei1999 moved this task from Backlog to Bugs on the Quarry board.Oct 8 2018, 7:04 PM
Framawiki added a comment.Dec 2 2018, 9:40 PM

Now (since Make surrogateescape the global default error handler for Unicode issues deployment?) 🇫🇷 appears as ????????.

Krinkle unsubscribed.Dec 3 2018, 2:43 AM
jcrespo added a comment.Jan 7 2019, 10:35 AM

Regarding the second error- binary strings are not text, so they must be converted to python strings explicitly after driver execution.

For the first, I can help- most likely there is no need to upgrade, just to configure the database in the right way. Latest versions already have innodb_large_prefix enabled by default.

• Bstorm moved this task from Bugs to Planning on the Quarry board.Mar 26 2021, 8:20 PM
• nskaggs moved this task from Planning to Backlog on the Quarry board.Aug 2 2021, 8:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits