Page MenuHomePhabricator

[Translate] Error: 1283 Column 'tmf_text' cannot be part of FULLTEXT index
Closed, ResolvedPublic

Description

When attempting to install Translate with Mariadb using a binary character set I am hit with:

Creating translate_groupreviews table ...done.
Creating translate_tms table ...[692c98dbf5cb102e1aebee3a] [no req]   Wikimedia\Rdbms\DBQueryError from line 1443 of /workspace/src/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: CREATE FULLTEXT INDEX tmf_text ON `translate_tmf` (tmf_text)

Function: Wikimedia\Rdbms\Database::sourceFile( /workspace/src/extensions/Translate/sql/translate_tm.sql )
Error: 1283 Column 'tmf_text' cannot be part of FULLTEXT index (localhost:/tmp/quibble-mysql-12skd_5r/socket)

For the Quibble jobs we use Debian Stretch and MariaDB configured with:

[client]
# Stretch defaults to utf8mb4. T193222
default-character-set = binary


[mysqld]

# Stretch defaults to utf8mb4. T193222
character_set_server     = binary
character_set_filesystem = binary
collation_server         = binary

# Stricter mode T119371
# Note: should also be set in MediaWiki via $wgSQLMode
sql_mode = 'TRADITIONAL'

On Stretch it defaults to utf8mb4 which causes various issues such as keys being too long. The consensus is to switch to binary T193222#4247329 and there is an RFC about it at T194125

Seems the root cause is a Translate change https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Translate/+/439899 which add DEFAULT CHARSET=utf8mb4 to the translate_tmf table.

Event Timeline

hashar created this task.Jun 19 2018, 7:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 19 2018, 7:15 PM

The change was added in attempt to fix the problem you mentioned so it definitely cannot be the cause. It seems it is a not a full fix (maybe not even a partial), but root cause is still unknown.

Change 441127 had a related patch set uploaded (by Hashar; owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/Translate@wmf/1.32.0-wmf.8] Add "DEFAULT CHARSET=utf8" to translate_tmf table

https://gerrit.wikimedia.org/r/441127

I noticed the issue when triggering an experimental job that tests several jobs together. I have triggered it by commenting check experimental on Gerrit and it failed against wmf/1.32.0-wmf.8, but passed on master or `wmf/1.32.0-wmf.999. Those two last branches have the fix.

My revert of the Translate fix (Gerrit 439899) fails as intended.

Checking Translate master branch, the experimental jobs pass. My conclusion is I got confused by only looking at the wmf8 failure and I have missed that wmf999 and master were fine :]

I have cherry picked the fix for wmf/1.32.0-wmf.8: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Translate/+/441127 And the experimental job should pass now.

@thiemowmde thank you to have spotted the issue and proposed a fix. That unblock my migration of CI jobs toward Docker \o/

Something left to investigate: sql/translate_tm.sql now hardcode the charset to utf8mb4. When we later enable the extension on a wiki, we would end up sourcing that page and using that charset.

I am not sure what production is using but it is probably binary. So we probably need to loop in the DBAs to confirm/sort it out.

WMF production doesn't use these tables. ElasticSearch is used instead.

hashar closed this task as Resolved.Jun 20 2018, 7:08 AM
hashar assigned this task to thiemowmde.

@Nikerabbit wrote:

WMF production doesn't use these tables. ElasticSearch is used instead.

Awesome thank you :]

Marking this task resolved via https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Translate/+/439899 \o/

Change 441127 merged by jenkins-bot:
[mediawiki/extensions/Translate@wmf/1.32.0-wmf.8] Add "DEFAULT CHARSET=utf8" to translate_tmf table

https://gerrit.wikimedia.org/r/441127

Vvjjkkii renamed this task from [Translate] Error: 1283 Column 'tmf_text' cannot be part of FULLTEXT index to 2maaaaaaaa.Jul 1 2018, 1:02 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed thiemowmde as the assignee of this task.
Vvjjkkii triaged this task as High priority.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: gerritbot, Aklapper.
CommunityTechBot renamed this task from 2maaaaaaaa to [Translate] Error: 1283 Column 'tmf_text' cannot be part of FULLTEXT index.Jul 2 2018, 1:20 PM
CommunityTechBot closed this task as Resolved.
CommunityTechBot assigned this task to thiemowmde.
CommunityTechBot raised the priority of this task from High to Needs Triage.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added subscribers: gerritbot, Aklapper.

Change 443620 had a related patch set uploaded (by Hashar; owner: Thiemo Kreuz (WMDE)):
[mediawiki/extensions/Translate@REL1_31] Add "DEFAULT CHARSET=utf8" to translate_tmf table

https://gerrit.wikimedia.org/r/443620

hashar reopened this task as Open.Jul 3 2018, 1:30 PM

It is still broken on REL1_31, and I guess on REL1_30 as well.

Change 443620 merged by Umherirrender:
[mediawiki/extensions/Translate@REL1_31] Add "DEFAULT CHARSET=utf8" to translate_tmf table

https://gerrit.wikimedia.org/r/443620

hashar closed this task as Resolved.Aug 20 2018, 9:50 AM