| • bzimport | |
| Mar 31 2012, 3:15 PM |
| F9048: 0001-Issue-1082-Gerrit-displays-instead-of-UTF-8.patch | |
| Nov 22 2014, 12:15 AM |
Author: beau
Description:
There is an account 'User:Szymon Świerkosz' on labsconsole wiki, however gerrit shows it as 'Szymon ?wierkosz'. I have provided the URL for an example page.
Version: unspecified
Severity: critical
URL: https://gerrit.wikimedia.org/r/4040
This is very likely an upstream problem, but it seems to be specific to user names. For example, in https://gerrit.wikimedia.org/r/4040 , Szymon's name is shown correctly in the "committer" field, but incorrectly in the "reviewer" and "owner" fields.
(In reply to comment #3)
This is very likely an upstream problem, but it seems to be specific to user
names.
What about this issue suggests it's an upstream problem?
Well pretty much everything with gerrit is an upstream problem ;-)
Like the other unicode bugs, we can probably work around this though.
Here's an interesting one:
http://code.google.com/p/gerrit/issues/detail?id=1082
They say UTF-8 won't work with MySQL :/
MediaWiki works absolutely fine with MySQL and Unicode.
The correct phasing would be Gerrit does not support Unicode when using MySQL as backend.
(In reply to comment #7)
MediaWiki works absolutely fine with MySQL and Unicode.
The correct phasing would be Gerrit does not support Unicode when using MySQL
as backend.
Was about to say, this definitely sounds like a Gerrit problem.
(In reply to comment #8)
(In reply to comment #7)
MediaWiki works absolutely fine with MySQL and Unicode.
The correct phasing would be Gerrit does not support Unicode when using MySQL
as backend.Was about to say, this definitely sounds like a Gerrit problem.
As I said upstream, Gerrit claiming this doesn't work is just silly. I've already theorized that we can just change the collations and this will work, but I haven't tested yet.
If someone wants to test this theory, we can set you up with access to the gerrit project on labs (which is already running 2.3).
Nope - tested with 2.3-rc0-158-g34ab429 - I have utf8_unicode_ci on all MySQL tables and I get question marks.
A bit newer Gerrit deployed on PostgreSQL is fine.
(In reply to comment #10)
Nope - tested with 2.3-rc0-158-g34ab429 - I have utf8_unicode_ci on all MySQL
tables and I get question marks.
We've got 2.3 final on gerrit-dev on labs so we can test there. Want me to add you? I'm wondering if making the fields binary like we do in MediaWiki would work...but that's a bigger change than just the collations on the tables.
A bit newer Gerrit deployed on PostgreSQL is fine.
I really don't see us moving to PG or H2, so we need to find a fix. I *refuse* to believe Gerrit that this is unfixable on MySQL.
Created attachment 10411
Tell gerrit to use UTF-8 with MySQL
My MySQL database is in UTF-8 and it sees that gerrit stores the values properly.
A patch attached forces gerrit to use UTF-8 when connecting to MySQL.
Attached:
^demon, can you try this change in the configuration (assuming we can have tables in UTF-8):
[database]
type = JDBC driver = com.mysql.jdbc.Driver url = jdbc:mysql://localhost/reviewdb?characterSetResults=utf8&characterEncoding=utf8&connectionCollation=utf8_unicode_ci username = gerrit2
"database" and "hostname" entries should be removed. "username" should stay.
The following tables are definitely affected and need some sort of fix:
account_external_ids accounts changes patch_comments
These tables aren't currently affected, but could be if we put non-ASCII data into them.
account_group_names account_groups approval_categories approval_category_values change_messages tracking_ids
Ok, collation has been updated on all tables, and https://gerrit.wikimedia.org/r/#change,6439 has been submitted to change the connection url.
(In reply to comment #13)
^demon, can you try this change in the configuration (assuming we can have
tables in UTF-8):[database]
type = JDBC driver = com.mysql.jdbc.Driver url =jdbc:mysql://localhost/reviewdb?characterSetResults=utf8&characterEncoding=utf8&connectionCollation=utf8_unicode_ci
username = Gerrit change #2"database" and "hostname" entries should be removed. "username" should stay.
Ok, I changed the collation/charset on all the tables, and we updated the connection string. The database is now showing the correct data (yay!), but we're still not getting the right data to the UI.
See the owner on https://gerrit.wikimedia.org/r/#change,6388 which is an improvement although still not correct.
Looks "better" now.
I would then try connecting to MySQL via JDBC directly and see if it's okay.
You can try https://gerrit-review.googlesource.com/#/c/34670/ to play live with data obtained from the SQL database via Gerrit's ORM or play directly.
You can try this code:
http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical/60187/focus=60206
to check what really JDBC sees.
I hope you didn't end up with a double-encoded UTF-8 in the database (quite easy to do with MySQL, harder to recover) - so that Ś is not 0xC5 0x9A but 0xC3 0x85 0xC2 0x9A instead.
Some data from my MySQL instance:
$ mysql -u root -p reviewdb --default-character-set=utf8
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1212
Server version: 5.0.92 FreeBSD port: mysql-server-5.0.92
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql Ver 14.12 Distrib 5.0.92, for portbld-freebsd8.2 (amd64) using 5.2
Connection id: 1212
Current database: reviewdb
Current user: root@localhost
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server version: 5.0.92 FreeBSD port: mysql-server-5.0.92
Protocol version: 10
Connection: Localhost via UNIX socket
Server characterset: latin1
Db characterset: utf8
Client characterset: utf8
Conn. characterset: utf8
UNIX socket: /tmp/mysql.sock
Uptime: 21 days 8 hours 24 min 17 sec
mysql> show full columns from accounts;
+----------------------------------------+--------------+-----------------+------+-----+-------------------+-------+---------------------------------+---------+
| Field | Type | Collation | Null | Key | Default | Extra | Privileges | Comment |
+----------------------------------------+--------------+-----------------+------+-----+-------------------+-------+---------------------------------+---------+
| registered_on | timestamp | NULL | NO | CURRENT_TIMESTAMP | select,insert,update,references | ||||
| full_name | varchar(255) | utf8_bin | YES | MUL | NULL | select,insert,update,references | |||
| preferred_email | varchar(255) | utf8_bin | YES | MUL | NULL | select,insert,update,references | |||
| contact_filed_on | timestamp | NULL | YES | NULL | select,insert,update,references | ||||
| maximum_page_size | smallint(6) | NULL | NO | 0 | select,insert,update,references | ||||
| show_site_header | char(1) | utf8_unicode_ci | NO | N | select,insert,update,references | ||||
| use_flash_clipboard | char(1) | utf8_unicode_ci | NO | N | select,insert,update,references | ||||
| download_url | varchar(20) | utf8_bin | YES | NULL | select,insert,update,references | ||||
| download_command | varchar(20) | utf8_bin | YES | NULL | select,insert,update,references | ||||
| copy_self_on_email | char(1) | utf8_unicode_ci | NO | N | select,insert,update,references | ||||
| date_format | varchar(10) | utf8_bin | YES | NULL | select,insert,update,references | ||||
| time_format | varchar(10) | utf8_bin | YES | NULL | select,insert,update,references | ||||
| display_patch_sets_in_reverse_order | char(1) | utf8_unicode_ci | NO | N | select,insert,update,references | ||||
| display_person_name_in_review_category | char(1) | utf8_unicode_ci | NO | N | select,insert,update,references | ||||
| inactive | char(1) | utf8_unicode_ci | NO | N | select,insert,update,references | ||||
| account_id | int(11) | NULL | NO | PRI | 0 | select,insert,update,references | |||
+----------------------------------------+--------------+-----------------+------+-----+-------------------+-------+---------------------------------+---------+
16 rows in set (0.02 sec)
mysql> select full_name from accounts where preferred_email like 'saper%' \G
full_name: Marcin Cieślak
1 row in set (0.00 sec)
Additionally, here's the output of my sane MySQL Gerrit instance via the Gerrit Inspector feature (patching your gerrit with https://gerrit-review.googlesource.com/#/c/34670/ should be mostly harmless :):
(lost of startup messages on Gerrit console)
"jettyserver" is "com.google.gerrit.pgm.http.jetty.JettyServer@1fdac8a5"
"db" is "com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$25@1c8aeedc"
Welcome to the Gerrit Inspector
Enter help() to see the above again, EOF to quit and stop Gerrit
Jython 2.5.2 (Release_2_5_2:7206, Mar 2 2011, 23:12:06)
[OpenJDK 64-Bit Server VM (Sun Microsystems Inc.)] on java1.6.0 running for Gerrit 2.4-rc0-78-g8ed6c15
for z in db.accounts().iterateAllEntities():
... print z.fullName
...
Marcin Cieślak
Marcin Cieslak (via gmail)
beau wrote:
Um... I am unable to log in to gerrit right now.
Application Error
Server Error
Cannot assign user name
Ok, everything should be squared away now. Usernames are now showing up properly[0], cover comments[1] and inline comments[2]. We also tested IRC--which works. E-mail notifs are working.
Only thing left to test is new user creation and login. Then we can mark this fixed.
[0] https://gerrit.wikimedia.org/r/#change,6008
[1] https://gerrit.wikimedia.org/r/#change,3962 (last comment)
[2] https://gerrit.wikimedia.org/r/#patch,sidebyside,3962,4,RELEASE-NOTES-1.20
sumanah wrote:
I've now created a user account via https://labsconsole.wikimedia.org/wiki/Special:CreateAccount for Paweł Sadowski and am waiting for Paweł to confirm that login for Labs & Gerrit works.
I went ahead and made myself a testing account so I can use it in the future. It worked
https://gerrit.wikimedia.org/r/#dashboard,240
Marking this FIXED.
As of now, the IRC bot says:
Lastlog:
04:42 < gerrit-wm> New review: Szymon ?wierkosz; "(no comment)" [mediawiki/extensions/ProofreadPage] (master) C: 1; - https://gerrit.wikimedia.org/r/6345
04:49 < gerrit-wm> New review: Szymon ?wierkosz; "(no comment)" [mediawiki/extensions/ProofreadPage] (master) C: 1; - https://gerrit.wikimedia.org/r/6340
13:39 < gerrit-wm> New patchset: Szymon ?wierkosz; "Convert a JS variable for horizontal layout to a preference." [mediawiki/extensions/ProofreadPage]
(master) - https://gerrit.wikimedia.org/r/6388
13:39 < gerrit-wm> New patchset: Szymon ?wierkosz; "Bug fixed : the proofreadpage_default_layout='horizontal' option doesn't work because of a change in the
html generated by wikieditor." [mediawiki/extensions/ProofreadPage] (master) - https://gerrit.wikimedia.org/r/6003
13:41 < gerrit-wm> New review: Szymon ?wierkosz; "Nothing changed between Patch Set 1 and Patch Set 2. It is one of my another failed attempts at usin..."
[mediawiki/extensions/ProofreadPage] (master) C: 1; - https://gerrit.wikimedia.org/r/6003
20:27 < gerrit-wm> New patchset: Szymon ?wierkosz; "Convert a JS variable for horizontal layout to a preference." [mediawiki/extensions/ProofreadPage]
(master) - https://gerrit.wikimedia.org/r/6388
20:27 < gerrit-wm> New patchset: Szymon ?wierkosz; "Bug fixed : the proofreadpage_default_layout='horizontal' option doesn't work because of a change in the
html generated by wikieditor." [mediawiki/extensions/ProofreadPage] (master) - https://gerrit.wikimedia.org/r/6003
13:08 < gerrit-wm> New review: Szymon ?wierkosz; "(no comment)" [mediawiki/core] (master) C: 0; - https://gerrit.wikimedia.org/r/6596
Fortunately, the HTML output seems fine - but something might have changed (is it because of 2.3)?
Can you have a look at 2.3 database again? Maybe it's just some interface to the IRC bot?
Did a simple test:
Added UTF-8 comment to:
https://gerrit.wikimedia.org/r/#/c/3289/
results:
$ ssh wikimedia gerrit stream-events
{"type":"comment-added","change":{"project":"test/mediawiki/core","branch":"master","topic":"master","id":"Icdc8f7e26c4cba920eda69a042702b8358797554","number":"3289","subject":"Testing git review...","owner":{"name":"IAlex","email":"ialex.wiki@gmail.com"},"url":"https://gerrit.wikimedia.org/r/3289"},"patchSet":{"number":"1","revision":"e5e3aafbce66df1b0a1094be7aa62c34a617c181","ref":"refs/changes/89/3289/1","uploader":{"name":"IAlex","email":"ialex.wiki@gmail.com"},"createdOn":1332230770},"author":{"name":"saper","email":"saper@saper.info"},"comment":"ąćęłńóśźć comment utf-8"}
But:
20:44 < gerrit-wm> New review: saper; "????????? comment utf-8" [test/mediawiki/core] (master) - https://gerrit.wikimedia.org/r/3289
(In reply to comment #27)
As of now, the IRC bot says:
Lastlog:
04:42 < gerrit-wm> New review: Szymon ?wierkosz; "(no comment)"
[mediawiki/extensions/ProofreadPage] (master) C: 1; -
https://gerrit.wikimedia.org/r/6345
04:49 < gerrit-wm> New review: Szymon ?wierkosz; "(no comment)"
[mediawiki/extensions/ProofreadPage] (master) C: 1; -
https://gerrit.wikimedia.org/r/6340
13:39 < gerrit-wm> New patchset: Szymon ?wierkosz; "Convert a JS variable for
horizontal layout to a preference." [mediawiki/extensions/ProofreadPage](master) - https://gerrit.wikimedia.org/r/638813:39 < gerrit-wm> New patchset: Szymon ?wierkosz; "Bug fixed : the
proofreadpage_default_layout='horizontal' option doesn't work because of a
change in thehtml generated by wikieditor."[mediawiki/extensions/ProofreadPage] (master) -
https://gerrit.wikimedia.org/r/6003
13:41 < gerrit-wm> New review: Szymon ?wierkosz; "Nothing changed between Patch
Set 1 and Patch Set 2. It is one of my another failed attempts at usin..."[mediawiki/extensions/ProofreadPage] (master) C: 1; -https://gerrit.wikimedia.org/r/6003
20:27 < gerrit-wm> New patchset: Szymon ?wierkosz; "Convert a JS variable for
horizontal layout to a preference." [mediawiki/extensions/ProofreadPage](master) - https://gerrit.wikimedia.org/r/638820:27 < gerrit-wm> New patchset: Szymon ?wierkosz; "Bug fixed : the
proofreadpage_default_layout='horizontal' option doesn't work because of a
change in thehtml generated by wikieditor."[mediawiki/extensions/ProofreadPage] (master) -
https://gerrit.wikimedia.org/r/6003
13:08 < gerrit-wm> New review: Szymon ?wierkosz; "(no comment)"
[mediawiki/core] (master) C: 0; - https://gerrit.wikimedia.org/r/6596Fortunately, the HTML output seems fine - but something might have changed (is
it because of 2.3)?Can you have a look at 2.3 database again? Maybe it's just some interface to
the IRC bot?
Could this be bug 36487?