Disable Esperanto conversion on Esperanto wikis
Closed, ResolvedPublic

Description

"Esperanto magical conversion" is a feature that automatically converts some character combinations, such as cx, jx and a few others to Esperanto letters such as ĉ and ĵ.

This feature was useful in the first years of Wikipedia when there were no smart and generic typing solutions, but by 2014 the input method functionality in the Universal Language Selector can do the same in a way that is more robust technically and more flexible to the user as well.

I already brought up the topic of replacing the magical conversion with ULS in conversations with several Esperanto Wikipedia editors, and there seemed to be general agreement to the step.

What is needed currently:

  1. More formal agreement of Esperanto wiki communities to doing this. Discussion pages in eo.wiki* projects should probably be used for that. The discussions can be held in Esperanto - Brion and I can read it.
  1. A technical approval from somebody who is familiar with the feature. For example, is it OK to just disable it, or should some conversion run on the the current data? There may also be other technical points that I am not thinking of now.
  1. ULS must be re-enabled by default, at least on these wikis.
  1. The community members should choose what should be the default keyboard layout. We have "h-f", "h", "plena", "q", "transliteration", "vi" and "x". If I'm not mistaken, the "x" system is the closest to the current "magical conversion", but the communities may prefer something different. (The will be able to choose another system, or to disable it altogether.)

Please add any other notes that I forgot.

(Related: Bug 21781.)


Version: wmf-deployment
Severity: normal

Details

Reference
bz60677
bzimport set Reference to bz60677.
bzimport added a subscriber: Unknown Object (MLST).
Amire80 created this task.Jan 31 2014, 2:53 PM
brion added a comment.Jan 31 2014, 4:15 PM

There won't be any conversion required; the way the old system worked was to convert on-the-fly when shipping data out to be edited, and convert it back when saving, so the database contains "regular" data with the real characters.

Disabling it should be a simple matter of removing languages/classes/LanguageEo.php -- there's nothing left in it other than the old conversion support, so with that removed it should fall back to standard behavior.

But yeah, need to confirm that ULS will be back on and either a conversion enabled by default or make sure people are comfortable turning it on manually.

brion added a comment.Jan 31 2014, 4:22 PM

I also just want to say how excited I am that one of my earlier MediaWiki features is being replaced with a better, more modern system. :D Yay!

(In reply to comment #0)

  1. The community members should choose what should be the default keyboard layout. We have "h-f", "h", "plena", "q", "transliteration", "vi" and "x". If I'm not mistaken, the "x" system is the closest to the current "magical conversion", but the communities may prefer something different. (The will be able to choose another system, or to disable it altogether.)

I meant to write "The ^user^ will be able to choose another system, or to disable it altogether."

(In reply to comment #2)

I also just want to say how excited I am that one of my earlier MediaWiki
features is being replaced with a better, more modern system. :D Yay!

Brion, thank you for the compliment, and for the technical clarifications!

Change 110589 had a related patch set uploaded by Brion VIBBER:
Remove old Esperanto character conversion support

https://gerrit.wikimedia.org/r/110589

What about editors who disable JavaScript? Does the ULS extension convert server-side?

KuboF added a subscriber: KuboF.Feb 23 2015, 11:42 PM

There is a discussion about this topic now on Esperanto Wikipedia https://eo.wikipedia.org/wiki/Vikipedio:Diskutejo/Diversejo#Anstata.C5.ADigo_de_x-konvertado_en_Vikipedio
There was raised concerns about users with disabled JavaScript. Do you have some statistics about ratio of editors (not readers) with disabled JS or ratio of edits made by such users? Can you advise some solution for transition?

I don't have such statistics.

IME in ULS cannot work without JavaScript. The only thing I can think of is to suggest all users to install a native Esperanto keyboard in their operating system.

I have contacted communities of all Esperanto Wikimedia projects with own domain. There was very active discussion on Esperanto Wikipedia; on other projects the proposal worked mostly like just announcement. There is clear consensus for disabling the nowadays conversion and simultaneously enable the "eo-x" conversion in ULS as default (for both logged in and anonymous) with no change for those who are already using customized ULS profile.

Discussions on the projects:

Thanks for your work guys, especially @brion for the first versions of the conversion *thumb up*

In order to further simplify the transition, I have set up an automatic filter that tags edits where the user has used surrogate letters instead of proper Esperanto letters. This way, we will be able to detect users who might be confused by the fact that the conversion now occurs on-the-fly (as opposed to after submit) and would force the surrogates into the edit window, and also any other users for whom the ULS did not work as expected. KuboF and I will be monitoring such tagged edits after the transition. The filter: https://eo.wikipedia.org/wiki/Special:Abusefilter/3

@KuboF and @Blahma, thank you so much for your help.

@brion, what is left to be done?

Blahma added a comment.Jun 8 2015, 8:34 AM

@brion, when do you think you'll find the time to have a look at this and complete the process?

I think the communities are ready (and that we should not wait too long until the issues gets forgotten and new people come to whom we would need to explain the reasons and the consequences of the transition once again).

Restricted Application added subscribers: Matanya, Aklapper. · View Herald TranscriptAug 12 2015, 12:41 AM

@Krenair: What kind of community consensus do we need here anymore? @KuboF's post from Apr 15 has demonstrated that community consensus has been reached on all Wikimedia wikis that use Esperanto as its content language. I think we are now waiting only for someone to review @brion's patch and apply it. Considering the specifity (and age) of this issue, how can we find a suitable reviewer?

Krenair set Security to None.

@Krenair: What kind of community consensus do we need here anymore? @KuboF's post from Apr 15 has demonstrated that community consensus has been reached on all Wikimedia wikis that use Esperanto as its content language. I think we are now waiting only for someone to review @brion's patch and apply it. Considering the specifity (and age) of this issue, how can we find a suitable reviewer?

That was just because it also included the Community-consensus-needed tag - I've been trying to clean up the Wikimedia-Site-requests workboard. Also this is not really a Wikimedia-Site-requests task, moved to MediaWiki-Internationalization

Change 110589 had a related patch set uploaded (by Alex Monk):
Remove old Esperanto character conversion support

https://gerrit.wikimedia.org/r/110589

Krenair assigned this task to brion.Aug 12 2015, 12:56 PM
Pinof added a subscriber: Pinof.Sep 24 2015, 7:27 AM

We are waiting, for us it is a critical stage, when can you hope the process to begin ?😕

LLarson added a subscriber: LLarson.Jun 6 2016, 9:48 PM

We’re all excited to finally move on. What else can be done to help expedite this and how can we go about testing @brion’s patch in the meantime?

Psychoslave added a subscriber: Psychoslave.

Is there any technical blocker here?

Is there any technical blocker here?

Saluton, @Amire80. I can’t find documentation on what a technical blocker refers to. But, this is a two‑year‑old thorn in our sides, so how can I help, please?

brion added a comment.Jul 18 2016, 7:33 PM

Ok as far as I know there's no blockers, this patch just got lost in review. :( Since nobody else is really poking at this end of the old localization support services I guess testing is up to me. ;) I'm doing a quick review to make sure the current state of the patch is working, then will merge it.

It should then go out either with the next scheduled update, or we can push it via one of the earlier update windows.

Change 110589 merged by jenkins-bot:
Remove old Esperanto character conversion support

https://gerrit.wikimedia.org/r/110589

Jdforrester-WMF closed this task as "Resolved".Jul 18 2016, 8:03 PM
Jdforrester-WMF edited projects, added User-notice; removed Patch-For-Review.

I'm not sure if this is a big enough impact to note in Tech/News; I'll leave that judgement to the wonderful T/N team. :-)

@Jdforrester-WMF: Impact of the change is for Esperanto language projects only, all of which have been notified and the issue was discussed with the communities in detail before advancing to making changes.

From history-writing point of view, the change might easily be made remarkable if linked to Brion Vibber's introducing of Unicode in Wikipedia of which this is an early relict. For full historical account and more context, see http://blogs.transparent.com/esperanto/unicoding-the-esperanto-wikipedia/ – and, of course, the iconic https://en.wikipedia.org/wiki/Wikipedia:Brion_Vibber_Day :-)

Thank you very much, @brion. A bit of history was made.

I believe that this is worth mentioning in Tech News.

Wonderful. What is the deployment process after the review is validated?

@Blahma, @Psychoslave, @LLarson, @Pinof, @KuboF: The X-system is supposed to disappear on Thursday. Morning U.S. time, evening European time, late night Asian time.

I suggest starting as soon as possible to test that the typing support using the built-in typing actually works. Just to be certain that we're all on the same page, I'm talking about the little button with a keyboard, which appears when you click any text box. You can select "x sistemo", "vi sistemo", "q sistemo" and several others there—whatever is comfortable for each user. If you haven't already, it makes sense to add some documentation to your help pages about this. I'll be happy to help you figure out how this works if you have any issues. (I am able to read Esperanto and even write it a bit.)

I tested a bit and it seems like it works. I find the "vi sistemo" rather inconvenient for typing things like "neu" (ne/i), or any word with "eu" but well it works as specified, and the combination is not that common in Esperanto. Also it looks like one have to select a input method before the corresponding help link appears. Now admittedly having a question mark at the end of each line would make it visually overloaded, but maybe having the question mark appearing on mouse other could make the trick.

Trizek-WMF added a subscriber: Trizek-WMF.

I'm not sure if this is a big enough impact to note in Tech/News; I'll leave that judgement to the wonderful T/N team. :-)

With only one language impacted, that change is out of Tech News inclusion criteria I'm afraid.

That said, it should be announced to Esperanto communities. @Psychoslave, do you plan to announce it to the community?

I was excited—I already announced on Wikipedia...

I've seen that, and that's cool! :)
Don't forget to inform the other wikis about that change too (Vikivortaro, Vikifontaro, Vikicitaro, Vikilibroj...).

Ok as far as I know there's no blockers, this patch just got lost in review. :( [...]

Apparently the change got merged without the addition of any sort of check to reject (or correctly process) edits started prior to the MediaWiki update, as I mentioned in my Jun 18, 2014 comment on the change. Is there some other plan to prevent pages from getting messed up just by clicking the save button?

brion added a comment.Jul 20 2016, 6:27 PM

Ok as far as I know there's no blockers, this patch just got lost in review. :( [...]

Apparently the change got merged without the addition of any sort of check to reject (or correctly process) edits started prior to the MediaWiki update, as I mentioned in my Jun 18, 2014 comment on the change. Is there some other plan to prevent pages from getting messed up just by clicking the save button?

Nope, there is no plan. Simply revert the presumably small number of affected revisions manually.

@LLarson, thank you for the announcement on Wikipedia. Tell me if you would like to make it on others projects as well, or if you are missing time for it, I may do it.

@LLarson, thank you for the announcement on Wikipedia. Tell me if you would like to make it on others projects as well, or if you are missing time for it, I may do it.

@Psychoslave: Please go ahead; I’m only up for a midnight snack. Thank you!!

Thank you all for achieving this together – it is a nice gift on the eve of the 101st World Esperanto Congress 2016 in Nitra, Slovakia! Me and KuboF will make sure to spread the word there too. In parallel to Brion Vibber Day, we might want to start calling July 21st the Amir E. Aharoni Day. Long live multilingualism and multilingual input!

Upon suggestion by @Venca24, I have just created a sitewide edit notice at https://eo.wikipedia.org/wiki/MediaWiki:Editpage-head-copy-warn that should keep being displayed above the wikitext editor until all active users may be safely supposed to have remarked the change. There have been a few reactions in the Village Pump (Vikipedio:Diskutejo), but all were mild and people received explanations on how to use the Universal Language Selector instead.

The abuse filter https://eo.wikipedia.org/wiki/Special:Abusefilter/3 already in place since April 2015 is capturing edits where the surrogate letters were actually saved into page content.

I also noted this important event in https://eo.wikipedia.org/wiki/Vikipedio:Historio#2016

Thank you all for achieving this together – it is a nice gift on the eve of the 101st World Esperanto Congress 2016 in Nitra, Slovakia! Me and KuboF will make sure to spread the word there too. In parallel to Brion Vibber Day, we might want to start calling July 21st the Amir E. Aharoni Day. Long live multilingualism and multilingual input!

I'm flattered, but it should be more like @santhosh Thottingal day, for developing jquery.ime :)

Ankoraŭan dankegon al ĉiuj—kaj ĝis la sekva Amir–Santhosh–tago! 💚💚💚