Today I built a web fontend for Marian MT with OPUS data models:

In T232986#5576061, @Jpita wrote:

@santhosh why does Google not show here

if it is available here https://cxserver.wikimedia.org/v2/list/mt/es/en?

As a start, I attempted to make the development and deployment of the system simple. I creaded a Docker container that takes care of setting up the system - https://github.com/Helsinki-NLP/Opus-MT/pull/1
It need more work. My aim is to get a webserver that abstract MT engines and complexities, and expose a web api that does translation

This topic was discussed in details and the current undestanding is given below:

<section id="cxTargetSection0" data-mw-cx-source="undefined">

<p id="mwAg">While the evolution of sex dates to the prokaryote or early eukaryote stage,
    <span typeof="mw:Nowiki" id="mwAw">
        Bernstein H and Bernstein C (2013). Evolutionary Origin and Adaptive Function of Meiosis. In Meiosis:
        Bernstein C and Bernstein H, editors. Chapter 3: pp. 41–75 </span>
    <span typeof="mw:Transclusion"
        data-mw="{&quot;parts&quot;:[{&quot;template&quot;:{&quot;target&quot;:{&quot;wt&quot;:&quot;ISBN&quot;,&quot;href&quot;:&quot;./Template:ISBN&quot;},&quot;params&quot;:{&quot;1&quot;:{&quot;wt&quot;:&quot;978-953-51-1197-9&quot;}},&quot;i&quot;:0}}]}"
        data-ve-no-generated-contents="true" about="#mwt2" data-mw-deduplicate="TemplateStyles:r886058088"
        id="mwBA">&nbsp;
    </span> <span data-segmentid="4" class="cx-segment">
        <a href="./International%20Standard%20Book%20Number" rel="mw:WikiLink" about="#mwt2" class="cx-link"
            data-linkid="5" title="International Standard Book Number">ISBN
        </a> <span about="#mwt2" typeof="mw:Entity">&nbsp;</span>
        <a href="./Special:BookSources/978-953-51-1197-9" rel="mw:WikiLink" data-linkid="6" about="#mwt2"
            class="cx-link" id="mwBQ" title="Special:BookSources/978-953-51-1197-9">978-953-51-1197-9
        </a> </span> </p>

</section>

Notes on how does section translation differs from full article translation

Technical implementation related notes:

Demo https://codepen.io/santhoshtr/pen/zjMMrG

https://cxserver.wikimedia.org/v2/list/mt/es/en lists all available MT engines now

For Quick QA: Using the content of

In my local testing I see that all params are adapted. Next step is to investigate the cxserver deployment related issues if any.

I think we need to recognize two extremes of category based suggestions. One is to find articles from a narrow category such as "rivers in serbia". Another one is "I am interested in Sports". The second type of users is more common because it is an abstract way of expressing an interest. The first one is about a specific search. If we try to serve both of these using a single interface, it is going to complicate. We should also see how this selection is done in non-wikipedia applications. I have used news reader applications, applications like magzter etc, where they provide a single flat list just enough to fill a mobile screen. For the first type of selection-narrow selection- a search box is given. If we enhance our source selector so that it can search in "Wikipedia article categories" we still can support such use cases. The nature of a second level or narrower topic is that, you can easily find one that is not in that list.

In T229242#5562697, @Pginer-WMF wrote:

I edited the description to capture a summary of the outcome and the design implications. Please, @santhosh and @Petar.petkovic, check and suggest any corrections.
I have some further questions about the implications of the approach:

• Is it possible to support the option to discard suggestions? Current suggestions have an option ("x") to discard them, preventing them to be shown again. I was wondering whether such system would work out-of-the-box with this new kind of suggestion, or they require further consideration (e.g., hiding such option for these suggestions, or support it differently).

In T232020#5551479, @Jpita wrote:

I just translated this article and is still not linking to wikidata
https://pt.wikipedia.org/wiki/Marianne_(serie_de_televis%C3%A3o)
https://www.wikidata.org/wiki/Q67521330

About the timeout of queries, one think I observed is, the time taken is a multiple of number of P31 values we use at a time. If we use only one topic at a time, I see that the time taken is in reasonable limits. This can be compensated by allowing only topic selection at a time as I did in my demo application.

I was exloring the Wikidata queries based on Property:P31 https://www.wikidata.org/wiki/Property:P31 and the subclasses. Here is a sample query https://w.wiki/9Cb - it tries to find all articles in a given wiki page with P31 or subclasses of P31 associated with.

@ssastry, when it is ready, let us know. We can do required testing and changes if any.

Screenshot from the above patch:

The VE config that shows the welcome dialog for first time editor is

After the patch https://gerrit.wikimedia.org/r/c/mediawiki/services/cxserver/+/537982

The adaptation happened correctly, but the en.wiki reference content has a te mplate style. - https://en.wikipedia.org/wiki/Module:Citation/CS1/styles.css. The templatestyle does not exist in he.wikipedia and MW parser is throwing that error in article page. yi and kab languages have corresponding template styles. To make it clear a ref with templatestyle is a valid markup. it becomes problematinc only when that template style does not exist in target wiki

Details for each version are based on the original tickets (T216032, T216500), but detailed below for additional clarity:

Here the word "version" means, one of the two designs, right?

I would request to have the language code as part of API path than in the domain name. It is an important semantic value affecting API output. Having it in domain name as in https://en.wikipedia.org/api/rest_v1/ has mutliple issues that became obvious while writing apis to host at wikipedia.org/api and xx.wikipedia.org/api at the same. See https://wikimedia.org/api/rest_v1/ and https://en.wikipedia.org/api/rest_v1/

@Ladsgroup, Need some help here. I guess the above list of modules are no longer needed, and I can just use

This is a regression from T203696.

@diego, Would it be possible to add a Dockerfile to the repo so that a ML-nonexperienced person can get it running and extract the outputs?

Seems working fine now. Here is a video

For future references, we can save time if following information is also present in bug report

// If replacing the suggestion cards by the button in the same place (below the text), makes it technically much more easy to support,
this is easy. But are we going to show this one every time or based on our old code to detect the probability that the translator will translate? That code was also removed since we decided to show this more than once to a user when a translation suggestion available. Please clarify.

we removed this dialog, and have new dialog with translation suggestion. https://phabricator.wikimedia.org/T216500 overrides this ticket

In T216500#5440713, @Petar.petkovic wrote:

Me and @Pginer-WMF were discussing why the design has changed from T216032, when the invite was not shown to anyone yet, since no wikis have Content Translation out of beta. Turns out that generic dialog from T216032 should be displayed when we don't have a good source article to suggest to the user. I will update description of this ticket to include new information.

If I understand correctly, https://gerrit.wikimedia.org/r/c/oojs/ui/+/530960 includes the combining marks after the search match to the highlight range, so that we don't add a highlight boundary before combining mark. A consequence of this approach is, search results that are completely different words will be highlighted. To illustrate this:

declining quality can only be attributed to MT service. It is either translated or not translated from cxserver perspective. How good the translation is always with MT service. But let us watch this ticket after https://gerrit.wikimedia.org/r/529727 is merged

In T223888#5361311, @Jpita wrote:

@santhosh anything I can check here?

Screenshot after the above patch - from en:Grüneisen_parameter

Here is an example where the MT calcualtion does not happen. From es.wiki

Screenshot based on the above patch

In T227571#5397881, @Jpita wrote:

@santhosh where can I find this api playground/documentation page?
Can you provide some info on how I can test this ?

Thanks. That works

Revert patch for T220495: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ContentTranslation/+/528406

I had to revert this fix because of T229906: Sentence pair highlighting broken
Revert patch: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ContentTranslation/+/528406

Assuming Apertium was the MT engine used.

We cannot remove the wrapping tags for math since that would cause unexpected rendering and editing behavior. Better keep the DOM as such. I propose excluding <dl> tags from validation. Math is under dl ->dd -> p tag.

The feature does not differentiate between wikis. I suspect caching related inconsistencies.

It seems there are two issues here. A gray reference link is unadapted reference. This ticket made the reference code show only the edit button for such references.

In T214256#5376632, @Jpita wrote:

This one is definitely a bug. Happening with Apertium. Checking.

@santhosh are you going to be fixing this issue on this task or a different one?

In T214256#5374430, @Jpita wrote:

@Pginer-WMF is this a different issue?
Is there a ticket already for this?

In T214256#5373818, @Jpita wrote:

update: this might not be related since the example on my last screenshot is a template and not a math formula

In T214256#5373782, @Jpita wrote:

not sure this is related to this ticket, but I never saw a math formula being translated like this.
it is like this in production as well.
waiting for @santhosh feedback to see if it is related or not.

Thanks @Krinkle. The usecase is not a widely used one to justify an effort to implement this feature. At least for now. For now, I used a workaround as https://gerrit.wikimedia.org/r/525521

yes, I think the math support need to improve and that can fix both. First thing to do is avoid sending math sections to MT engines so that they don't mangle it.

Found that if I used Yandex instead of Google, the above section translated correctly

I was not able to reproduce the case of https://en.wikipedia.org/wiki/Chronology_of_Gabon from en to es. But I was able to reproduce with https://en.wikipedia.org/wiki/Proof_that_%CF%80_is_irrational en to pt.

So the correct thing to do here is to implement this special mapping in LanguageTr::ucfirst().

When we say language in the context of wiki, there is a"interface language" - the language of user interface and then content language- that is the content language of wiki. So if you are in germal wikipedia, the content language is german. But the interface language can be anything(for example English). If interface language is English, the items in sidebar, top menu etc will be in English.