Page MenuHomePhabricator

CX2: ISBN doubled, one correctly formatted with {{ISBN}}, another incorrectly formatted with [[Special:BookSources]]
Open, MediumPublic

Description

I found this strange problem in several articles recently translated with CX2 : ISBN are added twice to the translation, the first one is correctly formatted with the {{ISBN}} template, the other is using [[Special:BookSources]] which should be avoided for readability.

Exampe on frwiki with Anarchisme explorateur, but I saw the same problem on other articles by different users (HMS Avon Vale (L06), Toxophilus...) and only recently (at most a few weeks).

{{ISBN|987-580-092-9}}  [[International Standard Book Number|ISBN]]   [[Special:BookSources/987-580-092-9|987-580-092-9]]

Related: T213255: CX2: Doesn't handle correctly ISBN, should not put nowiki tags around them


I created This page compiles examples of the issue and a a quick test link to try translating them.

Screenshot 2019-12-24 at 09.40.24.png (735×1 px, 193 KB)

As shown in the screenshot, the issue seems to be on how the ISBN elements are rendered in the source, where some instances are already decomposed in several elements (template + link) that get transferred into the translation. The most surprising part is that the issue happens on some instances (see 2nd bullet point) but not others (1st bullet point), being all of them using the same template. So maybe these being part of a list or processing them in sequence is making a difference.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
NicoV updated the task description. (Show Details)

Any news on this one?
CX2 continues to produce doubled ISBN number on articles in production wikis.

Example of recent fix on frwiki: Château de Takashima, Château de Nagaoka, Bridget Cleary...

Still happening, see for example Little Gidding (poème)

Checking the translation in the translation debugger I see the startTimestamp: 20190514133948 value which seems to indicate the translation was started on May 14 2019. So it may contain contents added before the issue was fix, even if it has been published recently.
In any case, we may want to test the fix by translating "Little Gidding (poem)" form English to French and check if there are still issues with ISBNs

I confirm this still happens in production, translating {{ISBN ends up as {{ISBN+ [[Special:BookSources/ on the target article
https://fr.wikipedia.org/w/index.php?title=Utilisateur:Test7206310/Little_Gidding_(poem)&action=edit

I confirm this still happens in production, translating {{ISBN ends up as {{ISBN+ [[Special:BookSources/ on the target article
https://fr.wikipedia.org/w/index.php?title=Utilisateur:Test7206310/Little_Gidding_(poem)&action=edit

Ok. I'll add it to the quarter board as a follow-up of T213255

Change 541772 had a related patch set uploaded (by Santhosh; owner: Santhosh):
[mediawiki/services/cxserver@master] Do not remove transclusions when they are also template styles

https://gerrit.wikimedia.org/r/541772

Change 541772 merged by jenkins-bot:
[mediawiki/services/cxserver@master] Do not remove transclusions when they are also template styles

https://gerrit.wikimedia.org/r/541772

Change 543386 had a related patch set uploaded (by KartikMistry; owner: KartikMistry):
[operations/deployment-charts@master] Update cxserver to 2019-10-15-091114-production

https://gerrit.wikimedia.org/r/543386

In production, translating this article https://en.wikipedia.org/wiki/User:Jpita23/test has this result

image.png (515×1 px, 148 KB)

As you can see, the {{ISBN}} template is missing from the original article and the translation when loaded in cx.

Change 543386 merged by jenkins-bot:
[operations/deployment-charts@master] Update cxserver to 2019-10-15-091114-production

https://gerrit.wikimedia.org/r/543386

Mentioned in SAL (#wikimedia-operations) [2019-10-16T12:00:13Z] <kart_> Updated cxserver to 2019-10-15-091114-production (T234773, T217585)

It seems the duplication of the ISBN is still happening.
translating https://en.wikipedia.org/wiki/User:Jpita23/test

While the evolution of sex dates to the prokaryote or early eukaryote stage,<nowiki>Bernstein H and Bernstein C (2013). Evolutionary Origin and Adaptive Function of Meiosis. In Meiosis: Bernstein C and Bernstein H, editors. Chapter 3: pp. 41–75 </nowiki>{{ISBN|978-953-51-1197-9}}

ends up as

Mientras que la evolución del sexo data de la etapa procariota o eucariota temprana, o ln,,m,m.,m C (2013). Origen evolutivo y función adaptativa de la meiosis. En Meiosis: Bernstein C y Bernstein H, editores. Capítulo 3: págs. 41–75 {{ISBN|978-953-51-1197-9}} [[International Standard Book Number|ISBN]] &nbsp; [[Special:BookSources/978-953-51-1197-9|978-953-51-1197-9]]

Input:

<section id="cxTargetSection0" data-mw-cx-source="undefined">

<p id="mwAg">While the evolution of sex dates to the prokaryote or early eukaryote stage,
    <span typeof="mw:Nowiki" id="mwAw">
        Bernstein H and Bernstein C (2013). Evolutionary Origin and Adaptive Function of Meiosis. In Meiosis:
        Bernstein C and Bernstein H, editors. Chapter 3: pp. 41–75 </span>
    <span typeof="mw:Transclusion"
        data-mw="{&quot;parts&quot;:[{&quot;template&quot;:{&quot;target&quot;:{&quot;wt&quot;:&quot;ISBN&quot;,&quot;href&quot;:&quot;./Template:ISBN&quot;},&quot;params&quot;:{&quot;1&quot;:{&quot;wt&quot;:&quot;978-953-51-1197-9&quot;}},&quot;i&quot;:0}}]}"
        data-ve-no-generated-contents="true" about="#mwt2" data-mw-deduplicate="TemplateStyles:r886058088"
        id="mwBA">&nbsp;
    </span> <span data-segmentid="4" class="cx-segment">
        <a href="./International%20Standard%20Book%20Number" rel="mw:WikiLink" about="#mwt2" class="cx-link"
            data-linkid="5" title="International Standard Book Number">ISBN
        </a> <span about="#mwt2" typeof="mw:Entity">&nbsp;</span>
        <a href="./Special:BookSources/978-953-51-1197-9" rel="mw:WikiLink" data-linkid="6" about="#mwt2"
            class="cx-link" id="mwBQ" title="Special:BookSources/978-953-51-1197-9">978-953-51-1197-9
        </a> </span> </p>

</section>

Actual Output:

<section data-mw-cx-source="undefined" id="cxTargetSection0">
<p id="mwAg"> Enquanto a evolução do sexo data do estágio procarionte ou eucarionte inicial, <span id="mwAw" typeof="mw:Nowiki">Bernstein H e Bernstein C (2013). Origem Evolutiva e Função Adaptativa da Meiose. In Meiosis: Bernstein C e Bernstein H, editores. Capítulo 3: pp. 41–75</span>
<span class="cx-segment" data-segmentid="4">
<a about="#mwt2" class="cx-link" data-linkid="5" href="./International%20Standard%20Book%20Number" rel="mw:WikiLink" title="International Standard Book Number">ISBN</a>
<span about="#mwt2" typeof="mw:Entity"> </span>
<a about="#mwt2" class="cx-link" data-linkid="6" href="./Special:BookSources/978-953-51-1197-9" id="mwBQ" rel="mw:WikiLink" title="Special:BookSources/978-953-51-1197-9">978-953-51-1197-9</a></span>
</p>
</section>

Expected output

<section data-mw-cx-source="undefined" id="cxTargetSection0">
<p id="mwAg"> Enquanto a evolução do sexo data do estágio procarionte ou eucarionte inicial,
<span id="mwAw" typeof="mw:Nowiki">Bernstein H e Bernstein C (2013). Origem Evolutiva e Função Adaptativa da Meiose. In Meiosis: Bernstein C e Bernstein H, editores. Capítulo 3: pp. 41–75</span> <span about="#mwt2" data-cx="[{&#34;adapted&#34;:true,&#34;partial&#34;:false,&#34;targetExists&#34;:true}]" data-mw="{&#34;parts&#34;:[{&#34;template&#34;:{&#34;target&#34;:{&#34;wt&#34;:&#34;ISBN&#34;,&#34;href&#34;:&#34;./Predefinição:ISBN&#34;},&#34;params&#34;:{&#34;1&#34;:{&#34;wt&#34;:&#34;978-953-51-1197-9&#34;}},&#34;i&#34;:0}}]}" data-mw-deduplicate="TemplateStyles:r886058088" data-ve-no-generated-contents="true" id="mwBA" typeof="mw:Transclusion"> sds

</span>

<a about="#mwt2" class="cx-link" data-linkid="5" href="./International%20Standard%20Book%20Number" rel="mw:WikiLink" title="International Standard Book Number">ISBN</a> <span about="#mwt2" typeof="mw:Entity"> </span>
<a about="#mwt2" class="cx-link" data-linkid="6" href="./Special:BookSources/978-953-51-1197-9" id="mwBQ" rel="mw:WikiLink" title="Special:BookSources/978-953-51-1197-9">978-953-51-1197-9</a> &#60;/span&#62; </p>
</section>

There seems to be a bug in the lineardoc that cause missing the transclusion.

Change 544163 had a related patch set uploaded (by Santhosh; owner: Santhosh):
[mediawiki/services/cxserver@master] LinearDoc: Do not allow to remove empty spans in expansion step

https://gerrit.wikimedia.org/r/544163

Screenshot after the above patch:

image.png (411×945 px, 181 KB)

Change 547164 had a related patch set uploaded (by Santhosh; owner: Santhosh):
[mediawiki/services/cxserver@master] Do not consider style with transclusion content as textblock

https://gerrit.wikimedia.org/r/547164

Change 547164 merged by jenkins-bot:
[mediawiki/services/cxserver@master] Do not consider style with transclusion content as textblock

https://gerrit.wikimedia.org/r/547164

Change 544163 merged by jenkins-bot:
[mediawiki/services/cxserver@master] LinearDoc: Do not allow to remove empty spans in expansion step

https://gerrit.wikimedia.org/r/544163

Change 555784 had a related patch set uploaded (by KartikMistry; owner: KartikMistry):
[operations/deployment-charts@master] Update cxserver to 2019-12-05-090549-production

https://gerrit.wikimedia.org/r/555784

Change 555784 merged by jenkins-bot:
[operations/deployment-charts@master] Update cxserver to 2019-12-05-090549-production

https://gerrit.wikimedia.org/r/555784

Mentioned in SAL (#wikimedia-operations) [2019-12-09T08:49:19Z] <kart_> Updated cxserver to 2019-12-05-090549-production (T217585, T230195)

NicoV reopened this task as Open.EditedDec 22 2019, 11:47 AM

Last comments say that it has been deployed in production the 9th, but CX2 is still producing the same problem the 15th... Example: frwiki 1, frwiki 2, frwiki 3.

Thanks for the examples @NicoV
I created a page that compiles examples from those three pages and a a quick test link to try translating them.

As shown in the screenshot above, the issue seems to be on how the ISBN elements are rendered in the source, where some instances are already decomposed in several elements (template + link) that get transferred into the translation. The most surprising part is that the issue happens on some instances (see 2nd bullet point) but not others (1st bullet point), being all of them using the same template. So maybe these being part of a list or processing them in sequence is making a difference.

Screenshot 2019-12-24 at 09.40.24.png (735×1 px, 193 KB)

Any progress on this problem?

It keeps happening on articles translated by CX2... Examples on frwiki:

Any progress on this problem?

No progress yet. I added the test example to the description for future reference.

@Pginer-WMF
It's been reported 2 months ago, with examples and it keeps happening on a regular basis, needing clean up after CX2 (usually from other people than the one creating the problems with CX2...). When will it be fixed ?

Other examples :

and that's only for frwiki for the last few days (between the dumps from April 20th and May 1st)

@Pginer-WMF
It's been reported 2 months ago, with examples and it keeps happening on a regular basis, needing clean up after CX2 (usually from other people than the one creating the problems with CX2...). When will it be fixed ?

Our team capacity to respond to bugs has been limited during the recent months, but I hope we can continue the work with this and other maintenance efforts soon.