Page MenuHomePhabricator

Unwanted HTML content sent to HTML MT systems often crosses character limit.
Closed, ResolvedPublic

Description

For example, a sample html sent for a 3 sentence paragraph

1<section id="cxTargetSection0" data-mw-cx-source="undefined"><p id="mwAw"><span data-segmentid="4" class="cx-segment"><b id="mwBA">Phantosmia</b> (phantom smell), also called an <b id="mwBQ">olfactory hallucination</b> or a <b id="mwBg">phantom odor</b><sup typeof="mw:Extension/ref" data-mw="{&quot;name&quot;:&quot;ref&quot;,&quot;attrs&quot;:{&quot;name&quot;:&quot;That stinks&quot;},&quot;body&quot;:{&quot;id&quot;:&quot;mw-reference-text-cite_note-That_stinks-1&quot;,&quot;html&quot;:&quot;<cite about=\&quot;#mwt7\&quot; class=\&quot;citation news\&quot; data-mw=\&quot;{&amp;quot;parts&amp;quot;:[{&amp;quot;template&amp;quot;:{&amp;quot;target&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;cite news &amp;quot;,&amp;quot;href&amp;quot;:&amp;quot;./Template:Cite_news&amp;quot;},&amp;quot;params&amp;quot;:{&amp;quot;title&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;That stinks! 1 in 15 Americans smell odors that aren’t there&amp;quot;},&amp;quot;url&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;https://www.nih.gov/news-events/news-releases/stinks-1-15-americans-smell-odors-arent-there&amp;quot;},&amp;quot;accessdate&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;23 August 2018&amp;quot;},&amp;quot;work&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;National Institutes of Health (NIH)&amp;quot;},&amp;quot;agency&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;National Institute on Deafness and Other Communication Disorders&amp;quot;},&amp;quot;publisher&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;National Institute on Deafness and Other Communication Disorders&amp;quot;},&amp;quot;date&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;August 16, 2018&amp;quot;},&amp;quot;language&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;en-US&amp;quot;}},&amp;quot;i&amp;quot;:0}}]}\&quot; id=\&quot;mwtA\&quot; typeof=\&quot;mw:Transclusion\&quot; data-ve-no-generated-contents=\&quot;true\&quot;><a class=\&quot;external text\&quot; href=\&quot;https://www.nih.gov/news-events/news-releases/stinks-1-15-americans-smell-odors-arent-there\&quot; id=\&quot;mwtQ\&quot; rel=\&quot;mw:ExtLink\&quot;>\&quot;That stinks! 1 in 15 Americans smell odors that aren't there\&quot;</a>. <i id=\&quot;mwtg\&quot;>National Institutes of Health (NIH)</i>. National Institute on Deafness and Other Communication Disorders. National Institute on Deafness and Other Communication Disorders. August 16, 2018<span class=\&quot;reference-accessdate\&quot; id=\&quot;mwtw\&quot;>. Retrieved <span class=\&quot;nowrap\&quot; id=\&quot;mwuA\&quot;>23 August</span> 2018</span>.</cite><span about=\&quot;#mwt7\&quot; class=\&quot;Z3988\&quot; id=\&quot;mwuQ\&quot; title=\&quot;ctx_ver=Z39.88-2004&amp;amp;rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&amp;amp;rft.genre=article&amp;amp;rft.jtitle=National+Institutes+of+Health+%28NIH%29&amp;amp;rft.atitle=That+stinks%21+1+in+15+Americans+smell+odors+that+aren%E2%80%99t+there&amp;amp;rft.date=2018-08-16&amp;amp;rft_id=https%3A%2F%2Fwww.nih.gov%2Fnews-events%2Fnews-releases%2Fstinks-1-15-americans-smell-odors-arent-there&amp;amp;rfr_id=info%3Asid%2Fen.wikipedia.org%3APhantosmia\&quot; data-ve-ignore=\&quot;true\&quot;><span id=\&quot;mwug\&quot; style=\&quot;display:none;\&quot;><span id=\&quot;mwuw\&quot; typeof=\&quot;mw:Entity\&quot;>&amp;nbsp;</span></span></span>&quot;}}" class="mw-ref" about="#mwt94" id="cite_ref-That_stinks_1-0" rel="dc:references"><a href="./Phantosmia#cite_note-That_stinks-1" style="counter-reset: mw-Ref 1;"><span class="mw-reflink-text">[1]</span></a></sup> is smelling an odor that is not actually there. </span><span data-segmentid="5" class="cx-segment">It can occur in one nostril or both.<sup typeof="mw:Extension/ref" data-mw="{&quot;name&quot;:&quot;ref&quot;,&quot;attrs&quot;:{&quot;name&quot;:&quot;Hong&quot;}}" class="mw-ref" about="#mwt12" id="cite_ref-Hong_2-0" rel="dc:references"><a href="./Phantosmia#cite_note-Hong-2" style="counter-reset: mw-Ref 2;"><span class="mw-reflink-text">[2]</span></a></sup> </span><span data-segmentid="6" class="cx-segment">Unpleasant phantosmia, <b id="mwBw">cacosmia</b>, is more common and is often described as smelling something that is burned, foul, spoiled, or rotten.<sup typeof="mw:Extension/ref" data-mw="{&quot;name&quot;:&quot;ref&quot;,&quot;attrs&quot;:{&quot;name&quot;:&quot;Leopold2002a&quot;},&quot;body&quot;:{&quot;html&quot;:&quot;<cite about=\&quot;#mwt56\&quot; class=\&quot;citation\&quot; data-mw=\&quot;{&amp;quot;parts&amp;quot;:[{&amp;quot;template&amp;quot;:{&amp;quot;target&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;citation &amp;quot;,&amp;quot;href&amp;quot;:&amp;quot;./Template:Citation&amp;quot;},&amp;quot;params&amp;quot;:{&amp;quot;author&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;Leopold DA&amp;quot;},&amp;quot;year&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;2002a&amp;quot;},&amp;quot;title&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;Distortion of Olfactory Perception: Diagnosis and Treatment&amp;quot;},&amp;quot;journal&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;Chemical Senses&amp;quot;},&amp;quot;volume&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;27&amp;quot;},&amp;quot;issue&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;7&amp;quot;},&amp;quot;pages&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;611–615&amp;quot;},&amp;quot;doi&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;10.1093/chemse/27.7.611&amp;quot;},&amp;quot;pmid&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;12200340&amp;quot;}},&amp;quot;i&amp;quot;:0}}]}\&quot; id=\&quot;CITEREFLeopold_DA2002a\&quot; typeof=\&quot;mw:Transclusion\&quot; data-ve-no-generated-contents=\&quot;true\&quot;>Leopold DA (2002a), \&quot;Distortion of Olfactory Perception: Diagnosis and Treatment\&quot;, <i id=\&quot;mwzQ\&quot;>Chemical Senses</i>, <b id=\&quot;mwzg\&quot;>27</b> (7): 611–615, <a class=\&quot;cx-link\&quot; data-linkid=\&quot;299\&quot; href=\&quot;./Digital_object_identifier\&quot; id=\&quot;mwzw\&quot; rel=\&quot;mw:WikiLink\&quot; title=\&quot;Digital object identifier\&quot;>doi</a>:<a class=\&quot;external text\&quot; href=\&quot;//doi.org/10.1093/chemse/27.7.611\&quot; id=\&quot;mw0A\&quot; rel=\&quot;mw:ExtLink\&quot;>10.1093/chemse/27.7.611</a>, <a class=\&quot;cx-link\&quot; data-linkid=\&quot;300\&quot; href=\&quot;./PubMed_Identifier\&quot; id=\&quot;mw0Q\&quot; rel=\&quot;mw:WikiLink\&quot; title=\&quot;PubMed Identifier\&quot;>PMID</a><span id=\&quot;mw0g\&quot; typeof=\&quot;mw:Entity\&quot;>&amp;nbsp;</span><a class=\&quot;external text\&quot; href=\&quot;//www.ncbi.nlm.nih.gov/pubmed/12200340\&quot; id=\&quot;mw0w\&quot; rel=\&quot;mw:ExtLink\&quot;>12200340</a></cite><span about=\&quot;#mwt56\&quot; class=\&quot;Z3988\&quot; id=\&quot;mw1A\&quot; title=\&quot;ctx_ver=Z39.88-2004&amp;amp;rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&amp;amp;rft.genre=article&amp;amp;rft.jtitle=Chemical+Senses&amp;amp;rft.atitle=Distortion+of+Olfactory+Perception%3A+Diagnosis+and+Treatment&amp;amp;rft.volume=27&amp;amp;rft.issue=7&amp;amp;rft.pages=611-615&amp;amp;rft.date=2002&amp;amp;rft_id=info%3Adoi%2F10.1093%2Fchemse%2F27.7.611&amp;amp;rft_id=info%3Apmid%2F12200340&amp;amp;rft.au=Leopold+DA&amp;amp;rfr_id=info%3Asid%2Fen.wikipedia.org%3APhantosmia\&quot; data-ve-ignore=\&quot;true\&quot;><span id=\&quot;mw1Q\&quot; style=\&quot;display:none;\&quot;><span id=\&quot;mw1g\&quot; typeof=\&quot;mw:Entity\&quot;>&amp;nbsp;</span></span></span>&quot;}}" class="mw-ref" about="#mwt14" id="cite_ref-Leopold2002a_3-0" rel="dc:references"><a href="./Phantosmia#cite_note-Leopold2002a-3" style="counter-reset: mw-Ref 3;"><span class="mw-reflink-text">[3]</span></a></sup> </span><span data-segmentid="7" class="cx-segment">Experiencing occasional phantom smells is normal and usually goes away on its own in time.<sup typeof="mw:Extension/ref" data-mw="{&quot;name&quot;:&quot;ref&quot;,&quot;attrs&quot;:{&quot;name&quot;:&quot;NHS&quot;},&quot;body&quot;:{&quot;id&quot;:&quot;mw-reference-text-cite_note-NHS-4&quot;,&quot;html&quot;:&quot;<cite about=\&quot;#mwt9\&quot; class=\&quot;citation\&quot; data-mw=\&quot;{&amp;quot;parts&amp;quot;:[{&amp;quot;template&amp;quot;:{&amp;quot;target&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;citation&amp;quot;,&amp;quot;href&amp;quot;:&amp;quot;./Template:Citation&amp;quot;},&amp;quot;params&amp;quot;:{&amp;quot;title&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;Phantosmia (Smelling Odours That Aren't There)&amp;quot;},&amp;quot;author&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;HealthUnlocked&amp;quot;},&amp;quot;website&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;NHS Choices&amp;quot;},&amp;quot;year&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;2014&amp;quot;},&amp;quot;url&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;http://www.nhs.uk/conditions/phantosmia/Pages/Introduction.aspx&amp;quot;},&amp;quot;accessdate&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;6 August 2016&amp;quot;}},&amp;quot;i&amp;quot;:0}}]}\&quot; id=\&quot;CITEREFHealthUnlocked2014\&quot; typeof=\&quot;mw:Transclusion\&quot; data-ve-no-generated-contents=\&quot;true\&quot;>HealthUnlocked (2014), <a class=\&quot;external text\&quot; href=\&quot;http://www.nhs.uk/conditions/phantosmia/Pages/Introduction.aspx\&quot; id=\&quot;mw1w\&quot; rel=\&quot;mw:ExtLink\&quot;>\&quot;Phantosmia (Smelling Odours That Aren't There)\&quot;</a>, <i id=\&quot;mw2A\&quot;>NHS Choices</i><span class=\&quot;reference-accessdate\&quot; id=\&quot;mw2Q\&quot;>, retrieved <span class=\&quot;nowrap\&quot; id=\&quot;mw2g\&quot;>6 August</span> 2016</span></cite><span about=\&quot;#mwt9\&quot; class=\&quot;Z3988\&quot; id=\&quot;mw2w\&quot; title=\&quot;ctx_ver=Z39.88-2004&amp;amp;rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&amp;amp;rft.genre=article&amp;amp;rft.jtitle=NHS+Choices&amp;amp;rft.atitle=Phantosmia+%28Smelling+Odours+That+Aren%27t+There%29&amp;amp;rft.date=2014&amp;amp;rft.au=HealthUnlocked&amp;amp;rft_id=http%3A%2F%2Fwww.nhs.uk%2Fconditions%2Fphantosmia%2FPages%2FIntroduction.aspx&amp;amp;rfr_id=info%3Asid%2Fen.wikipedia.org%3APhantosmia\&quot; data-ve-ignore=\&quot;true\&quot;><span id=\&quot;mw3A\&quot; style=\&quot;display:none;\&quot;><span id=\&quot;mw3Q\&quot; typeof=\&quot;mw:Entity\&quot;>&amp;nbsp;</span></span></span>&quot;}}" class="mw-ref" about="#mwt65" id="cite_ref-NHS_4-0" rel="dc:references"><a href="./Phantosmia#cite_note-NHS-4" style="counter-reset: mw-Ref 4;"><span class="mw-reflink-text">[4]</span></a></sup> </span><span data-segmentid="8" class="cx-segment">When <a href="./Hallucination" rel="mw:WikiLink" data-linkid="9" class="cx-link" id="mwCA" title="Hallucination">hallucinations</a> of this type do not seem to go away or when they keep coming back, it can be very upsetting and can disrupt an individual's quality of life.<sup typeof="mw:Extension/ref" data-mw="{&quot;name&quot;:&quot;ref&quot;,&quot;attrs&quot;:{&quot;name&quot;:&quot;Hong&quot;},&quot;body&quot;:{&quot;id&quot;:&quot;mw-reference-text-cite_note-Hong-2&quot;,&quot;html&quot;:&quot;<cite about=\&quot;#mwt10\&quot; class=\&quot;citation\&quot; data-mw=\&quot;{&amp;quot;parts&amp;quot;:[{&amp;quot;template&amp;quot;:{&amp;quot;target&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;citation&amp;quot;,&amp;quot;href&amp;quot;:&amp;quot;./Template:Citation&amp;quot;},&amp;quot;params&amp;quot;:{&amp;quot;title&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;Distorted Olfactory Perception: A Systematic Review&amp;quot;},&amp;quot;vauthors&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;Hong SC, Holbrook EH, Leopold DA, Hummel T&amp;quot;},&amp;quot;year&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;2012&amp;quot;},&amp;quot;journal&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;Acta Oto-Laryngologica Supplementum&amp;quot;},&amp;quot;volume&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;132&amp;quot;},&amp;quot;issue&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;S1&amp;quot;},&amp;quot;page&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;S27&amp;quot;},&amp;quot;doi&amp;quot;:{&amp;quot;wt&amp;quot;:&amp;quot;10.3109/00016489.2012.659759&amp;quot;}},&amp;quot;i&amp;quot;:0}}]}\&quot; id=\&quot;CITEREFHongHolbrookLeopoldHummel2012\&quot; typeof=\&quot;mw:Transclusion\&quot; data-ve-no-generated-contents=\&quot;true\&quot;>Hong SC, Holbrook EH, Leopold DA, Hummel T (2012), \&quot;Distorted Olfactory Perception: A Systematic Review\&quot;, <i id=\&quot;mwxg\&quot;>Acta Oto-Laryngologica Supplementum</i>, <b id=\&quot;mwxw\&quot;>132</b> (S1): S27, <a class=\&quot;cx-link\&quot; data-linkid=\&quot;298\&quot; href=\&quot;./Digital_object_identifier\&quot; id=\&quot;mwyA\&quot; rel=\&quot;mw:WikiLink\&quot; title=\&quot;Digital object identifier\&quot;>doi</a>:<a class=\&quot;external text\&quot; href=\&quot;//doi.org/10.3109/00016489.2012.659759\&quot; id=\&quot;mwyQ\&quot; rel=\&quot;mw:ExtLink\&quot;>10.3109/00016489.2012.659759</a></cite><span about=\&quot;#mwt10\&quot; class=\&quot;Z3988\&quot; id=\&quot;mwyg\&quot; title=\&quot;ctx_ver=Z39.88-2004&amp;amp;rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&amp;amp;rft.genre=article&amp;amp;rft.jtitle=Acta+Oto-Laryngologica+Supplementum&amp;amp;rft.atitle=Distorted+Olfactory+Perception%3A+A+Systematic+Review&amp;amp;rft.volume=132&amp;amp;rft.issue=S1&amp;amp;rft.pages=S27&amp;amp;rft.date=2012&amp;amp;rft_id=info%3Adoi%2F10.3109%2F00016489.2012.659759&amp;amp;rft.aulast=Hong&amp;amp;rft.aufirst=SC&amp;amp;rft.au=Holbrook%2C+EH&amp;amp;rft.au=Leopold%2C+DA&amp;amp;rft.au=Hummel%2C+T&amp;amp;rfr_id=info%3Asid%2Fen.wikipedia.org%3APhantosmia\&quot; data-ve-ignore=\&quot;true\&quot;><span id=\&quot;mwyw\&quot; style=\&quot;display:none;\&quot;><span id=\&quot;mwzA\&quot; typeof=\&quot;mw:Entity\&quot;>&amp;nbsp;</span></span></span>&quot;}}" class="mw-ref" about="#mwt67" id="cite_ref-Hong_2-1" rel="dc:references"><a href="./Phantosmia#cite_note-Hong-2" style="counter-reset: mw-Ref 2;"><span class="mw-reflink-text">[2]</span></a></sup></span></p>
has 13K+ characters. If all of these are going through Yandex MT it will fail because it cross the allowed character count.

We should find a way to reduce the size of data we send to HTML MT systems. As of now only Yandex accept HTML content. For text based MT systems, this is not a problem at all.

Event Timeline

Changing priority high since this issue makes Yandex practically not usable with CX2

Change 461386 had a related patch set uploaded (by Santhosh; owner: Santhosh):
[mediawiki/services/cxserver@master] Compact the html before sending to HTML MT engines

https://gerrit.wikimedia.org/r/461386

Change 461386 merged by jenkins-bot:
[mediawiki/services/cxserver@master] Compact HTML that is sent to machine translation

https://gerrit.wikimedia.org/r/461386

Nikerabbit removed a project: Patch-For-Review.
Nikerabbit subscribed.

QA note: this is a patch in cxserver which requires a separate deployment before testing in production.

Stashbot subscribed.

Mentioned in SAL (#wikimedia-operations) [2018-10-17T08:26:59Z] <kartik@deploy1001> Started deploy [cxserver/deploy@b30a323]: Update cxserver to 29e01e4 (T206305, T204668)

Mentioned in SAL (#wikimedia-operations) [2018-10-17T08:30:52Z] <kartik@deploy1001> Finished deploy [cxserver/deploy@b30a323]: Update cxserver to 29e01e4 (T206305, T204668) (duration: 03m 54s)

QA note: this is a patch in cxserver which requires a separate deployment before testing in production.

Deployed in Production.

Etonkovidova subscribed.

Checked the same paragraph in produciton (wmf.26) - the cound is reduced to 10,860.