Page MenuHomePhabricator

Different translation when same text is provided as html or plain text
Open, MediumPublic

Description

Translating the first line of text from the Moon article on Simple English Wikipedia is the following:

Plain-textHTML
The Moon is Earth's only natural satellite. We usually see it in the night sky and also during the day. Some other planets also have moons or natural satellites.<p>The <b>Moon</b> is <a href="/wiki/Earth" title="Earth">Earth</a>'s only <a href="/wiki/Natural_satellite" title="Natural satellite">natural satellite</a>. We usually see it in the night sky and also during the day. Some other planets also have moons or natural satellites.</p>

When translating the above contents with the MinT test instance for text and for HTML the results are different:

Plain-textHTML
translate.wmcloud.org_(Wiki Tablet).png (1,024×768 px, 71 KB)
translate.wmcloud.org_html(Wiki Tablet).png (1,024×768 px, 96 KB)

The resulting translations:

Plain-textHTML
चंद्रमा पृथ्वी का एकमात्र प्राकृतिक उपग्रह है। हम इसे आमतौर पर रात के आसमान में और दिन में भी देखते हैं। कुछ अन्य ग्रहों में भी चंद्रमा या प्राकृतिक उपग्रह होते हैं।द.चाँद।है।धरतीकेवलप्राकृतिक उपग्रह.. हम इसे आमतौर पर रात के आसमान में और दिन में भी देखते हैं। कुछ अन्य ग्रहों में भी चंद्रमा या प्राकृतिक उपग्रह होते हैं।

There are two kinds of observed issues:

  • Lack of spacing around some elements with an html tags such as bold text or links. For example the first link of the HTML version ("धरती") has a long sequence of characters until the next space.
  • The resulting text is different. For example the HTML translation starts with "द.चाँद।" while the plain-text version starts with "चंद्रमा"

The expected reult is for both translations to result in the same text (with html providing additional styling on top).

Event Timeline

Pginer-WMF triaged this task as Medium priority.Oct 5 2023, 10:26 AM

Trying to reproduce the issue:

image.png (1,233×428 px, 59 KB)

Trying the commandline interface to MinT:

 echo "<p>The <b>Moon</b> is <a href="/wiki/Earth" title="Earth">Earth</a>'s only <a href="/wiki/Natural_satellite" title="Natural satellite">natural satellite</a>. We usually see it in the night sky and also during the day. Some other planets also have moons or natural satellites.</p>" | python -m translator -s en -t  hi -f html                   
{"@timestamp":"2023-10-05T10:34:24.309Z","log.level":"info","message":"EnIndicTransModel initialized.","ecs":{"version":"1.6.0"},"log":{"logger":"root","origin":{"file":{"line":52,"name":"base.py"},"function":"init"},"original":"EnIndicTransModel initialized."},"process":{"name":"MainProcess","pid":168305,"thread":{"id":140007283159104,"name":"MainThread"}}}
{"@timestamp":"2023-10-05T10:34:24.309Z","log.level":"info","message":"inter_threads: 8, intra_threads: 0 ","ecs":{"version":"1.6.0"},"log":{"logger":"root","origin":{"file":{"line":53,"name":"base.py"},"function":"init"},"original":"inter_threads: 8, intra_threads: 0 "},"process":{"name":"MainProcess","pid":168305,"thread":{"id":140007283159104,"name":"MainThread"}}}
================================================================================
<p>चंद्रमा पृथ्वी का एकमात्र <a href="/wiki/Natural_satellite" satellite="" title="Natural">प्राकृतिक उपग्रह</a> है। हम इसे आमतौर पर रात के आसमान में और दिन में भी देखते हैं। कुछ अन्य ग्रहों में भी चंद्रमा या प्राकृतिक उपग्रह हैं।</p> 

Since the above methods shows no issues, we need to see what exactly got pasted in to the HTML field. It clearly has something more, as we see white background. I went to https://simple.wikipedia.org/wiki/Moon and copied the first paragraph and pasted to the html source. I don't see any white background in the pasted content.

@Pginer-WMF, can you inspect the source html in dev tools and give the actual input html?

@Pginer-WMF, can you inspect the source html in dev tools and give the actual input html?

It seems that the pated HTML from the test instance i not exactly as the source HTML from the article (maybe browser is trying to keep in-line styles):

<p id="mwCw"><span style="color: rgb(32, 33, 34); font-family: sans-serif; font-size: 14px; background-color: rgb(255, 255, 255);">The&nbsp;</span><b style="color: rgb(32, 33, 34); font-family: sans-serif; font-size: 14px; background-color: rgb(255, 255, 255);">Moon</b><span style="color: rgb(32, 33, 34); font-family: sans-serif; font-size: 14px; background-color: rgb(255, 255, 255);">&nbsp;is&nbsp;</span><a href="https://simple.wikipedia.org/wiki/Earth" title="Earth" style="text-decoration-line: none; color: rgb(51, 102, 204); background: none rgb(255, 255, 255); overflow-wrap: break-word; font-family: sans-serif; font-size: 14px;">Earth</a><span style="color: rgb(32, 33, 34); font-family: sans-serif; font-size: 14px; background-color: rgb(255, 255, 255);">'s only&nbsp;</span><a href="https://simple.wikipedia.org/wiki/Natural_satellite" title="Natural satellite" style="text-decoration-line: none; color: rgb(51, 102, 204); background: none rgb(255, 255, 255); overflow-wrap: break-word; font-family: sans-serif; font-size: 14px;">natural satellite</a><span style="color: rgb(32, 33, 34); font-family: sans-serif; font-size: 14px; background-color: rgb(255, 255, 255);">. We usually see it in the night sky and also during the day. Some other planets also have moons or natural satellites.</span></p>

Not only styles, but spaces are replaced by &nbsp;.

The&nbsp;Moon&nbsp;&nbsp;is&nbsp;Earth's only&nbsp;natural satellite-> That explains the Hindi text without spaces.