physik wrote:

Sure. I'm just developing those features.
What does ത്ര mean?
If it's a word with a conventional meaning instead of a variable it should be written as \text{ത്ര}^3
This will help blind people in the future to understand the content.

For example the first equation on
https://en.wikipedia.org/w/index.php?title=Body_mass_index
is written as
<math>\mathrm{BMI}</math>&nbsp;

This prevents wrong spacing between the letters BMI and mass and makes the equation accessible for disabled people and search engines.

ത്ര has no meaning. It is not a word. It is just a letter (ligature).

physik wrote:

Unfortunately the TeX system has not very good support of non ASCII characters.
It's the same problem with the German 'umlauts' öäü. In all math environments that I tried I had to use \text{ü} instead of ü.
So I think we have to cope with that as well. Maybe we can come up with an interface for easy editing with visual editor one day.
MathJaX can convert the samples if \text is used to MathML but not to SVG.
see
http://math-test2.instance-proxy.wmflabs.org/wiki/30463

This requires Math2.0 which is ready for code review at
https://gerrit.wikimedia.org/r/#/c/85801/

physik wrote:

*** Bug 30463 has been marked as a duplicate of this bug. ***

MathJaX can convert the samples if \text is used to MathML but not to SVG.
see
http://math-test2.instance-proxy.wmflabs.org/wiki/30463

That shouldn't happen. MathJax accepts unicode input and should fall back to system fonts for characters not covered by its fonts. I suspect it's a problem with svgtex / mathoid or more precisely with phantomjs.

Could you file a bug report at MathJax with some background information?

physik wrote:

Hi Peter,
what do you mean exactly with at MathJaX?

(In reply to comment #35)

Hi Peter,
what do you mean exactly with at MathJaX?

Hi Moritz,

With "at MathJax" I meant our bug tracker at https://github.com/mathjax/mathjax/issues.

But I just took another look and this might be yet another problem with embedding the SVG. That is, the "complex Malayalam equation" at http://math-test2.instance-proxy.wmflabs.org/wiki/30463 looks very different if viewed independently at http://math-test2.instance-proxy.wmflabs.org/w/index.php?title=Special:MathShowImage&hash=b3793eaa2110f756756386bb17b17d04.

physik wrote:

*** Bug 2458 has been marked as a duplicate of this bug. ***

physik wrote:

Is anyone around here that is familiar with OCaml?
In order to make process on all the Math related bugs this change
https://gerrit.wikimedia.org/r/#/c/90748
need so get a review.
I verified the correctness of the code, and wrote a small guide how you can verify it by yourself at
http://www.formulasearchengine.com/Verify%20texvc%20light
.
If you don't like to do a review I'd be interested in the reasons for that.
Even a very basic feedback concerning the commit helps. i.e. Do you understand what the change is about, or should the commit message be changed.

physik wrote:

I unassigned me, since I don't want to review my own code and I can not do more about it at the moment.

The patches seem to have been merged or abandoned, so setting back to NEW.

physik wrote:

See
https://www.mediawiki.org/wiki/Extension:Math/bug/48032
in MathML rendering mode.
I would say that's as good as it gets.
If that's not sufficent please seperate individual bugs for the chars that don't work so that we can fix one after the oter. We would also need a reference implementation (for example a LaTeX document) that demonstrates how the result should look like.

physik wrote:

48032 in MathML

I can't see what's wrong here.

Attached:

Created attachment 16875
PNG rendering gives Lexing Error

Attached:

Created attachment 16876
No proper rendering in MathML

Attached:

Created attachment 16877
No proper rendering in clientside MathJax

Bug still persists and no proper rendering available in any mode. Test page - [[:mlഉപയോക്താവ്:Praveenp/പരിശോധന]].

There are various books with mathematical equations still not possible to include (eg: https://ml.wikisource.org/wiki/%E0%B4%A4%E0%B4%BE%E0%B5%BE:Yukthibhasa.djvu/227)

Attached:

physik wrote:

I still do not understand the problem
I updated the demo at https://www.mediawiki.org/wiki/Extension:Math/bug/48032

I'd be happy if someone could explain it to me.

Here's my explanation:

• PNG output is totally broken and needs to be fixed to stop spewing red error messages. I think the problem here is quite obvious.
• MathML output renders very poorly. Characters are overlapping and/or severely truncated (e.g. only half the character is visible).
• MathJax seems acceptable, even though the characters are very small. Having said that, I do not read Malayalam so cannot confirm whether the rendering is indeed perfectly accurate.

Hopefully Praveen can also explain some more.

@Physikerwelt could PNG be generated from the SVG? MathJax-node has hooks for Apache Batik to do this. MathJax is Unicode friendly although it will often have to rely on system fonts to provide the glyphs.

physik wrote:

(In reply to This, that and the other (TTO) from comment #47)

Here's my explanation:

• PNG output is totally broken and needs to be fixed to stop spewing red

error messages. I think the problem here is quite obvious.

OK. This will be solved by Bug 72240.

• MathML output renders very poorly. Characters are overlapping and/or

severely truncated (e.g. only half the character is visible).

Here, we need to clarify if the generated MathML is correct.

• MathJax seems acceptable, even though the characters are very small.

Having said that, I do not read Malayalam so cannot confirm whether the
rendering is indeed perfectly accurate.

Hopefully Praveen can also explain some more.

The problem I see with this bugs and many other bugs, is that there is a gap between the bug report created by the users and the information needed by the programmers. For example this bug describes a whole complex of problems.
So there is no well defined problem that can be transformed into a testcase and then be resolved.

To my understanding a programmer compatible bug report would optimally be the following:

1. What is the scenario:

a) TeX input code
b) Environment variables
c) Which rendering mode is used?
d) Which browser is used?

2. Is the input valid?
3. If it's valid: How is it supposed to look?

Is there a TeX\LaTeX document that demonstrates the rendering?
What should be the MathML output.

(In reply to physikerwelt from comment #49)

The problem I see with this bugs and many other bugs, is that there is a gap
between the bug report created by the users and the information needed by
the programmers.

Sometimes that takes some back-and-forth in order to figure it out. I agree that "No proper rendering" should be made clearer (e.g. "bottom of character is truncated", "characters are not joined properly", or "wrong font is used", whatever the case may be).

Also, it helps to put the full markup (<math> through </math>) here in the bug report. If it's only in the screenshot, someone has to retype it or figure out where to copy it from, which is hard if it's in a language they don't know).

So there is no well defined problem that can be transformed into a testcase
and then be resolved.

The PNG one seems pretty clear. I've made a sub-task for this, bug 73285.

Created attachment 17099
Different renderings of Malayalam equations

Here's another try. Praveen, can you confirm that the renderings at the left of this image are indeed correct?

First example is: <math>\cfrac{\text{ച}}{\angle 4\text{ത്ര}^3}</math>
MS Word code (linear form) is: ച/(∠4〖ത്ര〗^3 )

Second example is: <math>ല \left ( \frac{വ്വ(ക്ലു \pm കു_ശ)}{ടി} \right )</math>
MS Word code (linear form) is: ല(വ്വ(ക്ലു±〖കു〗_ശ )/ടി)

Per comment 7, I tried using sharelatex to generate an example, but its pdfLaTeX renderer refuses to accept Malayalam at all, and the LuaLaTex and XeLaTeX renderers do it wrongly.

Attached:

physik wrote:

If you manage to make Mathjax render your input correctly, it will be easy to port that to the Math extension.

This will generally work (so 69702 should resolve this).

But I think for this specific case there's a bug in MathJax related to combining characters. I've filed https://github.com/mathjax/MathJax/issues/952

(In reply to physikerwelt from comment #52)

If you manage to make Mathjax render your input correctly, it will be easy
to port that to the Math extension.

Equation 1 is working properly in MathJax but not in the SVG fallback, so that can be a starting point.

comment #51 is correct

Just in case there is renewed interest: as mentioned in https://github.com/mathjax/MathJax/issues/952 and https://github.com/mathjax/MathJax/issues/474#issuecomment-38324717, we would be very happy to get some advice from language experts.

Since this issue saw some mild activity recently: it seems to me this should have been mostly resolved now that mathoid/mathajx-node runs on the backend.

Non-combining Unicode points should work everywhere and combining ones should work in \text{} (which is the right way to get something with a chance of backward compatibility to real TeX). Unless texvc filters them out, I suppose.

Does the problem persist?

Thank you for solving the issue by mild activity. BTW, URL in bug descriptions itself still render no improvement.

Thank you for solving the issue by mild activity.

Apologies for my previous message not being precise enough. By "mild activity" I only meant that this issue was added to certain projects, indicating that some people were becoming interested that previously weren't.

BTW, URL in bug descriptions itself still render no improvement.

Right. All I meant to raise is that the backend can now handle Unicode better (thanks to using MathJax) and so the issue might be (at least partially) solvable now.

@Physikerwelt: is perhaps texvc causing the parsing errors now?

@Pkra yes probably texvc (i.e texvcjs) is causing the parsing errors. Once https://phabricator.wikimedia.org/T74240 is closed that can be fixed.

@Arthur2e5 By reading the entire task I can't see any zh-related contents, and I can't find anything on zh.* that about T50032 or the former bugzilla:48032, so I'd love to know the reason of your action, if you or someone mentioned this task on non-MediaWiki platforms such as Telegram (which I rarely use... :p), then paste those discussions would help us.

@Liuxinyu970226 people regularly talk about not being able to use Chinese for the <chem> tag (and math too). You know things are wrong when you can't write "加热", "电解" in chemical equations.

zh Test page https://en.wikipedia.org/wiki/User:TheDJ/math

small notes:

1. Seems SVG rendered inside an <img> doesn't do proper font loading for me on Safari. When loading the SVG directly in the browser, the 加热 characters appear, but the characters are missing for me on the wiki page. This seems a recent Safari regression, I'm pretty sure this used to work at some point.
2. There seems to be some overlap between the two characters for me.. Possibly because the SVG rendering assumes monospacing, whereas my mac doens't have a monospace variant of these fonts ???

There seems to be some overlap between the two characters for me.. Possibly because the SVG rendering assumes monospacing, whereas my mac doens't have a monospace variant of these fonts ???

This is likely due to an issue in mathjax-node (which creates the SVGs). For Unicode points that the configured font does not have a glyph for, the SVG output will simply contain text elements with the text (instead of SVG paths).

In such a situation, mathjax-node has to guess the width of that text (which it cannot know ahead of time as it depends on system fonts). This estimation is not ideal in general but in particular does not take full-width forms into account.

This will be improved in the next release, cf. https://github.com/mathjax/MathJax-node/pull/358.

In T50032#3749405, @Pkra wrote:

There seems to be some overlap between the two characters for me.. Possibly because the SVG rendering assumes monospacing, whereas my mac doens't have a monospace variant of these fonts ???

This is likely due to an issue in mathjax-node (which creates the SVGs). For Unicode points that the configured font does not have a glyph for, the SVG output will simply contain text elements with the text (instead of SVG paths).

In such a situation, mathjax-node has to guess the width of that text (which it cannot know ahead of time as it depends on system fonts). This estimation is not ideal in general but in particular does not take full-width forms into account.

This will be improved in the next release, cf. https://github.com/mathjax/MathJax-node/pull/358.

IMHO, Shouldn't we handle those problems in another task? Because I still think that this task should be focused on, and only on Indic languages.

I still think that this task should be focused on, and only on Indic languages.

Sounds good.

As I wrote above, the main technical limitation was resolved: with the switch to mathoid and mathjax-node, you can now use Indic languages in principle.

What needs to be worked out are the details.

1. the input problem and backward compatibility with TeX engines.

Most TeX engines would not render, e.g., \frac{ച}{\angle 4ത്ര^3} well as they do not support such Unicode in math mode. Similarly, texvc throws an error while sanitizing such input.

Still, some TeX engines support such Unicode input in text mode, e.g., \frac{\text{ച}}{\angle 4\text{ത്ര}^3}.

So the question is: is this sufficient? For a long discussion with WMF devs and others, see https://github.com/mathjax/MathJax/issues/474

2. rendering

The next problem is how to render such input if you allow it. A TeX parser splits up strings in math mode by codepoint and treats them as separate characters (unless grouping is specified). This means combining characters would be split up, e.g., $4ത്ര$.

The resulting rendering should keep each character separate (garbage in, garbage out). This means combining characters will no longer combine (in the SVG output that mathoid uses).

3. mathjax-node rendering with \text{}

Finally, while \frac{\text{ച}}{\angle 4\text{ത്ര}^3} is supported by texvc and mathoid now , there are still lingering rendering issues specific to mathjax-node and the SVG output.

Since MathJax lacks a font with glyphs for these, it will fallback to system fonts by using SVG text elements. Unfortunately, these get split up into several pieces, leading to the same problem as above.

This is not as straight forward to fix as one might think. MathJax has to estimate the width of such a text element in its rendering (e.g., below a fraction line); combining characters naturally require less width than a string of the same length without combining characters.

Still, this is something that could probably be fixed relatively easily, especially in mathoid / mathjax-node.

To recap:

• you can use such Unicode characters when wrapping in \text{} but combining characters will not render well
  • this could be roughly fixed
• the community needs to decide whether it wants to stay backward compatible with TeX engines
  • if it wants easier input, then more implementation work is needed

Although, I am not "the community", I think this question is easy to answer:

• Unicode characters outside \text{} are not needed, not in regular LaTeX and not in Wikipedia

What is really important though is

• the "für alle" problem, i.e. ü being rendered (currently not always the case) and having proper size etc.
• characters that need more width e.g. ю not overlapping with others
• combining charakters like ത്ര

there are two issues concerning Unicode characters outside of \text:

• the command \AA used to produce the unit symbol Å for Ångström needs to work outside \text (it does that in LaTeX and also gets past texvc, but the rendering has the same issues as the ü in "für alle")
• the error message should be improved. I witnessed twice that people copied LaTeX source-code to some Microsoft Word document, which automatically replaces certain - with − and then they are puzzled by the "syntax-error" when they copy it back.

I hope that https://meta.wikimedia.org/wiki/2017_Community_Wishlist_Survey/Reading#Functional_and_beautiful_math_for_everyone will get some votes and then the mtextFontInherit option can take care of it. Still it would be nice if it gets fixed before all the öäü etc. have been replaced with some workarounds like https://de.wikipedia.org/w/index.php?diff=170484341&oldid=165943062&title=Fall_mit_Luftwiderstand again.

Some comments purely from a MathJax point of view.

Unicode characters outside \text{} are not needed, not in regular LaTeX and not in Wikipedia

Would it be possible to gather community input on that?

the "für alle" problem, i.e. ü being rendered (currently not always the case) and having proper size etc.
characters that need more width e.g. ю not overlapping with others

Would you have some examples for the problems you're seeing?

In general, I suspect it would be useful to consider switching to a different font. Currenetly (a derivative of) Computer Modern is used which doesn't have very good Unicode coverage.

the command \AA used to produce the unit symbol Å for Ångström needs to work outside \text (it does that in LaTeX and also gets past texvc, but the rendering has the same issues as the ü in "für alle")

Changing to a font that has a glyph would help or setting Mathoid up to replace it with a hack in the SVG rendering.

It would also help to improve the SVG configuration to get better visual rendering (comparable to MathJax's HTML output).

Would it be possible to gather community input on that?

https://ru.wikipedia.org/wiki/%D0%9E%D0%B1%D1%81%D1%83%D0%B6%D0%B4%D0%B5%D0%BD%D0%B8%D0%B5_%D0%92%D0%B8%D0%BA%D0%B8%D0%BF%D0%B5%D0%B4%D0%B8%D0%B8:%D0%A4%D0%BE%D1%80%D0%BC%D1%83%D0%BB%D1%8B#Question%20on%20unicode%20characters

Would you have some examples for the problems you're seeing?

Characters being too large (and getting cut-off), too small (like little dots) or not appearing at all. That is on my laptop using firefox, konqueror and midori respectively, but those issues I also get in more common browsers and devices.

Thanks for the link.

Characters being too large (and getting cut-off), too small (like little dots) or not appearing at all. That is on my laptop using firefox, konqueror and midori respectively, but those issues I also get in more common browsers and devices.

Could you collect some links to examples when you come across them? That would be very helpful.

Correcting my earlier statement on SVG output in mathjax-node.

Unfortunately, these get split up into several pieces, leading to the same problem as above.

We're avoiding this already; here's an example of what mathjax-node would produce.

<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="2.253ex" height="7.843ex" style="vertical-align: -3.338ex;" viewBox="0 -1939.5 970 3376.7" role="img" focusable="false" xmlns="http://www.w3.org/2000/svg" aria-labelledby="MathJax-SVG-1-Title">
<title id="MathJax-SVG-1-Title">\frac{\text{ക}}{\text{ച}}</title>
<defs aria-hidden="true"></defs>
<g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)" aria-hidden="true">
<g transform="translate(120,0)">
<rect stroke="none" width="729" height="60" x="0" y="220"></rect>
<g transform="translate(60,949)">
<text font-family="monospace" stroke="none" transform="scale(71.759) matrix(1 0 0 -1 0 0)">ക</text>
</g>
<g transform="translate(60,-881)">
<text font-family="monospace" stroke="none" transform="scale(71.759) matrix(1 0 0 -1 0 0)">ച</text>
</g>
</g>
</g>
</svg>

The length calculation for the text elements could be improved (as could the font-family settings).

Could you collect some links to examples when you come across them? That would be very helpful.

I don't understand. How does that help?

Sample of articles currenly affected by "für alle" problem:
https://de.wikipedia.org/w/index.php?search=insource%3A%2F%5C%5Ctext%5C%7B.%3Ff%C3%BCr%2F&title=Spezial:Suche&profile=default&fulltext=1&searchToken=3gmxbkhes8n0mbri6jkd24rep

Sample of articles using text in formulas:
https://de.wikipedia.org/w/index.php?search=insource%3A%2F%5C%5Ctext%5C%7B%2F&title=Spezial:Suche&profile=default&fulltext=1&searchToken=9ghssxjrta2o9d5cvnvpk4udj
where at least non-latin languages currently rely on workarounds.

Please add T36193 to related tasks

I didn't remember ever filing a Safari ticket about the problem with vanishing characters in the SVG images. I have therefor filed:
https://bugs.webkit.org/show_bug.cgi?id=191834

@TheDJ Awesome! Amazing that it's still there. I noticed that on a recent WebKitGTK+ 2.22.2, the characters are there but tiny (see attached screenshot).

I have therefor filed: https://bugs.webkit.org/show_bug.cgi?id=191834

This has been marked fixed.

Yay! Thank you, @TheDJ!

Cf. T130967 for a separate bug of this issue.

As recent discussions are generally Indic languages related only, I've removed this tag, @Arthur2e5 if you think this tag should be re-added, feel free to expand the task description and provide more zhwiki related informations (but as another better suggestion: it's better that you file another likely task for zhwiki).

FWIW, it seems that texvcjs, not MathJax, is the one blocking the characters from mathoid. The problem is in turn thrown from pegjs.

Honestly I think the issue is not about whether Unicode in math environment is needed, but more about whether texvcjs is doing what it's supposed to do (validating tex input for mathoid). Now it's serving as a roadblock (via texvcinfo) to something that MathJax can do, and that's obviously not cool. At minimum the peg rule should be relaxed.

Heck, I might go so far to say that pre-checking the input with texvcinfo is a waste of time. MathJax is perfectly competent at what it does and complaining when it should, and so long as we don't want the texvcinfo type of output there's no point in consulting it. The texvcinfo output is extremely short and confusing anyways -- at least when compared to MathJax's image error output.

RunKit test code:

var texvcjs = require("mathoid-texvcjs")
var result = texvcjs.check('\\sin(甲)+{}{}\\cos(乙)^2 ');
console.log(result);

This is not a technical problem it is a design question. The question is does the mathematical community in Wikimedia projects want to stick to the limitations of tex / pdflatex or do we WANT to CHANGE this and allow other characters in math. If so how far do we WANT to go.
If I run the example above aka

\documentclass[]{minimal}
\begin{document}
$\sin(甲)$
\end{document}

I see
Package inputenc Error: Unicode character 甲 (U+7532)(inputenc) not set up for use with LaTeX. $\sin(甲

The process to decide, should be to ask the Community via https://meta.wikimedia.org/wiki/Wikimedia_Community_User_Group_Math
If there is a decision, changing texvcjs is not a problem.

In T50032#6979865, @Physikerwelt wrote:

Package inputenc Error: Unicode character 甲 (U+7532)(inputenc) not set up for use with LaTeX. $\sin(甲

You are close to seeing the problem. LaTeX2e in 2020 has already started taking in UTF-8 by default, and what you are seeing is not a language issue but a font issue: latin script works. The following compiles on Overleaf despite LaTeX (fmtversion 2020-02-02; pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020) (preloaded format=pdflatex 2020.9.10)) warning me to use mathaccents for the umlauts:

\documentclass[]{article}
\usepackage{amsmath}
\begin{document}
This is \LaTeX\ \fmtversion\ doing some Unicode.
$$ö = \sin(Ö) = [ø] \not= \operatorname{majestik}(møøse)$$
Mynd you, møøse bites Kan be pretti nasti…
\end{document}

Now go to RunKit and guess what texvcjs says about it.

In T50032#7013971, @Arthur2e5 wrote:

Now go to RunKit and guess what texvcjs says about it.

You can also read it in the source code.
https://github.com/wikimedia/mediawiki-services-texvcjs/blob/master/lib/parser.pegjs#L288
It is not a surprise. The goal of texvc has been to reproduce the behavior of tex when it was created in 2003... So this is not a bug. If you propose to change this behavior you can make a proposal and we can change it in the source code unless there are objections from others, i.e., the Group. At work, we changed recently to XeLaTeX. On overleaf your example compiles https://www.overleaf.com/8447739181zcvdvhpfygry but all the special chars in math mode are gone. Thus one can argue in one or the other direction. In the end, the question is what does Wikipedia / the Wikimedia movement wants? After this is specified the implementation should not be very hard. Maybe there will be some issues with fallback to system fonts in SVG images, however, this is not a problem with MathML rendering.

If you propose to change this behavior you can make a proposal and we can change it in the source code unless there are objections from others

What would a "proposal" entail? What parts does it need to include, and where should it be posted? Would it be sufficient to open a ticket simply on phab, one that (a) rephrases some of the arguments for from above (b) specifically mentions the reason, that is texvcjs's refusal, suffice?

what does Wikipedia / the Wikimedia movement wants?

I only know what I want. I probably can infer the people at the Chinese-Sites group wants. I might also infer that the the person tagging the sites believes that people involved in Tamil-Sites, Malayalam-Sites, Hindi-Sites care about this. Would this be enough possible support?