Page MenuHomePhabricator

Math: Support phantom tags
Open, Needs TriagePublic5 Estimated Story Points

Description

The Math-extension sadly doesn't support the phantom tag and shows an parser error:

Failed to parse (unknown function "\phantom"): {\displaystyle \frac{x+y+z}{x \phantom{+}\phantom{y}+z}}

Example code for mediawiki / TeX:

<math>\frac{x+y+z}{x \phantom{+}\phantom{y}+z}</math>

MathML code:

<math xmlns = "http://www.w3.org/1998/Math/MathML">
   <mfrac>
      <mrow>
         <mi> x </mi>
         <mo> + </mo>
         <mi> y </mi>
         <mo> + </mo>
         <mi> z </mi>
      </mrow>

      <mrow>
         <mi> x </mi>
         <mphantom>
            <mo> + </mo>
         </mphantom>

         <mphantom>
            <mi> y </mi>
         </mphantom>
         <mo> + </mo>
         <mi> z </mi>
      </mrow>
   </mfrac>
</math>

and here an example edit:
https://en.wikipedia.org/w/index.php?title=Wikipedia:Sandbox&oldid=974880776

And whoever it wants can use the following code for LibreOffice or OpenOffice:

{x + y + z} over {x {phantom +} {phantom y} + z}

Hence: LibreOffice and OpenOffice simply ignores the tag in the mediawiki-export extension and thus showing the complete x + y + z twwo times (above the bracket and under it).

For the mediawiki exporter extension code:
https://opengrok.libreoffice.org/xref/core/external/xsltml/xsltml_2.1.2.patch?r=2b383d19#1118

Progress:

Event Timeline

Change 790640 had a related patch set uploaded (by Dom Walden; author: Dom Walden):

[mediawiki/services/texvcjs@master] Add support for phantom commands.

https://gerrit.wikimedia.org/r/790640

I have made an attempt at a patch above. @Physikerwelt let me know what you think.

I hope I have put it in the right place. The \[vh]phantom commands seem to take only one argument so I assume they should be handled the same as commands like \textbf.

I am not sure how much unit testing is necessary. I have also tested this on my local wiki comparing the output between Mathoid and LaTeX. I haven't done much in the way of regression testing.

I appreciate this is in the "Needs community consensus" column. But, what counts as community consensus? 45 people voted for it in the last wish list https://meta.wikimedia.org/wiki/Community_Wishlist_Survey_2022/Editing/Missing_LaTeX_capabilities_for_math_rendering.

Moreover, it seems like a low risk and uncontroversial change. We are only exposing more of the features of MathJax. We are not interfering with existing functionality.

The only risk I can see is if people start using these new commands and then MathJax stops supporting them. However, they are standard TeX commands so it seems unlikely this would happen.

Physikerwelt added a subscriber: WDoranWMF.

I have made an attempt at a patch above. @Physikerwelt let me know what you think.

I will have a look. Have you spoken to @WDoranWMF about math-related changes?

Also, there are some pending changes in the queue.

https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/787831

for the mathoid update

and a conflicting change

https://gerrit.wikimedia.org/r/c/mediawiki/services/texvcjs/+/785246

I hope I have put it in the right place. The \[vh]phantom commands seem to take only one argument so I assume they should be handled the same as commands like \textbf.

I guess so.

I am not sure how much unit testing is necessary. I have also tested this on my local wiki comparing the output between Mathoid and LaTeX. I haven't done much in the way of regression testing.

I think we should have a look at what the rendering with mathoid will look like and test there. Moreover, I guess we should create a follow up ticket to notify people about the change so that the respective documentation can be updated.

I appreciate this is in the "Needs community consensus" column. But, what counts as community consensus? 45 people voted for it in the last wish list https://meta.wikimedia.org/wiki/Community_Wishlist_Survey_2022/Editing/Missing_LaTeX_capabilities_for_math_rendering.

Yes. This is probably ok. I will run a quick query on the mailing list just to give people a way to stop this.

Moreover, it seems like a low risk and uncontroversial change. We are only exposing more of the features of MathJax. We are not interfering with existing functionality.

I think so too.

The only risk I can see is if people start using these new commands and then MathJax stops supporting them. However, they are standard TeX commands so it seems unlikely this would happen.

We will probably move away from MathJax to a custom-made rendering engine, so that should not be a problem.

I have made an attempt at a patch above. @Physikerwelt let me know what you think.

I will have a look. Have you spoken to @WDoranWMF about math-related changes?

Not yet. I will do though, thanks.

Also, there are some pending changes in the queue.

...

https://gerrit.wikimedia.org/r/c/mediawiki/services/texvcjs/+/785246

Ah, I didn't notice this when I pushed my patch. Should I wait for this to be merged and update my patch?

I am not sure how much unit testing is necessary. I have also tested this on my local wiki comparing the output between Mathoid and LaTeX. I haven't done much in the way of regression testing.

I think we should have a look at what the rendering with mathoid will look like and test there. Moreover, I guess we should create a follow up ticket to notify people about the change so that the respective documentation can be updated.

It looks good to me locally. For example, the code <math>-e\sqrt{\vphantom{p'}p},\; -e'\sqrt{p'},\; \ldots</math> renders like:

vphantom_example.png (47×175 px, 1014 B)

The same code without \vphantom (i.e. <math>-e\sqrt{p},\; -e'\sqrt{p'},\; \ldots</math>) renders like:

example_without_vphantom.png (43×148 px, 935 B)

All the other examples I tried compared favourably to the same code rendered in LaTeX.

Good point about the documentation.

The only risk I can see is if people start using these new commands and then MathJax stops supporting them. However, they are standard TeX commands so it seems unlikely this would happen.

We will probably move away from MathJax to a custom-made rendering engine, so that should not be a problem.

Interesting, I didn't know this.

Physikerwelt set the point value for this task to 5.

I did manual testing in mathoid (with no_check setting to true). I can confirm that there are cases where phantom rendering works;-)

I changed the test description to describe the steps required to move this into production. @dom_walden would you help me with that, by either making code reviews or reviewing my patches?

Are you familiar with the Mathoid deployment (https://wikitech.wikimedia.org/wiki/Mathoid), otherwise I will search for someone?

I did manual testing in mathoid (with no_check setting to true). I can confirm that there are cases where phantom rendering works;-)

Thanks!

I changed the test description to describe the steps required to move this into production. @dom_walden would you help me with that, by either making code reviews or reviewing my patches?

I am happy to help as much as I can.

I didn't realise there were so many steps. I had plans to also implement more of the functions mentioned in the wish. Should I do that first before we do the production release (so we only have to do one release)?

Are you familiar with the Mathoid deployment (https://wikitech.wikimedia.org/wiki/Mathoid), otherwise I will search for someone?

I am afraid not.

I changed the test description to describe the steps required to move this into production. @dom_walden would you help me with that, by either making code reviews or reviewing my patches?

I am happy to help as much as I can.

Amazing. Thank you.

I didn't realise there were so many steps. I had plans to also implement more of the functions mentioned in the wish. Should I do that first before we do the production release (so we only have to do one release)?

I think that would be a good idea. However, we should probably not change too much at once. Maybe can you add another task for specific commands? We need to have good tests otherwise, it will be hard to fix problems if someone with a very special setup runs into issues.

Are you familiar with the Mathoid deployment (https://wikitech.wikimedia.org/wiki/Mathoid), otherwise I will search for someone?

I am afraid not.

That's ok.

I think that would be a good idea. However, we should probably not change too much at once. Maybe can you add another task for specific commands? We need to have good tests otherwise, it will be hard to fix problems if someone with a very special setup runs into issues.

One task for each command? I can start doing that tomorrow.

One task for each command? I can start doing that tomorrow.

If it is 3 commands or fewer that would reasonable, otherwise I guess it is too much overhead.
From reading the whish it was not entirely clear to me which commands were debated. See also T137787 T188598 and https://phabricator.wikimedia.org/project/profile/4848/ for whishes.

Change 790640 merged by jenkins-bot:

[mediawiki/services/texvcjs@master] Add support for phantom commands.

https://gerrit.wikimedia.org/r/790640

One task for each command? I can start doing that tomorrow.

If it is 3 commands or fewer that would reasonable, otherwise I guess it is too much overhead.
From reading the whish it was not entirely clear to me which commands were debated. See also T137787 T188598 and https://phabricator.wikimedia.org/project/profile/4848/ for whishes.

I have started a discussion on the wish page. I think there are quite a few extra commands we could support (if my understanding is correct).

Change 792642 had a related patch set uploaded (by Physikerwelt; author: Physikerwelt):

[mediawiki/services/texvcinfo@master] Ignore identifiers in phantom environment

https://gerrit.wikimedia.org/r/792642

Change 799273 had a related patch set uploaded (by Dom Walden; author: Dom Walden):

[mediawiki/services/mathoid@master] Unit tests for the new phantom commands.

https://gerrit.wikimedia.org/r/799273

I added links to the ongoing code review

Change 792642 merged by jenkins-bot:

[mediawiki/services/texvcinfo@master] Ignore identifiers in phantom environment

https://gerrit.wikimedia.org/r/792642

Change 799273 merged by jenkins-bot:

[mediawiki/services/mathoid@master] Unit tests for the new phantom commands.

https://gerrit.wikimedia.org/r/799273

Change 809002 had a related patch set uploaded (by Physikerwelt; author: Physikerwelt):

[mediawiki/services/mathoid@master] Update texvcinfo for phantom support

https://gerrit.wikimedia.org/r/809002

Change 809002 merged by jenkins-bot:

[mediawiki/services/mathoid@master] Update texvcinfo for phantom support

https://gerrit.wikimedia.org/r/809002

@dom_walden how is your availability in the near future? If we deploy the new mathoid version you or someone from your team should be around. Also is there a mechanism from Community-Tech to announce the change to the community? I remember a few years ago @JStrodt_WMDE wrote a very nice post about a "whish" implemented by the math community that I could also use for distribution within the math channels.

@dom_walden how is your availability in the near future? If we deploy the new mathoid version you or someone from your team should be around. Also is there a mechanism from Community-Tech to announce the change to the community? I remember a few years ago @JStrodt_WMDE wrote a very nice post about a "whish" implemented by the math community that I could also use for distribution within the math channels.

I should be around.

I think we have a few places where we can announce this. I will ask the rest of the team and let you know.

@dom_walden can you coordinate with @akosiaris about the deployment of the latest image at the moment (https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/809194)

I can. I am also discussing with @NRodriguez about when we are happy to release this, where to announce it, etc.

I also assume we want to announce to the Math mailing list. Are you happy to do this?