Page MenuHomePhabricator

Math: Support phantom tags
Closed, ResolvedPublic5 Estimated Story Points

Description

The Math-extension sadly doesn't support the phantom tag and shows an parser error:

Failed to parse (unknown function "\phantom"): {\displaystyle \frac{x+y+z}{x \phantom{+}\phantom{y}+z}}

Example code for mediawiki / TeX:

<math>\frac{x+y+z}{x \phantom{+}\phantom{y}+z}</math>

MathML code:

<math xmlns = "http://www.w3.org/1998/Math/MathML">
   <mfrac>
      <mrow>
         <mi> x </mi>
         <mo> + </mo>
         <mi> y </mi>
         <mo> + </mo>
         <mi> z </mi>
      </mrow>

      <mrow>
         <mi> x </mi>
         <mphantom>
            <mo> + </mo>
         </mphantom>

         <mphantom>
            <mi> y </mi>
         </mphantom>
         <mo> + </mo>
         <mi> z </mi>
      </mrow>
   </mfrac>
</math>

and here an example edit:
https://en.wikipedia.org/w/index.php?title=Wikipedia:Sandbox&oldid=974880776

And whoever it wants can use the following code for LibreOffice or OpenOffice:

{x + y + z} over {x {phantom +} {phantom y} + z}

Hence: LibreOffice and OpenOffice simply ignores the tag in the mediawiki-export extension and thus showing the complete x + y + z twwo times (above the bracket and under it).

For the mediawiki exporter extension code:
https://opengrok.libreoffice.org/xref/core/external/xsltml/xsltml_2.1.2.patch?r=2b383d19#1118

Progress:

Event Timeline

Change 790640 had a related patch set uploaded (by Dom Walden; author: Dom Walden):

[mediawiki/services/texvcjs@master] Add support for phantom commands.

https://gerrit.wikimedia.org/r/790640

I have made an attempt at a patch above. @Physikerwelt let me know what you think.

I hope I have put it in the right place. The \[vh]phantom commands seem to take only one argument so I assume they should be handled the same as commands like \textbf.

I am not sure how much unit testing is necessary. I have also tested this on my local wiki comparing the output between Mathoid and LaTeX. I haven't done much in the way of regression testing.

I appreciate this is in the "Needs community consensus" column. But, what counts as community consensus? 45 people voted for it in the last wish list https://meta.wikimedia.org/wiki/Community_Wishlist_Survey_2022/Editing/Missing_LaTeX_capabilities_for_math_rendering.

Moreover, it seems like a low risk and uncontroversial change. We are only exposing more of the features of MathJax. We are not interfering with existing functionality.

The only risk I can see is if people start using these new commands and then MathJax stops supporting them. However, they are standard TeX commands so it seems unlikely this would happen.

Physikerwelt added a subscriber: WDoranWMF.

I have made an attempt at a patch above. @Physikerwelt let me know what you think.

I will have a look. Have you spoken to @WDoranWMF about math-related changes?

Also, there are some pending changes in the queue.

https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/787831

for the mathoid update

and a conflicting change

https://gerrit.wikimedia.org/r/c/mediawiki/services/texvcjs/+/785246

I hope I have put it in the right place. The \[vh]phantom commands seem to take only one argument so I assume they should be handled the same as commands like \textbf.

I guess so.

I am not sure how much unit testing is necessary. I have also tested this on my local wiki comparing the output between Mathoid and LaTeX. I haven't done much in the way of regression testing.

I think we should have a look at what the rendering with mathoid will look like and test there. Moreover, I guess we should create a follow up ticket to notify people about the change so that the respective documentation can be updated.

I appreciate this is in the "Needs community consensus" column. But, what counts as community consensus? 45 people voted for it in the last wish list https://meta.wikimedia.org/wiki/Community_Wishlist_Survey_2022/Editing/Missing_LaTeX_capabilities_for_math_rendering.

Yes. This is probably ok. I will run a quick query on the mailing list just to give people a way to stop this.

Moreover, it seems like a low risk and uncontroversial change. We are only exposing more of the features of MathJax. We are not interfering with existing functionality.

I think so too.

The only risk I can see is if people start using these new commands and then MathJax stops supporting them. However, they are standard TeX commands so it seems unlikely this would happen.

We will probably move away from MathJax to a custom-made rendering engine, so that should not be a problem.

I have made an attempt at a patch above. @Physikerwelt let me know what you think.

I will have a look. Have you spoken to @WDoranWMF about math-related changes?

Not yet. I will do though, thanks.

Also, there are some pending changes in the queue.

...

https://gerrit.wikimedia.org/r/c/mediawiki/services/texvcjs/+/785246

Ah, I didn't notice this when I pushed my patch. Should I wait for this to be merged and update my patch?

I am not sure how much unit testing is necessary. I have also tested this on my local wiki comparing the output between Mathoid and LaTeX. I haven't done much in the way of regression testing.

I think we should have a look at what the rendering with mathoid will look like and test there. Moreover, I guess we should create a follow up ticket to notify people about the change so that the respective documentation can be updated.

It looks good to me locally. For example, the code <math>-e\sqrt{\vphantom{p'}p},\; -e'\sqrt{p'},\; \ldots</math> renders like:

vphantom_example.png (47×175 px, 1014 B)

The same code without \vphantom (i.e. <math>-e\sqrt{p},\; -e'\sqrt{p'},\; \ldots</math>) renders like:

example_without_vphantom.png (43×148 px, 935 B)

All the other examples I tried compared favourably to the same code rendered in LaTeX.

Good point about the documentation.

The only risk I can see is if people start using these new commands and then MathJax stops supporting them. However, they are standard TeX commands so it seems unlikely this would happen.

We will probably move away from MathJax to a custom-made rendering engine, so that should not be a problem.

Interesting, I didn't know this.

Physikerwelt set the point value for this task to 5.

I did manual testing in mathoid (with no_check setting to true). I can confirm that there are cases where phantom rendering works;-)

I changed the test description to describe the steps required to move this into production. @dom_walden would you help me with that, by either making code reviews or reviewing my patches?

Are you familiar with the Mathoid deployment (https://wikitech.wikimedia.org/wiki/Mathoid), otherwise I will search for someone?

I did manual testing in mathoid (with no_check setting to true). I can confirm that there are cases where phantom rendering works;-)

Thanks!

I changed the test description to describe the steps required to move this into production. @dom_walden would you help me with that, by either making code reviews or reviewing my patches?

I am happy to help as much as I can.

I didn't realise there were so many steps. I had plans to also implement more of the functions mentioned in the wish. Should I do that first before we do the production release (so we only have to do one release)?

Are you familiar with the Mathoid deployment (https://wikitech.wikimedia.org/wiki/Mathoid), otherwise I will search for someone?

I am afraid not.

I changed the test description to describe the steps required to move this into production. @dom_walden would you help me with that, by either making code reviews or reviewing my patches?

I am happy to help as much as I can.

Amazing. Thank you.

I didn't realise there were so many steps. I had plans to also implement more of the functions mentioned in the wish. Should I do that first before we do the production release (so we only have to do one release)?

I think that would be a good idea. However, we should probably not change too much at once. Maybe can you add another task for specific commands? We need to have good tests otherwise, it will be hard to fix problems if someone with a very special setup runs into issues.

Are you familiar with the Mathoid deployment (https://wikitech.wikimedia.org/wiki/Mathoid), otherwise I will search for someone?

I am afraid not.

That's ok.

I think that would be a good idea. However, we should probably not change too much at once. Maybe can you add another task for specific commands? We need to have good tests otherwise, it will be hard to fix problems if someone with a very special setup runs into issues.

One task for each command? I can start doing that tomorrow.

One task for each command? I can start doing that tomorrow.

If it is 3 commands or fewer that would reasonable, otherwise I guess it is too much overhead.
From reading the whish it was not entirely clear to me which commands were debated. See also T137787 T188598 and https://phabricator.wikimedia.org/project/profile/4848/ for whishes.

Change 790640 merged by jenkins-bot:

[mediawiki/services/texvcjs@master] Add support for phantom commands.

https://gerrit.wikimedia.org/r/790640

One task for each command? I can start doing that tomorrow.

If it is 3 commands or fewer that would reasonable, otherwise I guess it is too much overhead.
From reading the whish it was not entirely clear to me which commands were debated. See also T137787 T188598 and https://phabricator.wikimedia.org/project/profile/4848/ for whishes.

I have started a discussion on the wish page. I think there are quite a few extra commands we could support (if my understanding is correct).

Change 792642 had a related patch set uploaded (by Physikerwelt; author: Physikerwelt):

[mediawiki/services/texvcinfo@master] Ignore identifiers in phantom environment

https://gerrit.wikimedia.org/r/792642

Change 799273 had a related patch set uploaded (by Dom Walden; author: Dom Walden):

[mediawiki/services/mathoid@master] Unit tests for the new phantom commands.

https://gerrit.wikimedia.org/r/799273

I added links to the ongoing code review

Change 792642 merged by jenkins-bot:

[mediawiki/services/texvcinfo@master] Ignore identifiers in phantom environment

https://gerrit.wikimedia.org/r/792642

Change 799273 merged by jenkins-bot:

[mediawiki/services/mathoid@master] Unit tests for the new phantom commands.

https://gerrit.wikimedia.org/r/799273

Change 809002 had a related patch set uploaded (by Physikerwelt; author: Physikerwelt):

[mediawiki/services/mathoid@master] Update texvcinfo for phantom support

https://gerrit.wikimedia.org/r/809002

Change 809002 merged by jenkins-bot:

[mediawiki/services/mathoid@master] Update texvcinfo for phantom support

https://gerrit.wikimedia.org/r/809002

@dom_walden how is your availability in the near future? If we deploy the new mathoid version you or someone from your team should be around. Also is there a mechanism from Community-Tech to announce the change to the community? I remember a few years ago @JStrodt_WMDE wrote a very nice post about a "whish" implemented by the math community that I could also use for distribution within the math channels.

@dom_walden how is your availability in the near future? If we deploy the new mathoid version you or someone from your team should be around. Also is there a mechanism from Community-Tech to announce the change to the community? I remember a few years ago @JStrodt_WMDE wrote a very nice post about a "whish" implemented by the math community that I could also use for distribution within the math channels.

I should be around.

I think we have a few places where we can announce this. I will ask the rest of the team and let you know.

@dom_walden can you coordinate with @akosiaris about the deployment of the latest image at the moment (https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/809194)

I can. I am also discussing with @NRodriguez about when we are happy to release this, where to announce it, etc.

I also assume we want to announce to the Math mailing list. Are you happy to do this?

I am happy to share a text or a link. However, you can also do it yourself. I can't find time to write reasonable documentation in the near future.

I am happy to share a text or a link. However, you can also do it yourself. I can't find time to write reasonable documentation in the near future.

OK, I can do it instead. I am happy to write any documentation as well.

That reminds me:

  • update documentation on ctan

I assume you mean https://ctan.org/. What would we need to update?

We were quite bad in the past in updating the documentation of the functionality. https://www.ctan.org/pkg/texvc The idea is that if you include only the texvc package in a plain latex document you can compile tex from MediaWiki without undefined macros. If phantom is part of core tex there might be no need to change anything in the package documentation.

We were quite bad in the past in updating the documentation of the functionality. https://www.ctan.org/pkg/texvc The idea is that if you include only the texvc package in a plain latex document you can compile tex from MediaWiki without undefined macros. If phantom is part of core tex there might be no need to change anything in the package documentation.

Oh, I didn't think we had updated that package. But [vh]phantom are part of core tex so I guess there is nothing to update.

@Physikerwelt When are we going to do the npm release (sorry I don't know how to do this myself).

@Physikerwelt When are we going to do the npm release (sorry I don't know how to do this myself).

will do... however, this is not a requirement. Just discuss with @akosiaris when the docker image can be deployed (independent of npm)

mathoid 0.7.6. was published

@Physikerwelt this was deployed to production yesterday afternoon. My team is preparing the announcements to the community. Thank you for all your work on this.

Thank you. Eventually, your feedback on T302628 would be highly appreciated. I am got the feeling that the current process is too complex and we should transition to a simpler model.

Quiddity subscribed.

Re: Tech News - Thank you @NRodriguez for adding a proposed entry.
Question: I wonder if there's a better onwiki page that could be linked to, from that entry?
I.e. Perhaps there is a translatable documentation page on MediaWiki or Meta-wiki that contains (or should contain) details about this newly-available feature?

Overall, I'd prefer to wait to include an entry for this, until it has a better link destination than the English-only and technical-implementation-focused Task Description above. If that exists, or can be created before Monday, we can include the entry this week; otherwise I suggest postponing (and have removed it temporarily). Thanks!

glad you added the announcements to that list! cool

Re: Tech News - Thank you @NRodriguez for adding a proposed entry.
Question: I wonder if there's a better onwiki page that could be linked to, from that entry?
I.e. Perhaps there is a translatable documentation page on MediaWiki or Meta-wiki that contains (or should contain) details about this newly-available feature?

Overall, I'd prefer to wait to include an entry for this, until it has a better link destination than the English-only and technical-implementation-focused Task Description above. If that exists, or can be created before Monday, we can include the entry this week; otherwise I suggest postponing (and have removed it temporarily). Thanks!

Is anyone able to help document this new feature? The most suitable location I can find is https://meta.wikimedia.org/wiki/Help:Displaying_a_formula (which is tagged as 'outdated', but seems to be the best location to document it, and might lead to further improvements in the wiki-way). Thanks.

Is anyone able to help document this new feature? The most suitable location I can find is https://meta.wikimedia.org/wiki/Help:Displaying_a_formula (which is tagged as 'outdated', but seems to be the best location to document it, and might lead to further improvements in the wiki-way). Thanks.

I can write some documentation when I have time.

@dom_walden thanks for offering to write documentation, let me know if you need support.

Thank you @dom_walden that's perfect! I've re-added the entry to next week's Tech News in https://meta.wikimedia.org/wiki/Tech/News/2022/31 linking to those meta-wiki docs.
Tech News will be frozen for translation on Friday, if any tweaks are needed before then.