Page MenuHomePhabricator

Mathoid instance not working with MW 1.28
Closed, ResolvedPublic

Description

Hi @Physikerwelt,

I'm hoping you can help me with the Math extension again. I am looking at moving my wiki from MW 1.26 to MW 1.28.

In MW 1.26 you previously helped me in T119187 to get the Math extension working which I greatly appreciated. For reasons I do not fathom eventually the Math extension just stopped working around June 2016 (without I think any intervention on the server). Short on time I used SimpleMathJax for quite a while.

I'm now trying to get MW 1.28 working nicely through a bunch of Docker images and trying to reinstate the Math extension.

If I specify the settings as shown below:

# Math extension
#  - this was removed June 2016 as not functioning correctly and repalced with SimpleMathJax
#  - reinstated for testing in MW 1.28
require_once "$IP/extensions/Math/Math.php";
# wfLoadExtension( 'Math' );
// Set Mathoid as default rendering option;
$wgMathValidModes[] = 'mathml';
$wgDefaultUserOptions['math'] = 'mathml';
$wgMathMathMLUrl = 'http://mediawiki-mathoid-1:10044/'; # IP of Mathoid server
# $wgMathFullRestbaseURL = 'http://mediawiki-mathoid-1:10044/'; # IP of Mathoid server
$wgMathFullRestbaseURL= 'https://api.formulasearchengine.com/';

Then going to Special:MathStatus gives me 'succeeded' for all tests.

However on a page which might have 20 or 30 formulas, I find some do not render correctly and show instead:

Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://api.formulasearchengine.com/v1/":): {\displaystyle \Omega}

I can't see anything much that I can do about this. It's quite repeatable and I'm not sure how to diagnose further.

So:

  • Problem 1: Using the default Math settings doesn't provide consistent performance. No particular idea of what diagnostics to do to explore this further.

I thought. "Why don't I set up a local mathoid install to see if I can get better performance?'. The commented out $wgMathFullRestbaseURL points to a local docker container. This runs in a separate container to the mediawiki instance, however to test I carry out (within the mediawiki container) a command like this:

root@6705d8cb6e2e:/var/www/html/mediawiki# curl -d 'q=E=mc^2' mediawiki-mathoid-1:10044
curl: (7) Failed to connect to mediawiki-mathoid-1 port 10044: Connection refused
root@6705d8cb6e2e:/var/www/html/mediawiki# curl -d 'q=E=mc^2' mediawiki-mathoid-1:10044
curl: (7) Failed to connect to mediawiki-mathoid-1 port 10044: Connection refused
root@6705d8cb6e2e:/var/www/html/mediawiki# curl -d 'q=E=mc^2' mediawiki-mathoid-1:10044
{"mml":"<math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"block\">\n  <semantics>\n    <mrow>\n      <mi>E</mi>\n      <mo>=</mo>\n      <mi>m</mi>\n      <msup>\n        <mi>c</mi>\n        <mn>2</mn>\n      </msup>\n    </mrow>\n    <annotation encoding=\"application/x-tex\">E=mc^2</annotation>\n  </semantics>\n</math>","svg":"<?xml version=\"1.0\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"9.025ex\" height=\"2.676ex\" style=\"vertical-align: -0.338ex;\" viewBox=\"0 -1006.6 3885.6 1152.1\" xmlns=\"http://www.w3.org/2000/svg\">\n<defs>\n<path stroke-width=\"10\" id=\"E1-MJMATHI-45\" d=\"M492 213Q472 213 472 226Q472 230 477 250T482 285Q482 316 461 323T364 330H312Q311 328 277 192T243 52Q243 48 254 48T334 46Q428 46 458 48T518 61Q567 77 599 117T670 248Q680 270 683 272Q690 274 698 274Q718 274 718 261Q613 7 608 2Q605 0 322 0H133Q31 0 31 11Q31 13 34 25Q38 41 42 43T65 46Q92 46 125 49Q139 52 144 61Q146 66 215 342T285 622Q285 629 281 629Q273 632 228 634H197Q191 640 191 642T193 659Q197 676 203 680H757Q764 676 764 669Q764 664 751 557T737 447Q735 440 717 440H705Q698 445 698 453L701 476Q704 500 704 528Q704 558 697 578T678 609T643 625T596 632T532 634H485Q397 633 392 631Q388 629 386 622Q385 619 355 499T324 377Q347 376 372 376H398Q464 376 489 391T534 472Q538 488 540 490T557 493Q562 493 565 493T570 492T572 491T574 487T577 483L544 351Q511 218 508 216Q505 213 492 213Z\"></path>\n<path stroke-width=\"10\" id=\"E1-MJMAIN-3D\" d=\"M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z\"></path>\n<path stroke-width=\"10\" id=\"E1-MJMATHI-6D\" d=\"M21 287Q22 293 24 303T36 341T56 388T88 425T132 442T175 435T205 417T221 395T229 376L231 369Q231 367 232 367L243 378Q303 442 384 442Q401 442 415 440T441 433T460 423T475 411T485 398T493 385T497 373T500 364T502 357L510 367Q573 442 659 442Q713 442 746 415T780 336Q780 285 742 178T704 50Q705 36 709 31T724 26Q752 26 776 56T815 138Q818 149 821 151T837 153Q857 153 857 145Q857 144 853 130Q845 101 831 73T785 17T716 -10Q669 -10 648 17T627 73Q627 92 663 193T700 345Q700 404 656 404H651Q565 404 506 303L499 291L466 157Q433 26 428 16Q415 -11 385 -11Q372 -11 364 -4T353 8T350 18Q350 29 384 161L420 307Q423 322 423 345Q423 404 379 404H374Q288 404 229 303L222 291L189 157Q156 26 151 16Q138 -11 108 -11Q95 -11 87 -5T76 7T74 17Q74 30 112 181Q151 335 151 342Q154 357 154 369Q154 405 129 405Q107 405 92 377T69 316T57 280Q55 278 41 278H27Q21 284 21 287Z\"></path>\n<path stroke-width=\"10\" id=\"E1-MJMATHI-63\" d=\"M34 159Q34 268 120 355T306 442Q362 442 394 418T427 355Q427 326 408 306T360 285Q341 285 330 295T319 325T330 359T352 380T366 386H367Q367 388 361 392T340 400T306 404Q276 404 249 390Q228 381 206 359Q162 315 142 235T121 119Q121 73 147 50Q169 26 205 26H209Q321 26 394 111Q403 121 406 121Q410 121 419 112T429 98T420 83T391 55T346 25T282 0T202 -11Q127 -11 81 37T34 159Z\"></path>\n<path stroke-width=\"10\" id=\"E1-MJMAIN-32\" d=\"M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z\"></path>\n</defs>\n<g stroke=\"currentColor\" fill=\"currentColor\" stroke-width=\"0\" transform=\"matrix(1 0 0 -1 0 0)\">\n <use xlink:href=\"#E1-MJMATHI-45\" x=\"0\" y=\"0\"></use>\n <use xlink:href=\"#E1-MJMAIN-3D\" x=\"1046\" y=\"0\"></use>\n <use xlink:href=\"#E1-MJMATHI-6D\" x=\"2107\" y=\"0\"></use>\n<g transform=\"translate(2990,0)\">\n <use xlink:href=\"#E1-MJMATHI-63\" x=\"0\" y=\"0\"></use>\n <use transform=\"scale(0.707)\" xlink:href=\"#E1-MJMAIN-32\" x=\"619\" y=\"583\"></use>\n</g>\n</g>\n</svg>","mathoidStyle":"vertical-align: -0.338ex; width:9.025ex; height:2.676ex;","success":true,"log":"success"}root@6705d8cb6e2e:/var/www/html/mediawiki#

This works - but sporadically. Sometimes I get a refused connection. Sometimes the curl command hangs and I get a 'Connection reset by peer'. I think this is more likely a problem with mathoid then a container connectivity issue.

My config.yaml is straightforward. I think only a port change. It would be great if there was some guidance on stats and logging if not part of a Wikimedia cluster (what should be changed and what dependencies are required). I don't know how to redirect logs to a local file, which would be quite useful.

# Number of worker processes to spawn.
# Set to 0 to run everything in a single process without clustering.
# Use 'ncpu' to run as many workers as there are CPU units
num_workers: ncpu

# Log error messages and gracefully restart a worker if v8 reports that it
# uses more heap (note: not RSS) than this many mb.
worker_heap_limit_mb: 500

# Logger info
logging:
  level: warn
  streams:
  # Use gelf-stream -> logstash
  - type: gelf
    host: logstash1003.eqiad.wmnet
    port: 12201

# Statsd metrics reporter
metrics:
  type: statsd
  host: statsd.eqiad.wmnet
  port: 8125

services:
  - name: mathoid
    # a relative path or the name of an npm package, if different from name
    module: ./app.js
    # optionally, a version constraint of the npm package
    # version: ^0.4.0
    # per-service config
    conf:
      port: 10044
      # interface: localhost # uncomment to only listen on localhost
      # more per-service config settings
      svg: true
      img: true
      texvcinfo: true
      speech: true
      speech_on: false
      no_check: true

The Dockerfile for this container (which I thought might be useful on the mathoid repo) is the following:

FROM buildpack-deps:jessie-curl
# from: https://www.mediawiki.org/wiki/Mathoid
# and: https://github.com/wikimedia/mathoid


###################################################################
#********************* Environment Variables *********************#
###################################################################

# terminal setup for nano
ENV TERM xterm

###################################################################
#************************* Install Mathoid ***********************#
###################################################################

WORKDIR /root

# Install Node 4.x
RUN sh -c "curl -sL https://deb.nodesource.com/setup_4.x | bash -"

# Dependencies
RUN apt-get install -y git nodejs librsvg2-dev build-essential nano

# Version check
RUN npm -v
RUN node -v

# Modules after mathoid required for testing
RUN npm install mathoid mocha extend rewire mocha-jshint

COPY import/docker-entrypoint.sh /entrypoint.sh
COPY import/config.yaml /root/node_modules/mathoid/config.yaml

# Port for accessing Mathoid API
EXPOSE 10044

ENTRYPOINT ["/entrypoint.sh"]

# Mathoid installed to this location
WORKDIR /root/node_modules/mathoid

CMD ["nodejs", "server.js"]

# try:  
#  - curl -d 'q=E=mc^2' localhost:10044

When I change the LocalSettings.php configuration file to use the local mathoid server:

$wgMathMathMLUrl = 'http://mediawiki-mathoid-1:10044/'; # IP of Mathoid server
$wgMathFullRestbaseURL = 'http://mediawiki-mathoid-1:10044/'; # IP of Mathoid server
# $wgMathFullRestbaseURL= 'https://api.formulasearchengine.com/';

then none of my formulas render when I force a page refresh. All give the same error:

Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "http://mediawiki-mathoid-1:10044/v1/":): {\displaystyle R_{arc}=\frac{28707L}{I^{{1.4}}}}

Now when I go to Special:MathStatus I see the following (after quite a lengthy delay and a couple of clicks on the page name):

Internal error
[5a8d27c5c95bf895f55ae13e] /mediawiki/index.php/Special:MathStatus MWException from line 104 of /var/www/html/mediawiki/extensions/Math/MathRestbaseInterface.php: TeX input is invalid.

Backtrace:

#0 /var/www/html/mediawiki/extensions/Math/MathRestbaseInterface.php(395): MathRestbaseInterface->calculateHash()
#1 /var/www/html/mediawiki/extensions/Math/MathRestbaseInterface.php(95): MathRestbaseInterface->getContentRequest(string)
#2 /var/www/html/mediawiki/extensions/Math/MathRestbaseInterface.php(224): MathRestbaseInterface->getContent(string)
#3 /var/www/html/mediawiki/extensions/Math/MathRenderer.php(657): MathRestbaseInterface->getSvg()
#4 /var/www/html/mediawiki/extensions/Math/MathMathML.php(357): MathRenderer->getSvg()
#5 /var/www/html/mediawiki/extensions/Math/MathMathML.php(398): MathMathML->correctSvgStyle(NULL)
#6 /var/www/html/mediawiki/extensions/Math/MathMathML.php(466): MathMathML->getFallbackImage()
#7 /var/www/html/mediawiki/extensions/Math/SpecialMathStatus.php(70): MathMathML->getHtmlOutput()
#8 /var/www/html/mediawiki/extensions/Math/SpecialMathStatus.php(52): SpecialMathStatus->testSpecialCaseText()
#9 /var/www/html/mediawiki/extensions/Math/SpecialMathStatus.php(42): SpecialMathStatus->runMathMLTest(string)
#10 /var/www/html/mediawiki/includes/specialpage/SpecialPage.php(522): SpecialMathStatus->execute(NULL)
#11 /var/www/html/mediawiki/includes/specialpage/SpecialPageFactory.php(576): SpecialPage->run(NULL)
#12 /var/www/html/mediawiki/includes/MediaWiki.php(283): SpecialPageFactory::executePath(Title, RequestContext)
#13 /var/www/html/mediawiki/includes/MediaWiki.php(851): MediaWiki->performRequest()
#14 /var/www/html/mediawiki/includes/MediaWiki.php(512): MediaWiki->main()
#15 /var/www/html/mediawiki/index.php(43): MediaWiki->run()
#16 {main}

So:

  • Problem 2: Even with a mathoid service running, integration with Mediawiki seems problematic and seems to cause failure of the MathStatus page.

One thing is I am a little unsure if wgMathFullRestbaseURL should include the port number for a custom instance. Could you confirm? I have tried both ways without either causing the MathStatus page to render correctly.

Any and all help appreciated :-)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 1 2017, 10:48 AM

I was planning to describe the setup. But I did not do it yet. The best resource I can point to is
http://cicm-conference.org/2016/ceur-ws/W48.pdf
i.e. Fig 2 and Section 3
Using mathoid without restbase has not been tested recently, and probably does not work.
There are two things you could try.

  1. Replace formulasearchengine.com with https://en.wikipedia.org/api/rest_
  2. If you set up your local restbase instance you should first the web ui which should look like https://en.wikipedia.org/api/rest_v1/#!/Math/post_media_math_check_type

I think the documentation of restbase is quite good, now. The configuration I am currently using for formulaseachengine.com is available from https://github.com/physikerwelt/restbase-config-fse/blob/master/formulasearchengine.com.yaml But you do not need to install mathpipe.

I hope this pointer can are somehow useful. However, I know that it's not well documented and that I should improve the documentation asap.

Physikerwelt triaged this task as Normal priority.Jan 1 2017, 6:04 PM
karel added a subscriber: karel.Jan 21 2017, 10:15 AM

Hello,

I think I have the same problem. I am using MediaWiki 1.28 with Math extension that says this in version file:

Math: REL1_28
2016-11-18T19:09:57

b6699f8

I have installed Mathoid from ubuntu package, I don't know where I can find Mathoid version.

I think Mathoid is working well, because I can successfully test it locally using curl or remotely using SoapUI. But when I open page in MediaWiki I get this error:

Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "http:/localhost:10044/v1/":): {\displaystyle \pi}

MathStatus special page gives me this:

Running backend tests for rendering mode MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools).

Test Rendering the input "x^2+\text{a sample Text}" succeeded.

Test Comparing to the reference rendering succeeded.

Test Rendering of a+b in plain MathML mode succeeded.

Test Checking the presence of '+' in the MathML output succeeded.

Test Comparing the generated SVG with the reference failed.

Test Checking if MathML input is supported succeeded.

Test Rendering Presentation MathML sample failed.

Test Checking if the link to SVG image is correct failed.
Returned string

<strong class='error texerror'>Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response (&quot;&lt;p&gt;There was a problem during the HTTP request: 502 Bad Gateway
&lt;/p&gt;&quot;) from server &quot;http://mathoid.testme.wmflabs.org&quot;:): {\displaystyle &lt;msub&gt;  &lt;mrow&gt;  &lt;mi&gt; P&lt;/mi&gt; &lt;/mrow&gt;  &lt;mrow&gt;  &lt;mi&gt; i&lt;/mi&gt;  &lt;mi&gt; j&lt;/mi&gt; &lt;/mrow&gt; &lt;/msub&gt;  &lt;mo&gt; =&lt;/mo&gt;  &lt;mfrac&gt;  &lt;mrow&gt;  &lt;mn&gt; 100&lt;/mn&gt;  &lt;msub&gt;  &lt;mrow&gt;  &lt;mi&gt; d&lt;/mi&gt; &lt;/mrow&gt;  &lt;mrow&gt;  &lt;mi&gt; i&lt;/mi&gt;  &lt;mi&gt; j&lt;/mi&gt; &lt;/mrow&gt; &lt;/msub&gt; &lt;/mrow&gt;  &lt;mrow&gt;  &lt;mn&gt; 6.75&lt;/mn&gt;  &lt;msub&gt;  &lt;mrow&gt;  &lt;mi&gt; r&lt;/mi&gt; &lt;/mrow&gt;  &lt;mrow&gt;  &lt;mi&gt; j&lt;/mi&gt; &lt;/mrow&gt; &lt;/msub&gt; &lt;/mrow&gt; &lt;/mfrac&gt;  &lt;mo&gt; ,&lt;/mo&gt; &lt;/math&gt;}</strong>

Also, when I set wgMathFullRestbaseURL to api.formulasearchengine.com, it works just fine. I think there is some bug in integration of latest mathoid and Math extension.

Hi @Physikerwelt

Thanks for your response and apologies for the lateness of mine.

I proceeded to install MW 1.28 and was delighted with the new VE Math interface and that Extension:Math was working for but today quite a number of formulas refuse to render.

Some may be because there was HTML in them for less than or equals (which worked previously) or maybe the cache is re-rendering (hard to say).

I have made a note in https://www.mediawiki.org/wiki/Extension:Math#Mathoid that Mathoid probably requires a RESTbase instance (which I don't have) but even using the formula api something persistently breaks. I've not been able to set up a good fallback (and I feel MathJax was delightfully simple in this regard compared to setting up texvc). I may have to head back to SimpleMathJax (again!). I'll probably try and diagnose a little more...

I finally gave up trying to install the Math extension on my Ubuntu 16.04 system. Instead, I made an extremely simple extension https://www.mediawiki.org/wiki/Extension:MathEasy which only depends on Mathoid, which I intended to be as a temporary replacement for the Math extension, pending the completion of its documentation and/or development.

If anyone is interested in improving this extension, please do so or send me a message.

Mbrt added a subscriber: Mbrt.Jun 8 2017, 10:27 PM

I get error with open a reference mywiki [WTnOf@v57z3nd7QKx11cIQAAAIY] /wiki/Istimewa:Preferensi MWException from line 151 of /home/ckmpga/public_html/wiki/includes/Preferences.php: Global default 'latexml' is invalid for field mathBacktrace:#0 /home/ckmpga/public_html/wiki/includes/Preferences.php(103): Preferences::loadPreferenceValues(User, RequestContext, array)#1 /home/ckmpga/public_html/wiki/includes/Preferences.php(1296): Preferences::getPreferences(User, RequestContext)#2 /home/ckmpga/public_html/wiki/includes/specials/SpecialPreferences.php(84): Preferences::getFormObject(User, RequestContext)#3 /home/ckmpga/public_html/wiki/includes/specialpage/SpecialPage.php(522): SpecialPreferences->execute(NULL)#4 /home/ckmpga/public_html/wiki/includes/specialpage/SpecialPageFactory.php(577): SpecialPage->run(NULL)#5 /home/ckmpga/public_html/wiki/includes/MediaWiki.php(283): SpecialPageFactory::executePath(Title, RequestContext)#6 /home/ckmpga/public_html/wiki/includes/MediaWiki.php(851): MediaWiki->performRequest()#7 /home/ckmpga/public_html/wiki/includes/MediaWiki.php(512): MediaWiki->main()#8 /home/ckmpga/public_html/wiki/index.php(43): MediaWiki->run()#9 {main}

@Mbrt: How is that stacktrace related to the information in this task?

I just found out how to configure Mathoid and Restbase.
In my case, Mathoid, Restbase, Parsoid and MW is all in one machine.

What I really found is there are hardcoded domains(/wikimedia.org/) are in restbase/v1/mathoid file. And I changed to /{domain}/.
Actually there are some hardcoded domain restbase/sys/mathoid.js. but I didn't changed them because I don't understand code fully.
If somebody know what the hardcode domain means, I guess they should changed to something.

Anyway my Wiki working well if [*] have "read" permission. Here are my configuration files.

  • mathoid/config.yaml
# Number of worker processes to spawn.
# Set to 0 to run everything in a single process without clustering.
# Use 'ncpu' to run as many workers as there are CPU units
num_workers: 0

# Log error messages and gracefully restart a worker if v8 reports that it
# uses more heap (note: not RSS) than this many mb.
worker_heap_limit_mb: 250

# Logger info
logging:
  level: trace
#  streams:
#  # Use gelf-stream -> logstash
#  - type: gelf
#    host: logstash1003.eqiad.wmnet
#    port: 12201

# Statsd metrics reporter
metrics:
  #type: log
  #host: localhost
  #port: 8125

services:
  - name: mathoid
    # a relative path or the name of an npm package, if different from name
    module: ./app.js
    # optionally, a version constraint of the npm package
    # version: ^0.4.0
    # per-service config
    conf:
      port: 10044
      # interface: localhost # uncomment to only listen on localhost
      # more per-service config settings
      # the location of the spec, defaults to spec.yaml if not specified
      # spec: ./spec.template.yaml
      # allow cross-domain requests to the API (default '*')
      #cors: '*'
      # to disable use:
      # cors: false
      # to restrict to a particular domain, use:
      # cors: restricted.domain.org
      # content for the CSP headers
      # csp: false  # uncomment this line to disable sending them
      # URL of the outbound proxy to use (complete with protocol)
      # proxy: http://my.proxy.org:8080
      # the list of domains for which not to use the proxy defined above
      # no_proxy_list:
      #   - domain1.com
      #   - domain2.org
      # the list of incoming request headers that can be logged; if left empty,
      # the following headers are allowed: cache-control, content-length,
      # content-type, if-match, user-agent, x-request-id
      # log_header_whitelist:
      #   - cache-control
      #   - content-length
      #   - content-type
      #   - if-match
      #   - user-agent
      #   - x-request-id
      # list of enabled renders
      svg: true
      img: true
      png: true #new feature
      speech: true #new feature
      texvcinfo: true
      speech_on: true
      no_check: false
      dpi: 180
      svgo: false
      # the user agent to use when issuing requests
      # user_agent: service-template-node
      # the template used for contacting the MW API
      #mwapi_req:
      #  method: post
      #  uri: https://{{domain}}/w/api.php
      #  headers:
      #    user-agent: '{{user-agent}}'
      #  body: '{{ default(request.query, {}) }}'
      # the template used for contacting RESTBase
      #restbase_req:
      #  method: '{{request.method}}'
      #  uri: https://{{domain}}/api/rest_v1/{+path}
      #  query: '{{ default(request.query, {}) }}'
      #  headers: '{{request.headers}}'
      #  body: '{{request.body}}'
  • restbase/config.yaml
services:
  - name: restbase
    module: hyperswitch
    conf:
      port: 7231
      salt: secret
      default_page_size: 125
      user_agent: RESTBase
      ui_name: RESTBase
      ui_url: https://www.mediawiki.org/wiki/RESTBase
      ui_title: RESTBase docs
      spec:
        x-request-filters:
          - path: lib/security_response_header_filter.js
        x-sub-request-filters:
          - type: default
            name: http
            options:
            allow:
              - pattern: http://localhost/api.php
                forward_headers: true
              - pattern: http://localhost:8000
                forward_headers: true
              - pattern: http://localhost:10044
                forward_headers: true
              - pattern: /^https?:\/\//
        paths:
          /{domain:localhost}: &arbitary_domain
            x-modules:
              - path: projects/MR.yaml
                options:
                  action:
                    # XXX Check API URL!
                    apiUriTemplate: http://localhost/api.php
                    # XXX Check the base RESTBase URI
                    baseUriTemplate: "{{'http://localhost:7231/{domain}/v1'}}"
                  parsoid:
                    # XXX Check Parsoid URL!
                    host: http://localhost:8000
                  mathoid:
                    # XXX Check Mathoid URL!
                    host: http://localhost:10044
                  table:
                    backend: sqlite
                    dbname: db.sqlite3
                    pool_idle_timeout: 20000
                    retry_delay: 250
                    retry_limit: 10
                    show_sql: false
#          /{domain:your.some.domain}: *arbitary_domain

# Finally, a standard service-runner config.
info:
  name: restbase

logging:
  name: restbase
  level: info
  • restbase/project/MR.yaml
# will work with Mathoid on any {domain}
paths:
  /{api:v1}:
    x-modules:
        # swagger options, overriding the shared ones from the merged specs (?)
      - spec:
          info:
            version: 1.0.0
            title: Wikimedia REST API
            description: Welcome to your RESTBase API.
          #securityDefinitions: &wp/content-security/1.0.0
          #  mediawiki_auth:
          #    description: Checks permissions using MW api
          #    type: apiKey
          #    in: header
          #    name: cookie
          #    x-internal-request-whitelist:
          #      - /http:\/\/[a-zA-Z0-9\.]+\/w\/api\.php/
          # Override the base path for host-based (proxied) requests. In our case,
          # we proxy https://{domain}/api/rest_v1/ to the API.
          #x-host-basePath: /api/rest_v1
          x-route-filters:
            - path: ./lib/normalize_title_filter.js
              options:
                redirect_cache_control: '{{options.purged_cache_control}}'
          paths:
            /media:
              x-modules:
                - path: v1/mathoid.yaml
                  options: '{{options.mathoid}}'
            #/testing:
            #  x-modules:
            #    - path: ../mathpipe.yaml
                 # options: '{{options.mathoid}}'
            /page:
              x-modules:
                - path: v1/content.yaml
                  options:
                    purged_cache_control: '{{options.purged_cache_control}}'
                - path: v1/common_schemas.yaml # Doesn't really matter where to mount it.
            /transform:
              x-modules:
                - path: v1/transform.yaml
        options: '{{options}}'

  /{api:sys}:
    x-modules:
      - spec:
          paths:
            /mathoid:
              x-modules:
                - path: sys/mathoid.js
                  options: '{{options.mathoid}}'
            /table:
              x-modules:
                - path: sys/table.js
                  options:
                    conf: '{{options.table}}'
            /key_value:
              x-modules:
                - path: sys/key_value.js
            /key_rev_value:
              x-modules:
                - path: sys/key_rev_value.js
            /key_rev_latest_value:
              x-modules:
                - path: sys/key_rev_latest_value.js
            /post_data: &sys_post_data
              x-modules:
                - path: sys/post_data.js
            /events:
              x-modules:
                - path: sys/events.js
                  options: '{{options.events}}'
            /page_revisions:
              x-modules:
                - path: sys/page_revisions.js
            /action:
              x-modules:
                - path: sys/action.js
                  options: '{{options.action}}'
            /page_save:
              x-modules:
                - path: sys/page_save.js
            /parsoid:
              x-modules:
                - path: sys/parsoid.js
                  options:
                    parsoidHost: '{{options.parsoid.host}}'
                    response_cache_control: '{{options.purged_cache_control}}'
        options: '{{options}}'
  • restbase/v1/mathoid.yaml
# Mathoid - math formula rendering service :simply replaced /wikimedia.org/sys to /{domain}/sys
tags:
  - name: Math
    description: formula rendering
paths:
  /math/check/{type}:
    post:
      tags: ['Math']
      summary: Check and normalize a TeX formula.
      description: |
        Checks the supplied TeX formula for correctness and returns the
        normalised formula representation as well as information about
        identifiers. Available types are tex and inline-tex. The response
        contains the `x-resource-location` header which can be used to retrieve
        the render of the checked formula in one of the supported rendering
        formats. Just append the value of the header to `/media/math/{format}/`
        and perform a GET request against that URL.

        Stability: [stable](https://www.mediawiki.org/wiki/API_versioning#Stable).
      produces:
        - application/json
      parameters:
        - name: type
          in: path
          description: The input type of the given formula; can be tex or inline-tex
          type: string
          required: true
          enum:
            - tex
            - inline-tex
            - chem
        - name: q
          in: formData
          description: The formula to check
          type: string
          required: true
      responses:
        '200':
          description: Information about the checked formula
        '400':
          description: Invalid type
          schema:
            $ref: '#/definitions/problem'
        default:
          description: Error
          schema:
            $ref: '#/definitions/problem'
      x-monitor: true
      x-amples:
        - title: Mathoid - check test formula
          request:
            params:
              domain: wikimedia.org
              type: tex
            body:
              q: E=mc^{2}
          response:
            status: 200
            headers:
              content-type: /^application\/json/
              x-resource-location: /.+/
              cache-control: 'no-cache'
            body:
              success: true
              checked: /.+/
      x-request-handler:
        - get_from_sys:
            request:
              method: post
              uri: /{domain}/sys/mathoid/check/{type}
              headers: '{{ request.headers }}'
              body: '{{ request.body }}'

  /math/formula/{hash}:
    get:
      tags: ['Math']
      summary: Get a previously-stored formula
      description: |
        Returns the previously-stored formula via `/media/math/check/{type}` for
        the given hash.

        Stability: [stable](https://www.mediawiki.org/wiki/API_versioning#Stable).
      produces:
        - application/json
      parameters:
        - name: hash
          in: path
          description: The hash string of the previous POST data
          type: string
          required: true
          minLength: 1
      responses:
        '200':
          description: Information about the checked formula
        '404':
          description: Data for the given hash cannot be found
          schema:
            $ref: '#/definitions/problem'
        default:
          description: Error
          schema:
            $ref: '#/definitions/problem'
      x-monitor: false
      x-request-handler:
        - get_from_sys:
            request:
              method: get
              uri: /{domain}/sys/mathoid/formula/{hash}
              headers: '{{ request.headers }}'

  /math/render/{format}/{hash}:
    get:
      tags: ['Math']
      summary: Get rendered formula in the given format.
      description: |
        Given a request hash, renders a TeX formula into its mathematic
        representation in the given format. When a request is issued to the
        `/media/math/check/{format}` POST endpoint, the response contains the
        `x-resource-location` header denoting the hash ID of the POST data. Once
        obtained, this endpoint has to be used to obtain the actual render.

        Stability: [stable](https://www.mediawiki.org/wiki/API_versioning#Stable).
      produces:
        - image/svg+xml
        - application/mathml+xml
        - image/png
      parameters:
        - name: format
          in: path
          description: The output format; can be svg or mml
          type: string
          required: true
          enum:
            - svg
            - mml
            - png
        - name: hash
          in: path
          description: The hash string of the previous POST data
          type: string
          required: true
          minLength: 1
      responses:
        '200':
          description: The rendered formula
        '404':
          description: Unknown format or hash ID
          schema:
            $ref: '#/definitions/problem'
        default:
          description: Error
          schema:
            $ref: '#/definitions/problem'
      x-monitor: false
      x-setup-handler:
        - init_svg:
            uri: /{domain}/sys/key_value/mathoid.svg
            body:
              keyType: string
              valueType: string
        - init_mml:
            uri: /{domain}/sys/key_value/mathoid.mml
            body:
              keyType: string
              valueType: string
        - init_png:
            uri: /{domain}/sys/key_value/mathoid.png
            body:
              keyType: string
              valueType: blob
      x-request-handler:
        - check_storage:
            request:
              method: get
              uri: /{domain}/sys/key_value/mathoid.{$.request.params.format}/{$.request.params.hash}
              headers:
                cache-control: '{{ cache-control }}'
            catch:
              status: 404
            return_if:
              status: '2xx'
            return:
              status: 200
              headers: "{{ merge({ 'cache-control': options.cache-control }, check_storage.headers) }}"
              body: '{{ check_storage.body }}'
        - postdata:
            request:
              uri: /{domain}/sys/mathoid/formula/{request.params.hash}
        - mathoid:
            request:
              method: post
              uri: /{domain}/sys/mathoid/render/{request.params.format}
              headers:
                content-type: application/json
                x-resource-location: '{{ request.params.hash }}'
              body: '{{postdata.body}}'
  • Apache proxy config : /etc/httpd/conf.d/restbase.conf
#
# Restbase
#

<VirtualHost *:80>
          ProxyPreserveHost On

          ProxyPass /your.some.domain/ http://localhost:7231/your.some.domain/
          ProxyPassReverse /your.some.domain/ http://localhost:7231/your.some.domain/

          ProxyPass /localhost/ http://localhost:7231/localhost/
          ProxyPassReverse /localhost/ http://localhost:7231/localhost/

          # MediaWiki is located here:
          DocumentRoot /var/www/html
 </VirtualHost>
  • VE section of my LocalSettings.php
wfLoadExtension( 'Math' );
$wgDefaultUserOptions['math'] = 'mathml';
...
wfLoadExtension( 'VisualEditor' );
$wgVisualEditorRestbaseURL = "http://My.Machine.Domain/localhost/v1/page/html/";
$wgVisualEditorFullRestbaseURL = "http://My.Machine.Domain/localhost/"; 
$wgVirtualRestConfig['modules']['parsoid'] = array(
	// URL to the Parsoid instance
	'url' => 'http://127.0.0.1:8000',
	// Parsoid "domain", see below (optional)
	'domain' => 'localhost',
);
...
Ianabc added a subscriber: Ianabc.EditedDec 1 2017, 5:23 AM

Unfortunately this seems to have stopped working again, I'm guessing something changed with restbase, but the configuration above gives me errors about missing routes

"status":404,"body":{"type":"not_found#route","title":"Not found.","internalURI":"/localhost/sys/table/mathoid.input","method":"put","depth":2}},"stack":"HTTPError: 404: not_found#route

Is mathoid still the recommended way of including math in wikimedia sites? This is really painful!

EDIT: Actually, scratch that, it seems to be working now. I was trying to use restbase and mathoid from git, but following the instructions to use the npm packages instead seemed to fix everything.

EDIT2: It looks like mathoid/restbase cooperate to produce the math, but then in browser I see a bunch of requests to localhost:7231/localhost which isn't going to work. /localhost will be proxied to restbase, but the connection localhost:7231 is going to be attempted on the client machine running the browser.

The configuration process is too complicated. There is now a new version of mathoid Math and MediaWiki core that does not require running mathoid as a service. Details will follow ... T188745

For really high-performance applications kubernetes is now used by Wikimedia. But the regular mathoid rendering should be good enough for most wikis.

Physikerwelt closed this task as Resolved.Mar 3 2018, 11:46 AM