I used the rest interface to get closer to the origin of the problem:
Formulae are returned as

</binding>
           <binding name='formula'>
                       <literal datatype='http://www.w3.org/1998/Math/MathML'>>&lt;math xmlns="http://www.w3.org/1998/Math/MathML" display="block"&gt;

which does not seem reasonable.
Where does the literal, and with the datatype attribute come from?
Is there a chance to fix that within the math extension?

Maybe the relevant code is https://github.com/wikimedia/mediawiki-extensions-Math/blob/master/MathMLRdfBuilder.php#L33

		$writer->say( $propertyValueNamespace, $propertyValueLName )
			->value( $mml, 'http://www.w3.org/1998/Math/MathML' );

Is there a vagrant role that creates an Wikidata Query Service for testing?

Can it be that the title is misleading?
You want the math to be rendered in the table and other?

This is the place where you could change the rendering:
https://github.com/wikimedia/wikidata-query-gui/blob/master/wikibase/queryService/ui/resultBrowser/helper/FormatterHelper.js#L69

@Jonas thank you. After your comment, that was very helpful for me, it turns out that the title is misleading;-)

And yes. Math should be rendered in that table.

For the implementation I would add

var DATATYPE_MATHML = 'http://www.w3.org/1998/Math/MathML';

extend the case statement with

case DATATYPE_MATHML:
 html = data;
 break;

Is it really that simple?

Yes, this should work :)
Thank you for your effort!

Doesn't html need to be a JQuery object? data is only string.

https://github.com/wikimedia/wikidata-query-gui/pull/2

@Jonas do you have an idea how to test this change?
I have set up a vagrant test instance at wmflabs
http://wikidata-math-de.wmflabs.org/wiki/Q15
But the information at https://github.com/wikimedia/wikidata-query-gui is quite brief;-)

Thanks for your pull request, but this is just a mirror, please submit your patch here:
https://gerrit.wikimedia.org/r/#/admin/projects/wikidata/query/gui

Just checkout the repo and open index.html that's it.

Change 300055 had a related patch set uploaded (by Physikerwelt):
Display MathML in the table view

https://gerrit.wikimedia.org/r/300055

^^ the patch above works for me

Change 300055 merged by jenkins-bot:
Display MathML in the table view

https://gerrit.wikimedia.org/r/300055

We have a test instance at wdqs-test.wikidata.org, where we could test things. It runs on Wikidata data so should be good for testing.

There's also role::wdqs role which allows to deploy in labs, but there are a bunch of manual steps - you have to check out WDQS deploy repo manually (scap doesn't work that well in labs IIRC) and then manually load data. Ping me if you need help with that, it's not hard but needs some manual steps now.

Change 300074 had a related patch set uploaded (by Smalyshev):
Merging from 3df1df468a66a22996e4b75d4826f81056997647:

https://gerrit.wikimedia.org/r/300074

Change 300074 merged by Smalyshev:
Merging from 3df1df468a66a22996e4b75d4826f81056997647:

https://gerrit.wikimedia.org/r/300074

I've deployed latest master on wdqs-test, but it doesn't seem to produce any difference:

http://wdqs-test.wmflabs.org/#PREFIX%20wdt%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fprop%2Fdirect%2F%3E%0APREFIX%20wikibase%3A%20%3Chttp%3A%2F%2Fwikiba.se%2Fontology%23%3E%0A%0ASELECT%20%3Fp%20%3FpLabel%20%3Fw%20%3FwLabel%20WHERE%20%7B%0A%20%20%20%3Fp%20wdt%3AP2534%20%3Fw%20.%0A%20%20%20SERVICE%20wikibase%3Alabel%20%7B%0A%20%20%20%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22%20.%0A%20%20%20%7D%0A%20%7D%20LIMIT%2010

@Jonas, could you check what's going on?

In T137784#2481327, @Smalyshev wrote:

I've deployed latest master on wdqs-test, but it doesn't seem to produce any difference:

http://wdqs-test.wmflabs.org/#PREFIX%20wdt%3A%20%3Chttp%3A%2F%2Fwww.wikidata.org%2Fprop%2Fdirect%2F%3E%0APREFIX%20wikibase%3A%20%3Chttp%3A%2F%2Fwikiba.se%2Fontology%23%3E%0A%0ASELECT%20%3Fp%20%3FpLabel%20%3Fw%20%3FwLabel%20WHERE%20%7B%0A%20%20%20%3Fp%20wdt%3AP2534%20%3Fw%20.%0A%20%20%20SERVICE%20wikibase%3Alabel%20%7B%0A%20%20%20%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22%20.%0A%20%20%20%7D%0A%20%7D%20LIMIT%2010

@Jonas, could you check what's going on?

I can actually see difference. The w column on the test server gives me rendered math formulas, while the production server gives mathml code.

Oh, cool. Looks like it's my browser that is missing math... I don't see it in Chrome, but Safari works just fine. Great then. Will be deployed on production next deployment window (Monday).

Unfortunately only works in some browsers:
http://caniuse.com/#search=mathml

On Wikidata.org we have png fallback for this, I am not sure if it is necessary here.

It's still an improvement for some browsers, but a PNG fallback would be nice I think.

what about using KaTeX or MathJax instead of PNG fallback ?
https://github.com/Khan/KaTeX
https://www.mathjax.org/