Page MenuHomePhabricator
Create Task
Maniphest T314823

Math test failures in 1.35
Closed, ResolvedPublic
Actions

Description

Seen on a no-op on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Math/+/821331; https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php73-noselenium-docker/32664/console

Tested after https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Math/+/820883 was failing

22:52:06 There were 7 failures:
22:52:06 
22:52:06 1) MathCoverageTest::testCoverage with data set #157 ('\oint_{(x,y)\in C} x^3\, dx +...2\, dy', array(), '<img src="9bd8f3ee582e64365ce...dy}"/>')
22:52:06 Failed to render \oint_{(x,y)\in C} x^3\, dx + 4y^2\, dy
22:52:06 Failed asserting that two strings are equal.
22:52:06 --- Expected
22:52:06 +++ Actual
22:52:06 @@ @@
22:52:06 -'<img src="9bd8f3ee582e64365cefbfd44860b60a14b4424b" class="mwe-math-fallback-image-inline" aria-hidden="true" style="vertical-align: -2.838ex; width:24.461ex; height:6.343ex;" alt="{\displaystyle \oint_{(x,y)\in C} x^3\, dx + 4y^2\, dy}"/>'
22:52:06 +'<img src="9bd8f3ee582e64365cefbfd44860b60a14b4424b" class="mwe-math-fallback-image-inline" aria-hidden="true" style="vertical-align: -2.671ex; width:22.044ex; height:6.009ex;" alt="{\displaystyle \oint_{(x,y)\in C} x^3\, dx + 4y^2\, dy}"/>'
22:52:06 
22:52:06 /workspace/src/extensions/Math/tests/phpunit/MathCoverageTest.php:40
22:52:06 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:446
22:52:06 /workspace/src/maintenance/doMaintenance.php:107
22:52:06 === Logs generated by test case
22:52:06 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
22:52:06 [Math] [debug] Start rendering ${\displaystyle \oint_{(x,y)\in C} x^3\, dx + 4y^2\, dy}$ in mode png []
22:52:06 [http] [debug] POST: https://wikimedia.org/api/rest_v1/media/math/check/tex []
22:52:06 [http] [debug] GET: https://wikimedia.org/api/rest_v1/media/math/render/mml/9bd8f3ee582e64365cefbfd44860b60a14b4424b []
22:52:06 ===
22:52:06 
22:52:06 2) MathCoverageTest::testCoverage with data set #361 ('\oint', array(), '<img src="dbbea2707a4ee041ddc...nt}"/>')
22:52:06 Failed to render \oint
22:52:06 Failed asserting that two strings are equal.
22:52:06 --- Expected
22:52:06 +++ Actual
22:52:06 @@ @@
22:52:06 -'<img src="dbbea2707a4ee041ddc4da8dd11924d25aa59c89" class="mwe-math-fallback-image-inline" aria-hidden="true" style="vertical-align: -2.338ex; width:3.709ex; height:5.843ex;" alt="{\displaystyle \oint}"/>'
22:52:06 +'<img src="dbbea2707a4ee041ddc4da8dd11924d25aa59c89" class="mwe-math-fallback-image-inline" aria-hidden="true" style="vertical-align: -2.338ex; width:2.194ex; height:5.676ex;" alt="{\displaystyle \oint}"/>'
22:52:06 
22:52:06 /workspace/src/extensions/Math/tests/phpunit/MathCoverageTest.php:40
22:52:06 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:446
22:52:06 /workspace/src/maintenance/doMaintenance.php:107
22:52:06 === Logs generated by test case
22:52:06 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
22:52:06 [Math] [debug] Start rendering ${\displaystyle \oint}$ in mode png []
22:52:06 [http] [debug] POST: https://wikimedia.org/api/rest_v1/media/math/check/tex []
22:52:06 [http] [debug] GET: https://wikimedia.org/api/rest_v1/media/math/render/mml/dbbea2707a4ee041ddc4da8dd11924d25aa59c89 []
22:52:06 ===
22:52:06 
22:52:06 3) MathCoverageTest::testCoverage with data set #367 ('( \nabla \times \bold{F} ) \c...\ell} ', array(), '<img src="561ae6d178581f8938a...} }"/>')
22:52:06 Failed to render ( \nabla \times \bold{F} ) \cdot {\rm d}\bold{S} = \oint_{\partial S} \bold{F} \cdot {\rm d}\boldsymbol{\ell} 
22:52:06 Failed asserting that two strings are equal.
22:52:06 --- Expected
22:52:06 +++ Actual
22:52:06 @@ @@
22:52:06 -'<img src="561ae6d178581f8938ae976090135ee66a81d801" class="mwe-math-fallback-image-inline" aria-hidden="true" style="vertical-align: -2.505ex; width:27.901ex; height:6.009ex;" alt="{\displaystyle ( \nabla \times \bold{F} ) \cdot {\rm d}\bold{S} = \oint_{\partial S} \bold{F} \cdot {\rm d}\boldsymbol{\ell} }"/>'
22:52:06 +'<img src="561ae6d178581f8938ae976090135ee66a81d801" class="mwe-math-fallback-image-inline" aria-hidden="true" style="vertical-align: -2.338ex; width:25.484ex; height:5.676ex;" alt="{\displaystyle ( \nabla \times \bold{F} ) \cdot {\rm d}\bold{S} = \oint_{\partial S} \bold{F} \cdot {\rm d}\boldsymbol{\ell} }"/>'
22:52:06 
22:52:06 /workspace/src/extensions/Math/tests/phpunit/MathCoverageTest.php:40
22:52:06 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:446
22:52:06 /workspace/src/maintenance/doMaintenance.php:107
22:52:06 === Logs generated by test case
22:52:06 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
22:52:06 [Math] [debug] Start rendering ${\displaystyle ( \nabla \times \bold{F} ) \cdot {\rm d}\bold{S} = \oint_{\partial S} \bold{F} \cdot {\rm d}\boldsymbol{\ell} }$ in mode png []
22:52:06 [http] [debug] POST: https://wikimedia.org/api/rest_v1/media/math/check/tex []
22:52:06 [http] [debug] GET: https://wikimedia.org/api/rest_v1/media/math/render/mml/561ae6d178581f8938ae976090135ee66a81d801 []
22:52:06 ===
22:52:06 
22:52:06 4) MathCoverageTest::testCoverage with data set #369 ('( \nabla \times \bold{F} ) \c...\ell} ', array(), '<img src="561ae6d178581f8938a...} }"/>')
22:52:06 Failed to render ( \nabla \times \bold{F} ) \cdot {\rm d}\bold{S} = \oint_{\partial S} \bold{F} \cdot {\rm d}\boldsymbol{\ell} 
22:52:06 Failed asserting that two strings are equal.
22:52:06 --- Expected
22:52:06 +++ Actual
22:52:06 @@ @@
22:52:06 -'<img src="561ae6d178581f8938ae976090135ee66a81d801" class="mwe-math-fallback-image-inline" aria-hidden="true" style="vertical-align: -2.505ex; width:27.901ex; height:6.009ex;" alt="{\displaystyle ( \nabla \times \bold{F} ) \cdot {\rm d}\bold{S} = \oint_{\partial S} \bold{F} \cdot {\rm d}\boldsymbol{\ell} }"/>'
22:52:06 +'<img src="561ae6d178581f8938ae976090135ee66a81d801" class="mwe-math-fallback-image-inline" aria-hidden="true" style="vertical-align: -2.338ex; width:25.484ex; height:5.676ex;" alt="{\displaystyle ( \nabla \times \bold{F} ) \cdot {\rm d}\bold{S} = \oint_{\partial S} \bold{F} \cdot {\rm d}\boldsymbol{\ell} }"/>'
22:52:06 
22:52:06 /workspace/src/extensions/Math/tests/phpunit/MathCoverageTest.php:40
22:52:06 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:446
22:52:06 /workspace/src/maintenance/doMaintenance.php:107
22:52:06 === Logs generated by test case
22:52:06 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
22:52:06 [Math] [debug] Start rendering ${\displaystyle ( \nabla \times \bold{F} ) \cdot {\rm d}\bold{S} = \oint_{\partial S} \bold{F} \cdot {\rm d}\boldsymbol{\ell} }$ in mode png []
22:52:06 [http] [debug] POST: https://wikimedia.org/api/rest_v1/media/math/check/tex []
22:52:06 [http] [debug] GET: https://wikimedia.org/api/rest_v1/media/math/render/mml/561ae6d178581f8938ae976090135ee66a81d801 []
22:52:06 ===
22:52:06 
22:52:06 5) MathCoverageTest::testCoverage with data set #370 ('\oint_C \bold{B} \cdot {\rm d...\mu_0 ', array(), '<img src="e7acad6ef86a9151e97...0 }"/>')
22:52:06 Failed to render \oint_C \bold{B} \cdot {\rm d} \boldsymbol{\ell} = \mu_0 
22:52:06 Failed asserting that two strings are equal.
22:52:06 --- Expected
22:52:06 +++ Actual
22:52:06 @@ @@
22:52:06 -'<img src="e7acad6ef86a9151e97ac0617cfc752a9f092de1" class="mwe-math-fallback-image-inline" aria-hidden="true" style="vertical-align: -2.505ex; width:17.107ex; height:6.009ex;" alt="{\displaystyle \oint_C \bold{B} \cdot {\rm d} \boldsymbol{\ell} = \mu_0 }"/>'
22:52:06 +'<img src="e7acad6ef86a9151e97ac0617cfc752a9f092de1" class="mwe-math-fallback-image-inline" aria-hidden="true" style="vertical-align: -2.338ex; width:14.69ex; height:5.676ex;" alt="{\displaystyle \oint_C \bold{B} \cdot {\rm d} \boldsymbol{\ell} = \mu_0 }"/>'
22:52:06 
22:52:06 /workspace/src/extensions/Math/tests/phpunit/MathCoverageTest.php:40
22:52:06 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:446
22:52:06 /workspace/src/maintenance/doMaintenance.php:107
22:52:06 === Logs generated by test case
22:52:06 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
22:52:06 [Math] [debug] Start rendering ${\displaystyle \oint_C \bold{B} \cdot {\rm d} \boldsymbol{\ell} = \mu_0 }$ in mode png []
22:52:06 [http] [debug] POST: https://wikimedia.org/api/rest_v1/media/math/check/tex []
22:52:06 [http] [debug] GET: https://wikimedia.org/api/rest_v1/media/math/render/mml/e7acad6ef86a9151e97ac0617cfc752a9f092de1 []
22:52:06 ===
22:52:06 
22:52:06 6) MathCoverageTest::testCoverage with data set #455 ('', array(), '<strong class="error texerror...ong>\n')
22:52:06 Failed to render 
22:52:06 Failed asserting that two strings are equal.
22:52:06 --- Expected
22:52:06 +++ Actual
22:52:06 @@ @@
22:52:06 -'<strong class="error texerror">Failed to parse (Conversion error. Server (&quot;https://wikimedia.org/api/rest_&quot;) reported: &quot;Cannot get mml. Expected width &gt; 0.&quot;): {\displaystyle }</strong>\n
22:52:06 -'
22:52:06 +'<img src="df4dcd61276328f7c7ec5bdc399b6e11114a2c68" class="mwe-math-fallback-image-inline" aria-hidden="true" style="vertical-align: -0.171ex; width:0; height:0.343ex;" alt="{\displaystyle }"/>'
22:52:06 
22:52:06 /workspace/src/extensions/Math/tests/phpunit/MathCoverageTest.php:40
22:52:06 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:446
22:52:06 /workspace/src/maintenance/doMaintenance.php:107
22:52:06 === Logs generated by test case
22:52:06 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
22:52:06 [Math] [debug] Start rendering ${\displaystyle }$ in mode png []
22:52:06 [http] [debug] POST: https://wikimedia.org/api/rest_v1/media/math/check/tex []
22:52:06 [http] [debug] GET: https://wikimedia.org/api/rest_v1/media/math/render/mml/df4dcd61276328f7c7ec5bdc399b6e11114a2c68 []
22:52:06 ===
22:52:06 
22:52:06 7) MathCoverageTest::testCoverage with data set #456 (' ', array(), '<strong class="error texerror...ong>\n')
22:52:06 Failed to render  
22:52:06 Failed asserting that two strings are equal.
22:52:06 --- Expected
22:52:06 +++ Actual
22:52:06 @@ @@
22:52:06 -'<strong class="error texerror">Failed to parse (Conversion error. Server (&quot;https://wikimedia.org/api/rest_&quot;) reported: &quot;Cannot get mml. Expected width &gt; 0.&quot;): {\displaystyle  }</strong>\n
22:52:06 -'
22:52:06 +'<img src="df4dcd61276328f7c7ec5bdc399b6e11114a2c68" class="mwe-math-fallback-image-inline" aria-hidden="true" style="vertical-align: -0.171ex; width:0; height:0.343ex;" alt="{\displaystyle  }"/>'
22:52:06 
22:52:06 /workspace/src/extensions/Math/tests/phpunit/MathCoverageTest.php:40
22:52:06 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:446
22:52:06 /workspace/src/maintenance/doMaintenance.php:107
22:52:06 === Logs generated by test case
22:52:06 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
22:52:06 [Math] [debug] Start rendering ${\displaystyle  }$ in mode png []
22:52:06 [http] [debug] POST: https://wikimedia.org/api/rest_v1/media/math/check/tex []
22:52:06 [http] [debug] GET: https://wikimedia.org/api/rest_v1/media/math/render/mml/df4dcd61276328f7c7ec5bdc399b6e11114a2c68 []

Details

SubjectRepoBranchLines +/-
Skip some erronous tests in MathCoverageTestmediawiki/extensions/MathREL1_35+18 -8
Customize query in gerrit

Event Timeline

Reedy created this task.Aug 8 2022, 9:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 8 2022, 9:53 PM
Reedy updated the task description. (Show Details)Aug 8 2022, 9:53 PM
Reedy moved this task from Blocker to Not a blocker on the MW-1.35-release board.Aug 8 2022, 10:10 PM
Reedy triaged this task as High priority.Nov 8 2022, 6:36 PM
Physikerwelt subscribed.Nov 8 2022, 7:38 PM

The most reasonable thing is probably to skip the failing tests. What do you think? We removed integration tests in later versions and are currently implementing the entire rendering process in pure php without calling third party libraries or using external binaries.

Reedy added a comment.Nov 8 2022, 8:32 PM

Seems fine to me!

Physikerwelt assigned this task to Stegmujo.Nov 9 2022, 9:37 AM
Physikerwelt added a subscriber: Stegmujo.

@Stegmujo can you disable the failing tests please?

gerritbot added a comment.Nov 22 2022, 10:46 AM

Change 859454 had a related patch set uploaded (by Stegmujo; author: Stegmujo):

[mediawiki/extensions/Math@REL1_35] Skip tests in coverage

https://gerrit.wikimedia.org/r/859454

gerritbot added a project: Patch-For-Review.Nov 22 2022, 10:46 AM
gerritbot added a comment.Nov 22 2022, 3:02 PM

Change 859454 merged by Stegmujo:

[mediawiki/extensions/Math@REL1_35] Skip some erronous tests in MathCoverageTest

https://gerrit.wikimedia.org/r/859454

Maintenance_bot removed a project: Patch-For-Review.Nov 22 2022, 3:30 PM
Physikerwelt added a comment.Nov 23 2022, 3:50 PM

@Reedy does this work now? Can you resolve, if tests are fixed?

Reedy added a comment.Jan 8 2023, 11:59 PM
In T314823#8417606, @Physikerwelt wrote:

@Reedy does this work now? Can you resolve, if tests are fixed?

It's not easy to tell when https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Math/+/859454/ wasn't merged correctly (ie CI wasn't left to merge it).

Reedy added a comment.Jan 9 2023, 12:16 AM

On a no-op change to Math on REL1_35 - https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Math/+/876304

https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php73-noselenium-docker/37966/console

00:02:44 There was 1 failure:
00:02:44 
00:02:44 1) MathRestbaseInterfaceTest::testFail
00:02:44 Failed asserting that two strings are equal.
00:02:44 --- Expected
00:02:44 +++ Actual
00:02:44 @@ @@
00:02:44 -'Illegal TeX function'
00:02:44 +'Math extension cannot connect to Restbase.'
00:02:44 
00:02:44 /workspace/src/extensions/Math/tests/phpunit/MathRestbaseInterfaceTest.php:56
00:02:44 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:446
00:02:44 /workspace/src/maintenance/doMaintenance.php:107

https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php74-noselenium-docker/30641/console

00:06:24 There was 1 failure:
00:06:24 
00:06:24 1) MathInputCheckRestbaseTest::testGetError
00:06:24 Failed asserting that '<strong class="error texerror">Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response (&quot;Math extension cannot connect to Restbase.&quot;) from server &quot;https://wikimedia.org/api/rest_v1/&quot;:): \newcommand{\text{do evil things}}</strong>\n
00:06:24 ' contains "unknown function &quot;\newcommand&quot;".
00:06:24 
00:06:24 /workspace/src/extensions/Math/tests/phpunit/MathInputCheckRestbaseTest.php:47
00:06:24 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:446
00:06:24 /workspace/src/maintenance/doMaintenance.php:107

I saw Wikibase related failures on some of the tests... But that obviously isn't the fault of the Math extension.

Physikerwelt added a comment.Jan 28 2023, 9:18 AM

^^The problem might be that CI tests are (no longer) allowed to call external services. Maybe this was enforced in the meantime. So all we can do is mark this test as skipped.

Stegmujo closed this task as Resolved.Nov 30 2023, 1:26 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL