Page MenuHomePhabricator

New wikidiff2 version has broken ProofreadPage's PageSlotDiffRendererTest::testGetDiff and MW's REST API DiffCompare test
Closed, ResolvedPublic

Description

20:49:52 There were 2 failures:
20:49:52 
20:49:52 1) ProofreadPage\Page\PageSlotDiffRendererTest::testGetDiff with data set #1 (null, ProofreadPage\Page\PageContent Object (...), 'Page statusPage status-Not pr...ooter ')
20:49:52 Failed asserting that two strings are identical.
20:49:52 --- Expected
20:49:52 +++ Actual
20:49:52 @@ @@
20:49:52 -'Page statusPage status-Not proofread+ProblematicHeader (noinclude):Header (noinclude): + header Page body (to be transcluded):Page body (to be transcluded): + body2 Footer (noinclude):Footer (noinclude): + footer '
20:49:52 +'Page statusPage status-Not proofread+ProblematicHeader (noinclude):Header (noinclude): header Page body (to be transcluded):Page body (to be transcluded): body2 Footer (noinclude):Footer (noinclude): footer '
20:49:52 
20:49:52 /workspace/src/extensions/ProofreadPage/tests/phpunit/Page/PageSlotDiffRendererTest.php:52
20:49:52 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:452
20:49:52 === Logs generated by test case
20:49:52 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
20:49:52 [localisation] [debug] LocalisationCache using store LCStoreNull []
20:49:52 [MessageCache] [debug] MessageCache using store {class} {"class":"HashBagOStuff"}
20:49:52 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
20:49:52 [localisation] [debug] LocalisationCache using store LCStoreNull []
20:49:52 [ContentHandler] [info] Registered handler for wikitext: WikitextContentHandler []
20:49:52 [ContentHandler] [info] Registered handler for proofread-page: ProofreadPage\Page\PageContentHandler []
20:49:52 [GlobalTitleFail] [info] RequestContext::getTitle called with no title set. {"exception":{}}
20:49:52 [objectcache] [debug] MainObjectStash using store {class} {"class":"HashBagOStuff"}
20:49:52 [localisation] [debug] LocalisationCache::isExpired(en): cache missing, need to make one []
20:49:52 [wfDebug] [debug] ParserFactory: using default preprocessor {"private":false}
20:49:52 [localisation] [debug] LocalisationCache::isExpired(en): cache missing, need to make one []
20:49:52 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"thisWikiIsTheRepo","logValue":"true"}
20:49:52 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"entitySources","logValue":"array (\n  'local' => \n  array (\n    'entityNamespaces' => \n    array (\n      'item' => 120,\n      'property' => 122,\n      'mediainfo' => '6\/mediainfo',\n    ),\n    'repoDatabase' => false,\n    'baseUri' => 'http:\/\/127.0.0.1:9412\/entity\/',\n    'rdfNodeNamespacePrefix' => 'wd',\n    'rdfPredicateNamespacePrefix' => '',\n    'interwikiPrefix' => '',\n  ),\n)"}
20:49:52 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"entitySources","logValue":"array (\n  'local' => \n  array (\n    'repoDatabase' => false,\n    'entityNamespaces' => \n    array (\n      'item' => '120\/main',\n      'property' => '122\/main',\n      'mediainfo' => '6\/mediainfo',\n    ),\n    'baseUri' => 'http:\/\/127.0.0.1:9412\/entity\/',\n    'rdfNodeNamespacePrefix' => 'wd',\n    'rdfPredicateNamespacePrefix' => '',\n    'interwikiPrefix' => '',\n  ),\n)"}
20:49:52 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"itemAndPropertySourceName","logValue":"'local'"}
20:49:52 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"siteGlobalID","logValue":"'wikidb'"}
20:49:52 [wfDebug] [debug] DifferenceEngine old '0' new '0' rcid '0' {"private":false}
20:49:52 [wfDebug] [debug] DifferenceEngine old '0' new '0' rcid '0' {"private":false}
20:49:52 [wfDebug] [debug] DifferenceEngine old '0' new '0' rcid '0' {"private":false}
20:49:52 ===
20:49:52 
20:49:52 2) ProofreadPage\Page\PageSlotDiffRendererTest::testGetDiff with data set #2 (ProofreadPage\Page\PageContent Object (...), null, 'Page statusPage status-Proble...ooter ')
20:49:52 Failed asserting that two strings are identical.
20:49:52 --- Expected
20:49:52 +++ Actual
20:49:52 @@ @@
20:49:52 -'Page statusPage status-Problematic+Not proofreadHeader (noinclude):Header (noinclude): - header Page body (to be transcluded):Page body (to be transcluded): - body Footer (noinclude):Footer (noinclude): - footer '
20:49:52 +'Page statusPage status-Problematic+Not proofreadHeader (noinclude):Header (noinclude): header Page body (to be transcluded):Page body (to be transcluded): body Footer (noinclude):Footer (noinclude): footer '
20:49:52 
20:49:52 /workspace/src/extensions/ProofreadPage/tests/phpunit/Page/PageSlotDiffRendererTest.php:52
20:49:52 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:452
20:49:52 === Logs generated by test case
20:49:52 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
20:49:52 [localisation] [debug] LocalisationCache using store LCStoreNull []
20:49:52 [MessageCache] [debug] MessageCache using store {class} {"class":"HashBagOStuff"}
20:49:52 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
20:49:52 [localisation] [debug] LocalisationCache using store LCStoreNull []
20:49:52 [ContentHandler] [info] Registered handler for wikitext: WikitextContentHandler []
20:49:52 [ContentHandler] [info] Registered handler for proofread-page: ProofreadPage\Page\PageContentHandler []
20:49:52 [GlobalTitleFail] [info] RequestContext::getTitle called with no title set. {"exception":{}}
20:49:52 [objectcache] [debug] MainObjectStash using store {class} {"class":"HashBagOStuff"}
20:49:52 [localisation] [debug] LocalisationCache::isExpired(en): cache missing, need to make one []
20:49:52 [wfDebug] [debug] ParserFactory: using default preprocessor {"private":false}
20:49:52 [localisation] [debug] LocalisationCache::isExpired(en): cache missing, need to make one []
20:49:52 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"thisWikiIsTheRepo","logValue":"true"}
20:49:52 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"entitySources","logValue":"array (\n  'local' => \n  array (\n    'entityNamespaces' => \n    array (\n      'item' => 120,\n      'property' => 122,\n      'mediainfo' => '6\/mediainfo',\n    ),\n    'repoDatabase' => false,\n    'baseUri' => 'http:\/\/127.0.0.1:9412\/entity\/',\n    'rdfNodeNamespacePrefix' => 'wd',\n    'rdfPredicateNamespacePrefix' => '',\n    'interwikiPrefix' => '',\n  ),\n)"}
20:49:52 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"entitySources","logValue":"array (\n  'local' => \n  array (\n    'repoDatabase' => false,\n    'entityNamespaces' => \n    array (\n      'item' => '120\/main',\n      'property' => '122\/main',\n      'mediainfo' => '6\/mediainfo',\n    ),\n    'baseUri' => 'http:\/\/127.0.0.1:9412\/entity\/',\n    'rdfNodeNamespacePrefix' => 'wd',\n    'rdfPredicateNamespacePrefix' => '',\n    'interwikiPrefix' => '',\n  ),\n)"}
20:49:52 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"itemAndPropertySourceName","logValue":"'local'"}
20:49:52 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"siteGlobalID","logValue":"'wikidb'"}
20:49:52 [wfDebug] [debug] DifferenceEngine old '0' new '0' rcid '0' {"private":false}
20:49:52 [wfDebug] [debug] DifferenceEngine old '0' new '0' rcid '0' {"private":false}
20:49:52 [wfDebug] [debug] DifferenceEngine old '0' new '0' rcid '0' {"private":false}
20:49:52 ===
20:49:52 
20:49:52 FAILURES!
20:49:52 Tests: 24385, Assertions: 79577, Failures: 2, Skipped: 100.

Event Timeline

RhinosF1 triaged this task as Unbreak Now! priority.Oct 6 2021, 9:05 PM

It looks like CI for core and multiple extensions is failing. At least GrowthExperiments, TemplateData and VisualEditor, probably many more.

Zabe renamed this task from mediawiki/core CI fails with 'Failed asserting that two strings are identical.' to ProofreadPage\Page\PageSlotDiffRendererTest::testGetDif makes CI fail with 'Failed asserting that two strings are identical.'.Oct 6 2021, 9:07 PM
Zabe added a subscriber: matmarex.

Btw. there is a second failure, see https://gerrit.wikimedia.org/r/c/mediawiki/core/+/726953. Very likely happening for the same reason as the other error.

21:00:52   253 passing (48s)
21:00:52   1 pending
21:00:52   2 failing
21:00:52 
21:00:52   1) Diff Compare with Variables
21:00:52        should compare revisions 1 and 4:
21:00:52      AssertionError: expected '<tr>\n  <td colspan="2" class="diff-lineno">Line 1:</td>\n  <td colspan="2" class="diff-lineno">Line 1:</td>\n</tr>\n<tr>\n  <td class="diff-marker"></td>\n  <td class="diff-context diff-side-deleted"><div>Counting: </div></td>\n  <td class="diff-marker"></td>\n  <td class="diff-context diff-side-added"><div>Counting: </div></td>\n</tr>\n<tr>\n  <td class="diff-marker" data-marker="−"></td>\n  <td class="diff-deletedline diff-side-deleted"><div>* One</div></td>\n  <td class="diff-marker" data-marker="+"></td>\n  <td class="diff-addedline diff-side-added"><div>* One<ins class="diffchange diffchange-inline"> </ins></div></td>\n</tr>\n<tr>\n  <td colspan="2" class="diff-empty diff-side-deleted"></td>\n  <td class="diff-marker" data-marker="+"></td>\n  <td class="diff-addedline diff-side-added"><div>* Two</div></td>\n</tr>\n' to match /<td class=.diff-addedline.>.*\* Two/
21:00:52       at Context.it (tests/api-testing/action/DiffCompare.js:39:10)
21:00:52       at process._tickCallback (internal/process/next_tick.js:68:7)
21:00:52 
21:00:52   2) Diff Compare with Variables
21:00:52        should compare revisions 3 and 4:
21:00:52      AssertionError: expected '<tr>\n  <td colspan="2" class="diff-lineno">Line 1:</td>\n  <td colspan="2" class="diff-lineno">Line 1:</td>\n</tr>\n<tr>\n  <td class="diff-marker"></td>\n  <td class="diff-context diff-side-deleted"><div>Counting: </div></td>\n  <td class="diff-marker"></td>\n  <td class="diff-context diff-side-added"><div>Counting: </div></td>\n</tr>\n<tr>\n  <td class="diff-marker"></td>\n  <td class="diff-context diff-side-deleted"><div>* One </div></td>\n  <td class="diff-marker"></td>\n  <td class="diff-context diff-side-added"><div>* One </div></td>\n</tr>\n<tr>\n  <td class="diff-marker" data-marker="−"></td>\n  <td class="diff-deletedline diff-side-deleted"><div>* Two<del class="diffchange diffchange-inline"> </del></div></td>\n  <td class="diff-marker" data-marker="+"></td>\n  <td class="diff-addedline diff-side-added"><div>* Two</div></td>\n</tr>\n<tr>\n  <td class="diff-marker" data-marker="−"></td>\n  <td class="diff-deletedline diff-side-deleted"><div>* Three</div></td>\n  <td colspan="2" class="diff-empty diff-side-added"></td>\n</tr>\n' to match /<td class=.diff-deletedline.>.*\* Three/
21:00:52       at Context.it (tests/api-testing/action/DiffCompare.js:45:10)
21:00:52       at process._tickCallback (internal/process/next_tick.js:68:7)
Krinkle renamed this task from ProofreadPage\Page\PageSlotDiffRendererTest::testGetDif makes CI fail with 'Failed asserting that two strings are identical.' to ProofreadPage\Page\PageSlotDiffRendererTest::testGetDiff makes CI fail with 'Failed asserting that two strings are identical.'.Oct 6 2021, 9:26 PM
Krinkle subscribed.

The most likely cause is the addition of diff-side-deleted which made the (fragile) regex no longer match. Ref T285956, T285857, which just released the relevant change to php-wikidiff2 to Beta and thus likely CI as well.

I think in this case it's a fair failure and the test should be fixed to be a bit more permissive.

Currently:

input: '… <td class="diff-context diff-side-deleted"> … '
regex: /<td class=.diff-deletedline.>.*\* Three/

For example: /<td [^>]*class=[^>]*diff-deletedline[^>]*>.*\* Three/

Jdforrester-WMF renamed this task from ProofreadPage\Page\PageSlotDiffRendererTest::testGetDiff makes CI fail with 'Failed asserting that two strings are identical.' to New wikidiff2 version has broken ProofreadPage's PageSlotDiffRendererTest::testGetDiff as it's doing an exact string match.Oct 6 2021, 10:14 PM
Jdforrester-WMF added a project: wikidiff2.

Change 726987 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/ProofreadPage@master] PageSlotDiffRendererTest::testGetDiff: Skip as new wikidiff2 breaks this test

https://gerrit.wikimedia.org/r/726987

Change 726989 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/core@master] api-testing: Adjust DiffCompare expected outcome to cope with new wikidiff2 output

https://gerrit.wikimedia.org/r/726989

Hmm, yes, MW's own REST API tests are using a regex test but assuming only the single class on its TD element.

Yes, it'll fail every patch in the world right now, hence UBN. :-)

Jdforrester-WMF renamed this task from New wikidiff2 version has broken ProofreadPage's PageSlotDiffRendererTest::testGetDiff as it's doing an exact string match to New wikidiff2 version has broken ProofreadPage's PageSlotDiffRendererTest::testGetDiff and MW's REST API DiffCompare test.Oct 6 2021, 10:34 PM

Change 726987 merged by jenkins-bot:

[mediawiki/extensions/ProofreadPage@master] PageSlotDiffRendererTest::testGetDiff: Skip as new wikidiff2 breaks this test

https://gerrit.wikimedia.org/r/726987

Change 726989 merged by jenkins-bot:

[mediawiki/core@master] api-testing: Adjust DiffCompare expected outcome to cope with new wikidiff2 output

https://gerrit.wikimedia.org/r/726989

Jdforrester-WMF claimed this task.

OK, this should now be resolved. If any other extension or other repo's test suite is broken, please comment here or open a new task.

Btw. there is a second failure, see https://gerrit.wikimedia.org/r/c/mediawiki/core/+/726953. Very likely happening for the same reason as the other error.

This is a different failure that I think requires a different fix. Appears to come from Core's api-testing/action/DiffCompare.js

sorry I see https://gerrit.wikimedia.org/r/c/mediawiki/core/+/726989/ now :)

Change 726959 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/ProofreadPage@wmf/1.38.0-wmf.3] PageSlotDiffRendererTest::testGetDiff: Skip as new wikidiff2 breaks this test

https://gerrit.wikimedia.org/r/726959

Change 726960 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/ProofreadPage@wmf/1.38.0-wmf.2] PageSlotDiffRendererTest::testGetDiff: Skip as new wikidiff2 breaks this test

https://gerrit.wikimedia.org/r/726960

Change 726961 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/core@wmf/1.38.0-wmf.3] api-testing: Adjust DiffCompare expected outcome to cope with new wikidiff2 output

https://gerrit.wikimedia.org/r/726961

Change 726962 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/core@wmf/1.38.0-wmf.2] api-testing: Adjust DiffCompare expected outcome to cope with new wikidiff2 output

https://gerrit.wikimedia.org/r/726962

Change 726959 merged by jenkins-bot:

[mediawiki/extensions/ProofreadPage@wmf/1.38.0-wmf.3] PageSlotDiffRendererTest::testGetDiff: Skip as new wikidiff2 breaks this test

https://gerrit.wikimedia.org/r/726959

Change 726960 merged by Ladsgroup:

[mediawiki/extensions/ProofreadPage@wmf/1.38.0-wmf.2] PageSlotDiffRendererTest::testGetDiff: Skip as new wikidiff2 breaks this test

https://gerrit.wikimedia.org/r/726960

Change 726961 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.3] api-testing: Adjust DiffCompare expected outcome to cope with new wikidiff2 output

https://gerrit.wikimedia.org/r/726961

Change 726962 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.2] api-testing: Adjust DiffCompare expected outcome to cope with new wikidiff2 output

https://gerrit.wikimedia.org/r/726962

Note that the ProofreadPage test failure was not a broken test, it detected actual breakage. ProofreadPage stores multiple semi-structured parts of a page using noinclude sections (it predates MCR and other relevant facilities), and then hides these from users by hooking into the diff.

See this page: open it for editing and hit "Show changes". The displayed diff should be empty, but isn't.

Note that the ProofreadPage test failure was not a broken test, it detected actual breakage. ProofreadPage stores multiple semi-structured parts of a page using noinclude sections (it predates MCR and other relevant facilities), and then hides these from users by hooking into the diff.

The new version of wikidiff2 only changes the HTML output, and nothing is expected to break, unless somebody is parsing the resulting HTML with 0 flexibility, which is something that shouldn't happen and is error-prone anyway.

See this page: open it for editing and hit "Show changes". The displayed diff should be empty, but isn't.

Cannot reproduce, the diff is empty for me, as it should be.

See this page: open it for editing and hit "Show changes". The displayed diff should be empty, but isn't.

Cannot reproduce, the diff is empty for me, as it should be.

Oh, interesting. Doesn't happen when logged out, so clearly something more or something else is going on. That particular failure at this particular time is kinda suspicious though.

See this page: open it for editing and hit "Show changes". The displayed diff should be empty, but isn't.

Cannot reproduce, the diff is empty for me, as it should be.

Ah. It seems to be dependent on the Editing → Show previews without reloading the page setting being turned on in Preferences. When that pref is on it shows:

2021-10-11_160516_1464x758_screenshot.png (758×1 px, 248 KB)

When it's off I get a clean diff.

Krinkle closed this task as Resolved.
Krinkle lowered the priority of this task from Unbreak Now! to High.
Zabe raised the priority of this task from High to Unbreak Now!.Oct 11 2021, 5:53 PM

Change 884331 had a related patch set uploaded (by Umherirrender; author: Jforrester):

[mediawiki/extensions/ProofreadPage@REL1_35] PageSlotDiffRendererTest::testGetDiff: Skip as new wikidiff2 breaks this test

https://gerrit.wikimedia.org/r/884331

Change 884331 merged by jenkins-bot:

[mediawiki/extensions/ProofreadPage@REL1_35] PageSlotDiffRendererTest::testGetDiff: Skip as new wikidiff2 breaks this test

https://gerrit.wikimedia.org/r/884331