Page MenuHomePhabricator

TitlePermissionTest failing on travis-ci after ContentLanguage service conversion
Closed, ResolvedPublicPRODUCTION ERROR

Description

1) TitlePermissionTest::testQuickPermissions
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     0 => Array (
         0 => 'badaccess-groups'
-        1 => '*, [[Traviswiki:Users|Users]]'
+        1 => '*, [[MyWiki:Users|Users]]'
/home/travis/build/wikimedia/mediawiki/tests/phpunit/includes/TitlePermissionTest.php:324
/home/travis/build/wikimedia/mediawiki/tests/phpunit/MediaWikiTestCase.php:475
/home/travis/build/wikimedia/mediawiki/maintenance/doMaintenance.php:94
2) TitlePermissionTest::testPageRestrictions
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     0 => Array (
         0 => 'badaccess-groups'
-        1 => '*, [[Traviswiki:Users|Users]]'
+        1 => '*, [[MyWiki:Users|Users]]'
/home/travis/build/wikimedia/mediawiki/tests/phpunit/includes/TitlePermissionTest.php:687
/home/travis/build/wikimedia/mediawiki/tests/phpunit/MediaWikiTestCase.php:475
/home/travis/build/wikimedia/mediawiki/maintenance/doMaintenance.php:94

https://travis-ci.org/wikimedia/mediawiki/jobs/414806103

Event Timeline

Legoktm created this task.Aug 12 2018, 6:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 12 2018, 6:59 AM

This looks related to the ContentLanguage service change, not the ParserFactory one. I can't reproduce the failure locally, but probably adding an overrideMwServices() to setUp() will fix the problem.

Change 452279 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/core@master] Call overrideMwServices() in TitlePermissionTest

https://gerrit.wikimedia.org/r/452279

Legoktm renamed this task from TitlePermissionTest failing on travis-ci after ParserFactory? service conversion to TitlePermissionTest failing on travis-ci after ContentLanguage service conversion .Aug 13 2018, 2:49 AM

Change 452279 merged by jenkins-bot:
[mediawiki/core@master] Call overrideMwServices() in TitlePermissionTest

https://gerrit.wikimedia.org/r/452279

@Legoktm @Simetrical Is there a commit we can revert meanwhile to unbreak the build? The lack of visibility has already allowed one regression to be lost in the noise - T201900.

Reedy triaged this task as High priority.Aug 15 2018, 3:39 PM
Reedy added a subscriber: Reedy.

@Legoktm @Simetrical Is there a commit we can revert meanwhile to unbreak the build? The lack of visibility has already allowed one regression to be lost in the noise - T201900.

I think it's going to be harder as there's been a few followup commits, including reordering ServiceWiring.php alphabetically

Change 453069 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] title: Disable the failing tests from TitlePermissionTest

https://gerrit.wikimedia.org/r/453069

Change 453069 merged by jenkins-bot:
[mediawiki/core@master] title: Disable the failing tests from TitlePermissionTest

https://gerrit.wikimedia.org/r/453069

Reedy lowered the priority of this task from High to Medium.Aug 16 2018, 1:52 PM
Reedy removed a project: Patch-For-Review.

Change 464139 had a related patch set uploaded (by simetrical; owner: simetrical):
[mediawiki/core@master] Re-enable tests from TitlePermissionTest

https://gerrit.wikimedia.org/r/464139

Change 464139 merged by jenkins-bot:
[mediawiki/core@master] Re-enable tests from TitlePermissionTest

https://gerrit.wikimedia.org/r/464139

Krinkle raised the priority of this task from Medium to High.Oct 5 2018, 8:18 PM

(This is breaking the Travis CI build again, and probably local tests as well when sitename != 'MyWiki')

Well, one solution would be to force the sitename to MyWiki for these tests, right? Or we could just leave them disabled. (Assuming nobody wants to figure out what the actual problem is, which isn't so easy if it can't be reproduced locally.)

Yes, tests that produce asserted values that vary by configuration should mock or set those configuration key explicitly. That's done in most other tests as well. I imagine it may've been avoided here if it's difficult to do so, but until we find away, the test should't be breaking CI for several days, so I've re-reverted it for now.

See also https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/464386/ / T206130, which triggered a similar build failure.

Krinkle added a subscriber: kostajh.Oct 9 2018, 1:23 AM

There is no way for every test to compile a list of all config options that might cause it to fail. There are far too many config options. The only solution is https://gerrit.wikimedia.org/r/451638.

I can actually reproduce locally, but only if I run all of includes/, not just if I run the file by itself.

I can reproduce by editing suite.xml to run only HtmlTest.php, TitleMethodsTest.php, and TitlePermissionTest.php (by adding <exclude> for everything else in includes/). Now working on narrowing down more.

In retrospect, grepping for "MyWiki" would have been a vastly quicker way to narrow down the problem.

Change 465423 had a related patch set uploaded (by simetrical; owner: simetrical):
[mediawiki/core@master] Reset Language/MWNamespace caches between tests

https://gerrit.wikimedia.org/r/465423

The fix is depressingly simple. We just needed to reset the Language cache in between test runs.

Change 465537 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] title: Disable flaky TitlePermission test cases

https://gerrit.wikimedia.org/r/465537

The fix is depressingly simple. We just needed to reset the Language cache in between test runs.

Yep, but doing it for everything has larger consequences and may break compat with existing tests as we found. Let's first unbreak the immediate test and improve the other tests that aren't broken, later :)

I've submitted https://gerrit.wikimedia.org/r/465537 which resets the site name for that test specifically. That's our convention elsewhere and is what should've been done in the first place.

Change 465537 abandoned by Krinkle:
title: Fix flaky TitlePermission test cases

Reason:
I don't know what's wrong with it. But I do know the bloody thing is more complicated than it should be.

https://gerrit.wikimedia.org/r/465537

Change 465537 restored by Krinkle:
title: Fix flaky TitlePermission test cases

https://gerrit.wikimedia.org/r/465537

Change 465537 abandoned by Krinkle:
title: Fix flaky TitlePermission test cases

Reason:
I give up. Something somewhere is messing with the global state in the shared job. Presumably in such a way that TestCase is unable to reset.

That probably means our resets are too brave by stashing and restoring, instead of actually clearing/resetting.

Anyhow, given we've spent 7 days and over a dozen commits trying to do this, I give up. Kosta has a patch that reduces the test to not needlessly assert what the wgSitename value is, which makes a ton of sense, given the test isn't about that at all. I'm merging that now.

We can continue pondering about why we can't have nice resets after that, including how to make them work with our existing problems in SimpleCaptcha, Flow, and Wikibase.

https://gerrit.wikimedia.org/r/465537

Change 465555 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@master] Fix TitlePermissionTest failures due to test leakage

https://gerrit.wikimedia.org/r/465555

Krinkle removed a subscriber: Krinkle.Oct 10 2018, 11:24 PM

Change 465555 merged by jenkins-bot:
[mediawiki/core@master] Fix TitlePermissionTest failures due to test leakage

https://gerrit.wikimedia.org/r/465555

kostajh added a subscriber: Krinkle.Nov 6 2018, 2:54 AM

@CCicalese_WMF the builds in Travis CI are passing again https://travis-ci.org/wikimedia/mediawiki/builds but not sure if @Krinkle wanted to do anything further here.

Krinkle closed this task as Resolved.Dec 3 2018, 3:23 AM
Krinkle claimed this task.

LGTM. Thanks!

Change 465423 abandoned by simetrical:
Reset Language/MWNamespace caches between tests

Reason:
This has been superseded by the LanguageFactory and NamespaceInfo patches.

https://gerrit.wikimedia.org/r/465423

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:09 PM

https://gerrit.wikimedia.org/r/c/mediawiki/core/+/634409/6 has refactored and restored the test. If this issue returns, we can just mark it as @group Broken again.