Page MenuHomePhabricator
Create Task
Maniphest T201776

TitlePermissionTest failing on travis-ci after ContentLanguage service conversion
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

1) TitlePermissionTest::testQuickPermissions
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     0 => Array (
         0 => 'badaccess-groups'
-        1 => '*, [[Traviswiki:Users|Users]]'
+        1 => '*, [[MyWiki:Users|Users]]'
/home/travis/build/wikimedia/mediawiki/tests/phpunit/includes/TitlePermissionTest.php:324
/home/travis/build/wikimedia/mediawiki/tests/phpunit/MediaWikiTestCase.php:475
/home/travis/build/wikimedia/mediawiki/maintenance/doMaintenance.php:94
2) TitlePermissionTest::testPageRestrictions
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     0 => Array (
         0 => 'badaccess-groups'
-        1 => '*, [[Traviswiki:Users|Users]]'
+        1 => '*, [[MyWiki:Users|Users]]'
/home/travis/build/wikimedia/mediawiki/tests/phpunit/includes/TitlePermissionTest.php:687
/home/travis/build/wikimedia/mediawiki/tests/phpunit/MediaWikiTestCase.php:475
/home/travis/build/wikimedia/mediawiki/maintenance/doMaintenance.php:94

https://travis-ci.org/wikimedia/mediawiki/jobs/414806103

Details

SubjectRepoBranchLines +/-
Reset Language/MWNamespace caches between testsmediawiki/coremaster+2 -6
Fix TitlePermissionTest failures due to test leakagemediawiki/coremaster+3 -0
title: Fix flaky TitlePermission test casesmediawiki/coremaster+22 -40
Re-enable tests from TitlePermissionTestmediawiki/coremaster+0 -8
title: Disable the failing tests from TitlePermissionTestmediawiki/coremaster+8 -0
Call overrideMwServices() in TitlePermissionTestmediawiki/coremaster+1 -0
Customize query in gerrit

Related Objects

Event Timeline

Legoktm created this task.Aug 12 2018, 6:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 12 2018, 6:59 AM
Simetrical added a comment.Aug 12 2018, 7:43 AM

This looks related to the ContentLanguage service change, not the ParserFactory one. I can't reproduce the failure locally, but probably adding an overrideMwServices() to setUp() will fix the problem.

gerritbot subscribed.Aug 13 2018, 2:46 AM

Change 452279 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/core@master] Call overrideMwServices() in TitlePermissionTest

https://gerrit.wikimedia.org/r/452279

gerritbot added a project: Patch-For-Review.Aug 13 2018, 2:46 AM
Legoktm renamed this task from TitlePermissionTest failing on travis-ci after ParserFactory? service conversion to TitlePermissionTest failing on travis-ci after ContentLanguage service conversion .Aug 13 2018, 2:49 AM
gerritbot added a comment.Aug 13 2018, 4:45 PM

Change 452279 merged by jenkins-bot:
[mediawiki/core@master] Call overrideMwServices() in TitlePermissionTest

https://gerrit.wikimedia.org/r/452279

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-08-14 (1.32.0-wmf.17)).Aug 13 2018, 5:00 PM
Krinkle removed projects: MW-1.32-notes (WMF-deploy-2018-08-14 (1.32.0-wmf.17)), Patch-For-Review.Aug 13 2018, 6:50 PM

@Simetrical Unfortunately, it is still failing – https://travis-ci.org/wikimedia/mediawiki/builds/415538025.

Krinkle moved this task from Untriaged to In progress on the Wikimedia-production-error (ARCHIVED -- Shared Build Failure) board.Aug 13 2018, 6:52 PM
Krinkle added a comment.Aug 14 2018, 9:41 PM

@Legoktm @Simetrical Is there a commit we can revert meanwhile to unbreak the build? The lack of visibility has already allowed one regression to be lost in the noise - T201900.

Reedy triaged this task as High priority.Aug 15 2018, 3:39 PM
Reedy subscribed.
In T201776#4503278, @Krinkle wrote:

@Legoktm @Simetrical Is there a commit we can revert meanwhile to unbreak the build? The lack of visibility has already allowed one regression to be lost in the noise - T201900.

I think it's going to be harder as there's been a few followup commits, including reordering ServiceWiring.php alphabetically

gerritbot added a comment.Aug 15 2018, 11:04 PM

Change 453069 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] title: Disable the failing tests from TitlePermissionTest

https://gerrit.wikimedia.org/r/453069

gerritbot added a project: Patch-For-Review.Aug 15 2018, 11:04 PM
Reedy mentioned this in T202050: Several travis-ci builds failing for mediawiki.Aug 16 2018, 1:14 PM
MarcoAurelio merged a task: T202050: Several travis-ci builds failing for mediawiki.Aug 16 2018, 1:17 PM
MarcoAurelio subscribed.
gerritbot added a comment.Aug 16 2018, 1:33 PM

Change 453069 merged by jenkins-bot:
[mediawiki/core@master] title: Disable the failing tests from TitlePermissionTest

https://gerrit.wikimedia.org/r/453069

Reedy lowered the priority of this task from High to Medium.Aug 16 2018, 1:52 PM
Reedy removed a project: Patch-For-Review.
ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)).Aug 16 2018, 2:00 PM
Krinkle moved this task from In progress to Resolved on the Wikimedia-production-error (ARCHIVED -- Shared Build Failure) board.Aug 16 2018, 5:58 PM
CCicalese_WMF edited projects, added Platform Team Legacy (Designing); removed Core-Platform-Team-Old (CPT-Q1-Jul-Sep-2018).Oct 1 2018, 1:48 PM
CCicalese_WMF added a project: Platform Engineering.Oct 2 2018, 2:15 PM
gerritbot added a comment.Oct 3 2018, 12:38 PM

Change 464139 had a related patch set uploaded (by simetrical; owner: simetrical):
[mediawiki/core@master] Re-enable tests from TitlePermissionTest

https://gerrit.wikimedia.org/r/464139

Restricted Application edited projects, added Platform Team Legacy; removed Platform Team Legacy (Designing). · View Herald TranscriptOct 3 2018, 12:38 PM
gerritbot added a project: Patch-For-Review.Oct 3 2018, 12:38 PM
CCicalese_WMF edited projects, added Platform Team Legacy (Designing), Platform Engineering (Needs Cleaning - Code Health (TEC13)); removed Platform Team Legacy, Platform Engineering.Oct 3 2018, 1:25 PM
CCicalese_WMF moved this task from Inbox to Doing on the Platform Team Legacy (Designing) board.
CCicalese_WMF moved this task from Needs Cleaning - Code Health (TEC13) to Needs Cleaning - Security, stability, performance, and scalability (TEC1) on the Platform Engineering board.Oct 3 2018, 2:21 PM
CCicalese_WMF edited projects, added Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1)); removed Platform Engineering (Needs Cleaning - Code Health (TEC13)).
gerritbot added a comment.Oct 4 2018, 3:40 PM

Change 464139 merged by jenkins-bot:
[mediawiki/core@master] Re-enable tests from TitlePermissionTest

https://gerrit.wikimedia.org/r/464139

ReleaseTaggerBot edited projects, added MW-1.32-notes (WMF-deploy-2018-10-16 (1.32.0-wmf.26)); removed MW-1.32-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)).Oct 4 2018, 4:00 PM
Krinkle raised the priority of this task from Medium to High.Oct 5 2018, 8:18 PM
Krinkle moved this task from Resolved to In progress on the Wikimedia-production-error (ARCHIVED -- Shared Build Failure) board.

(This is breaking the Travis CI build again, and probably local tests as well when sitename != 'MyWiki')

Krinkle removed projects: MW-1.32-notes (WMF-deploy-2018-10-16 (1.32.0-wmf.26)), Patch-For-Review.Oct 5 2018, 8:20 PM
Krinkle moved this task from Inbox to PHPUnit on the MediaWiki-Core-Tests board.
Simetrical added a comment.Oct 7 2018, 9:05 AM

Well, one solution would be to force the sitename to MyWiki for these tests, right? Or we could just leave them disabled. (Assuming nobody wants to figure out what the actual problem is, which isn't so easy if it can't be reproduced locally.)

Krinkle added a comment.Oct 9 2018, 1:23 AM

Yes, tests that produce asserted values that vary by configuration should mock or set those configuration key explicitly. That's done in most other tests as well. I imagine it may've been avoided here if it's difficult to do so, but until we find away, the test should't be breaking CI for several days, so I've re-reverted it for now.

See also https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/464386/ / T206130, which triggered a similar build failure.

Krinkle added a subscriber: kostajh.Oct 9 2018, 1:23 AM
Simetrical added a comment.Oct 9 2018, 9:48 AM

There is no way for every test to compile a list of all config options that might cause it to fail. There are far too many config options. The only solution is https://gerrit.wikimedia.org/r/451638.

Simetrical added a comment.Oct 9 2018, 12:16 PM

I can actually reproduce locally, but only if I run all of includes/, not just if I run the file by itself.

Simetrical added a comment.Oct 9 2018, 12:37 PM

I can reproduce by editing suite.xml to run only HtmlTest.php, TitleMethodsTest.php, and TitlePermissionTest.php (by adding <exclude> for everything else in includes/). Now working on narrowing down more.

Simetrical added a comment.Oct 9 2018, 12:41 PM

In retrospect, grepping for "MyWiki" would have been a vastly quicker way to narrow down the problem.

gerritbot added a comment.Oct 9 2018, 1:31 PM

Change 465423 had a related patch set uploaded (by simetrical; owner: simetrical):
[mediawiki/core@master] Reset Language/MWNamespace caches between tests

https://gerrit.wikimedia.org/r/465423

gerritbot added a project: Patch-For-Review.Oct 9 2018, 1:31 PM
Simetrical added a comment.Oct 9 2018, 1:31 PM

The fix is depressingly simple. We just needed to reset the Language cache in between test runs.

Krinkle moved this task from In progress to Resolved on the Wikimedia-production-error (ARCHIVED -- Shared Build Failure) board.Oct 10 2018, 12:05 AM
gerritbot added a comment.Oct 10 2018, 12:37 AM

Change 465537 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] title: Disable flaky TitlePermission test cases

https://gerrit.wikimedia.org/r/465537

Krinkle added a comment.Oct 10 2018, 12:53 AM
In T201776#4651944, @Simetrical wrote:

The fix is depressingly simple. We just needed to reset the Language cache in between test runs.

Yep, but doing it for everything has larger consequences and may break compat with existing tests as we found. Let's first unbreak the immediate test and improve the other tests that aren't broken, later :)

I've submitted https://gerrit.wikimedia.org/r/465537 which resets the site name for that test specifically. That's our convention elsewhere and is what should've been done in the first place.

gerritbot added a comment.Oct 10 2018, 1:42 AM

Change 465537 abandoned by Krinkle:
title: Fix flaky TitlePermission test cases

Reason:
I don't know what's wrong with it. But I do know the bloody thing is more complicated than it should be.

https://gerrit.wikimedia.org/r/465537

gerritbot added a comment.Oct 10 2018, 1:47 AM

Change 465537 restored by Krinkle:
title: Fix flaky TitlePermission test cases

https://gerrit.wikimedia.org/r/465537

gerritbot added a comment.Oct 10 2018, 4:02 AM

Change 465537 abandoned by Krinkle:
title: Fix flaky TitlePermission test cases

Reason:
I give up. Something somewhere is messing with the global state in the shared job. Presumably in such a way that TestCase is unable to reset.

That probably means our resets are too brave by stashing and restoring, instead of actually clearing/resetting.

Anyhow, given we've spent 7 days and over a dozen commits trying to do this, I give up. Kosta has a patch that reduces the test to not needlessly assert what the wgSitename value is, which makes a ton of sense, given the test isn't about that at all. I'm merging that now.

We can continue pondering about why we can't have nice resets after that, including how to make them work with our existing problems in SimpleCaptcha, Flow, and Wikibase.

https://gerrit.wikimedia.org/r/465537

gerritbot added a comment.Oct 10 2018, 11:17 PM

Change 465555 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@master] Fix TitlePermissionTest failures due to test leakage

https://gerrit.wikimedia.org/r/465555

Krinkle unsubscribed.Oct 10 2018, 11:24 PM
gerritbot added a comment.Oct 10 2018, 11:36 PM

Change 465555 merged by jenkins-bot:
[mediawiki/core@master] Fix TitlePermissionTest failures due to test leakage

https://gerrit.wikimedia.org/r/465555

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-10-16 (1.32.0-wmf.26)).Oct 11 2018, 12:00 AM
CCicalese_WMF edited projects, added Platform Team Workboards (Doing); removed Platform Team Legacy (Designing).Oct 14 2018, 3:56 PM
CCicalese_WMF subscribed.Nov 6 2018, 2:32 AM

Is this fixed?

kostajh added a subscriber: Krinkle.Nov 6 2018, 2:54 AM

@CCicalese_WMF the builds in Travis CI are passing again https://travis-ci.org/wikimedia/mediawiki/builds but not sure if @Krinkle wanted to do anything further here.

kostajh mentioned this in T209056: Use Travis CI cache and reduce git clone depth to speed up build setup time.Nov 8 2018, 1:36 PM
Krinkle closed this task as Resolved.Dec 3 2018, 3:23 AM
Krinkle claimed this task.

LGTM. Thanks!

CCicalese_WMF moved this task from Doing to Done with CPT on the Platform Team Workboards board.Dec 13 2018, 8:09 PM
CCicalese_WMF edited projects, added Platform Team Workboards (Done with CPT); removed Platform Team Workboards (Doing).
gerritbot added a comment.Apr 8 2019, 2:47 PM

Change 465423 abandoned by simetrical:
Reset Language/MWNamespace caches between tests

Reason:
This has been superseded by the LanguageFactory and NamespaceInfo patches.

https://gerrit.wikimedia.org/r/465423

• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:09 PM
• Pchelolo subscribed.Oct 18 2020, 12:47 AM

https://gerrit.wikimedia.org/r/c/mediawiki/core/+/634409/6 has refactored and restored the test. If this issue returns, we can just mark it as @group Broken again.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL