Page MenuHomePhabricator
Create Task
Maniphest T69216

Have unit tests of all wmf deployed extensions pass when installed together, in both PHP-Zend and HHVM (tracking)
Closed, ResolvedPublic
Actions

Description

When installing all the MediaWiki extensions currently deployed on the Wikimedia cluster with a dummy LocalSettings.php, we end up with a lot of failures, bad interactions between extensions or core tests broken via hooks. Some extensions even end up throwing fatal errors.

To help migrate to HHVM, we would need to be able to run the unit tests of core and of all the extensions and have them pass.

Ori has worked on this a bit for HHVM and already reported a fair amount of bugs.

Antoine has been working on an experimental Jenkins Job that run all wmf extensions tests.

Lets join the effort, fill bug and track them there. Then reach out to the other developers and get everything fixed.

Version: wmf-deployment
Severity: normal

Details

Reference
bz67216

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedhasharT60772 common gating job for mediawiki core and extensions
 ResolvedhasharT69216 Have unit tests of all wmf deployed extensions pass when installed together, in both PHP-Zend and HHVM (tracking)
 ResolvedNoneT57886 CLDR breaks core timestamp tests
 DeclinedNoneT70231 wfWikiID() returns different values from test and data provider
 ResolvedNoneT68937 HHVM: TemplateDataBlobTest::testGetDataInLanguage fails with "Invalid operand type was used: expecting an array"
 ResolvedNoneT68942 HHVM: assertion from optimizedCallIniGet
 ResolvedNoneT68943 HHVM: Assertion in Scribunto_LuaEngine::callParserFunction() during uksort() call
 ResolvedNoneT69036 SpecialPagesTest::testSpecialPage errors with: array_merge() expects at least 1 parameter, 0 given
 ResolvedNoneT69043 Enabling SpamBlacklist breaks EditPageTest in core
 DeclinedNoneT69060 Gadgets: Fix failing unit test "GadgetsTest::testPreferences: isset( $options['<gadget-section-keep-section2>'] )"
 InvalidNoneT70629 PHP Notice: Undefined index: REQUEST_TIME in extensions/EventLogging/includes/EventLogging.php on line 56
 ResolvedNoneT70634 Exception: .agora-field-styling is undefined: failed at `.agora-field-styling();`
 OpenNoneT70636 No enabled connection [Called from CirrusSearch\ElasticsearchIntermediary::failure in /srv/vagrant/mediawiki/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php at line 97]
 InvalidTptT70644 phpunit failure: Pages::testParserTest: Failed asserting that two strings are equal.
 ResolvedNoneT70646 phpunit failure: LuaStandalone: TitleLibraryTests: Failed asserting that two strings are identical
 ResolvedNoneT69139 BetaFeatures: Unit tests trigger notices in MobileFrontend
 ResolvedNoneT69138 BetaFeatures: AutoEnrollmentTest produce a fatal error with OATHAuth extension
 OpenNoneT70648 phpunit failure: Module prefix '' is shared between ApiQueryProofread and ApiTranscodeStatus
 ResolvedNoneT70649 phpunit failure: ParserTests::testParserTest for gallery and files with UploadWizard and data-file-*
 ResolvedNoneT69142 ZeroPortal unit tests throws Warning: JsonConfig: Invalid $wgJsonConfigs['JsonZeroConfig']['url']: API URL is not set ...
 InvalidNoneT70652 phpunit failure: Magic Word: {{NUMBEROFFILES}} Failed asserting that two strings are equal.
 InvalidNoneT70651 phpunit failure: UserTest::testAllRightsWithMessage with some additional rights
 ResolvedNoneT69148 Math tests dies when run with the Echo extensions
 OpenNoneT70654 phpunit failure: ApiOptionsTest::testReset User::saveSettings() was not expected to be called more than once
 ResolvedEBernhardsonT70655 Flow: Fix phpunit failure in SpamBlacklistTest::testSpam
 Invalid• demonT70672 hhvm: SolrTTMServerTest::testSearchableTTMServer BadMethodCallException: Call to a member function getMessage() on a non-object (NULL)
 ResolvedNoneT70697 PHP Fatal error: Call to a member function getContentModel() on a non-object in /srv/vagrant/mediawiki/extensions/Wikibase/repo/Wikibase.hooks.php on line 1005
 InvalidNoneT70700 phpunit failure: TitlePermissionTest
 DeclinedNoneT70699 TemplateData.hooks.php:71: MWException: Invalid callback in hooks for ParserFirstCallInit
 ResolvedNoneT69296 oauth_registered_consumer table not created under sqlite
 DeclinedNoneT69295 Fatal error: Call to a member function getMessage() on a non-object in /mediawiki/extensions/Translate/tag/TranslatablePage.php on line 253
 ResolvedNoneT70491 HHVM Failed Assertion: assertion `false && "Unsupported DataType"' failed.
 ResolvedhasharT69297 unittest_oathauth_users table not created under sqlite
 ResolvedReedyT69300 GlobalBlocking is lacking a database installer/updater
 OpenNoneT69302 MultimediaViewer thumbnailBeforeProduceHTML hook breaks other extensions parser tests
 ResolvedNoneT69353 GlobalBlocking cause other extensions tests to fail (globalblocking database not found)
 ResolvedNoneT69366 PageTriage: test fail ArticleMetadataTest::testValidatePageId MWException: DatabaseBase::makeList: empty input for field ptrp_page_id
 ResolvedJdforrester-WMFT70956 VisualEditor: ResourcesTest::testUnsatisfiableDependencies: The module 'mobile.editor.ve' must not have target 'desktop'
 ResolvedNikerabbitT70955 ResourcesTest::testMissingDependencies: Failed asserting that an array contains 'schema.UniversalLanguageSelector'.
 ResolvedNoneT70958 hhvm: BadMethodCallException: Call to a member function getOption() on a non-object
 OpenNoneT70957 phpunit failure: Scribunto_LuaStandaloneInterpreterTest::testTimeLimit
 ResolvedhasharT67875 AccountAudit does not pass Jenkins unit tests
 ResolvedhasharT67877 CreditsSource does not pass Jenkins unit tests (sqlite compatibility)
 ResolvedhasharT67876 CommunityVoice does not pass Jenkins unit tests (sqlite compatibility)
 ResolvedLegoktmT67895 OAI does not pass Jenkins tests (sqlite compatibility)
 ResolvedhasharT67896 TitleKey does not pass Jenkins unit tests (sqlite compatibility)
 ResolvedhasharT69598 EducationProgram dies if cldr is required after it
 DeclinedNoneT69608 EducationProgram test fail with permission error caused by WikimediaIncubator
 DeclinedNoneT69615 PageTriage: test SpecialNewPagesFeedTest::testAddArticles has 'Unknown error: "wminc-error-unprefixed"'
 ResolvedNoneT75521 Jenkins: Implement hhvm based voting jobs for mediawiki and extensions
 ResolvedhasharT73413 [OPS] hhvm 3.3.0-20140925+wmf3 has some annoying build dependency
 ResolvedhasharT75356 [OPS] Jenkins: Slaves running Ubuntu Trusty should have hhvm installed
 ResolvedJoeT75531 ApiFormatWddxTest::testValidSyntax fails in mediawiki-core-regression-hhvm-master test
 ResolvedJoeT84853 Build a new HHVM package
 ResolvedhasharT78166 MediaWiki tests using Tidy do not pass on HHVM
 ResolvedhasharT78556 hhvm on CI slaves can not find PEAR.php
 ResolvedEBernhardsonT78592 EchoTitleLocalCacheTest::testGet fails when tests are run with other extensions
 ResolvedQChrisT78680 EventLogging ValidateSchemaTest::testValidEvent() fails under HHVM
 ResolvedLegoktmT101392 Trigger PHPUnit job for MediaWiki extensions with HHVM instead of Zend in 'test' pipeline
 ResolvedhasharT101382 Only run Wikibase Zend Jenkins jobs on gate-and-submit
 Duplicate• Mattflaschen-WMFT103346 phpunit hhvm failure: LuaSandbox: TextLibraryTests[87]: json decode, invalid values (trailing comma)
 InvalidRyasmeenT90647 Create Jenkins builds for Editing across repositories (MobileFrontend, VisualEditor etc)
 ResolvedhasharT47499 [EPIC] Run CI jobs in disposable VMs
 ResolvedhasharT55594 Jenkins: Configure jobs to be runnable on any slaves
 ResolvedhasharT55683 [OPS] publish Zuul git repositories over git protocol
 ResolvedhasharT56762 find a way to deploy PHPUnit on production machines
 ResolvedhasharT58317 Jenkins: Use phpunit from deployment system (srv/deployment/integration/phpunit) instead of local pear install
 DuplicateNoneT55978 Create labs project for continuous integration nodepool
 Resolvedbd808T53556 Investigate Linux containers
 DuplicateNoneT84988 Create labs project for CI disposables instances + OpenStack API credentials
 DeclinedNoneT84989 Support dedicating a specific virt node to a specific nova project
 DeclinedhasharT84940 Acquire old production API servers for use in CI
 ResolvedhasharT85933 Evaluate JClouds Jenkins plugin
 DeclinedNoneT86168 Isolate contintcloud nova project from the rest of the wmflabs cloud
 Declined• chasempT85611 Neutron networking, with IPv6 at eqiad
 DeclinedNoneT85609 Labs available in the new data centre (with Neutron/IPv6)
 ResolvedcorenT85605 Set storage service up in codfw
 InvalidhasharT86172 Write a migration plan for CI infra to the disposable VMs infrastructure
 ResolvedhasharT86171 Design the Jenkins isolation architecture
 DeclinedhasharT97472 Create a Trusty labs image for CI isolation project
 Resolved• CmjohnsonT86658 Phase out lanthanum.eqiad.wmnet
 Resolved• CmjohnsonT105901 wipe disks for lanthanum
 ResolvedhasharT86659 Migrate all jobs to labs slaves
 ResolvedKrinkleT86175 Migrate jsduck jobs to run in labs (and jsduck-publish via integration-publisher)
 ResolvedKrinkleT86174 Migrate JSDuck to Ubuntu Trusty
 ResolvedKrinkleT93143 Migrate .*testextension-zend jobs to labs slaves
 ResolvedhasharT93559 Migrate mediawiki-core-code-coverage job to labs
 ResolvedKrinkleT93558 Add publishing macro for integration.wikimedia.org/cover
 ResolvedhasharT96463 Move integration-zuul-layout* jobs to labs slaves
 ResolvedhasharT95675 Migrate Jenkins job "Global-Dev Dashboard Data" to JJB and Zuul trigger
 ResolvedhasharT98737 Migrate operations-dns-lint Jenkins job to labs instances
 ResolvedhasharT101966 Remove use of label "hasSlaveScripts" in jobs
 ResolvedhasharT87294 Nodepool images need Gerrit mirror for git-clone performance
 ResolvedAndrewT113394 Evaluate impact of giant VM images on labs infrastructure
 ResolvedKrinkleT91996 Consolidate jobs to test entry points
 ResolvedhasharT91997 Replace project-specific "{name}-thing" jobs with generic "thing" ones
 ResolvedKrinkleT91854 Migrate jsduck-publish to use a single job and variable destination path
 ResolvedPaladoxT90943 Convert existing legacy phpcs jobs to use composer entry point + versioning
 ResolvedLegoktmT91176 Create composer validate --no-check-publish job for MediaWiki extensions
 ResolvedLegoktmT92557 Create generic "tox" jobs
 ResolvedLegoktmT94326 Create generic qunit job for MediaWiki extensions
 ResolvedLegoktmT94327 Create generic phpunit job for MediaWiki extensions
 ResolvedPaladoxT126682 Convert all MediaWiki extension phpunit jobs to use generic jobs
 ResolvedLegoktmT127362 Switch passing mwext-*-jslint jobs to jshint/jsonlint
 ResolvedLegoktmT96690 Have extensions with dependencies use the generic mwext-testextension-* job
 ResolvedhasharT109913 [keyresult] boot instances from OpenStack API
 ResolvedhasharT86170 OpenStack API account to control `contintcloud` labs project
 ResolvedyuvipandaT86167 Create labs project for CI disposables instances
 OpenNoneT86930 Add i18n related extensions to the CI gate
 ResolvedJdforrester-WMFT225068 Add a PHPUnit group to skip test on gated CI runs

Event Timeline

• bzimport raised the priority of this task from to High.Nov 22 2014, 3:31 AM
• bzimport added projects: Continuous-Integration-Infrastructure, Tracking-Neverending.
• bzimport set Reference to bz67216.
• bzimport added a subscriber: Unknown Object (MLST).
hashar created this task.Jun 27 2014, 9:35 PM
hashar added a comment.Jun 27 2014, 9:41 PM

Add bugs Ori and I filled.

Jdforrester-WMF added a comment.Jun 27 2014, 9:48 PM

Relabel per discussion.

hashar added a comment.Jun 30 2014, 2:12 PM

A few more extension that can pass the installer / fail tests.

Nemo_bis added a comment.Jun 30 2014, 9:56 PM

Thanks a bunch for working on this. Those who shouted for help on wikitech-l and have been unable for years to run tests on their own environment / with their own configuration will be enthusiastic.

hashar added a comment.Jun 30 2014, 10:30 PM

I am merely feeling bugs and we will attempt to fix as many possible. But we could use some help :-) Please spread the word to wikitech-l! New bugs would be specially appreciated.

The aim is to have core + all wmf extensions to pass the PHPUnit test suite.

ori added a comment.Jul 3 2014, 6:59 AM

This is awesome :) Thanks for pushing this, Antoine.

hashar added a comment.Jul 7 2014, 8:57 PM

Status as of today.

I have commented out the following extensions for various reasons mentioned in the commit message https://git.wikimedia.org/commit/integration%2Fjenkins-job-builder-config/50d2cda7ffca7655889a48b25fccd3ce15977ff6

MediaWiki-General/extensions/CentralAuth
MediaWiki-General/extensions/Math
MediaWiki-General/extensions/OAI
MediaWiki-General/extensions/Oversight
MediaWiki-General/extensions/RandomRootPage
MediaWiki-General/extensions/SemanticForms
MediaWiki-General/extensions/SemanticMediaWiki
MediaWiki-General/extensions/SemanticResultFormats
MediaWiki-General/extensions/Translate
MediaWiki-General/extensions/Validator
MediaWiki-General/extensions/ZeroRatedMobileAccess
MediaWiki-General/extensions/timeline

There are bug filled for most of them.

The complete list from the Jenkins configuration:

https://git.wikimedia.org/blob/integration%2Fjenkins-job-builder-config/50d2cda7ffca7655889a48b25fccd3ce15977ff6/mediawiki-extensions.yaml#L147

A test report can be find at: https://integration.wikimedia.org/ci/job/mediawiki-core-extensions-integration/68/testReport/

Almost all of the failures are parser tests which hardcode expected HTML output. But since some extensions register hooks that further adjust the HTML output, that does not play well. Bug 67302 - ""MultimediaViewer breaks Cite extension parser tests"" has some more details.

Nemo_bis added a comment.Jul 26 2014, 11:20 PM

Created attachment 16044
php5 phpunit.php

I run php5 phpunit.php after enabling all gerrit extension having unit tests and almost all MediaWiki-Vagrant roles (http://paste.debian.net/111811/ minus roles globalcssjs math mathsearch cirrussearch) and there's quite some output to process...

Attached:

file_67216.txt314 KBDownload

hashar added a comment.Jul 28 2014, 9:27 AM

(In reply to Nemo from comment #8)

Created attachment 16044 [details]
php5 phpunit.php

I run php5 phpunit.php after enabling all gerrit extension having unit tests
and almost all MediaWiki-Vagrant roles (http://paste.debian.net/111811/
minus roles globalcssjs math mathsearch cirrussearch) and there's quite some
output to process...

I am going to have a Jenkins job triggered for all extensions. Gotta polish up some details but I have a working base to make it happen soon ™

Attached:

file_67216.txt314 KBDownload

• demon closed subtask T70672: hhvm: SolrTTMServerTest::testSearchableTTMServer BadMethodCallException: Call to a member function getMessage() on a non-object (NULL) as Invalid.Nov 25 2014, 3:11 PM
Physikerwelt closed subtask T69148: Math tests dies when run with the Echo extensions as Resolved.Jan 10 2015, 3:34 PM
hashar added a comment.Jan 16 2015, 11:11 AM

The extensions are slowly being added to the shared jobs mediawiki-extensions-hhvm and mediawiki-extensions-zend

An example is https://gerrit.wikimedia.org/r/#/c/185167/ which adds:

  • AbuseFilter
  • CheckUser
  • ConfirmEdit
  • EventLogging
  • SpamBlacklist
hashar added a comment.Jan 16 2015, 11:13 AM

Moreover, the hhvm flavor of the job is available on ALL extensions in the experimental pipeline. If one want to verify whether its extension is a good candidate for inclusion, simply comment in Gerrit 'check experimental'. The extension will be injected in the list of cloned extensions and the tests will be run for the resulting set.

See also my recent announcement on wikitech-l.

hashar claimed this task.Jan 16 2015, 11:14 AM
hashar moved this task from Untriaged to In-progress on the Continuous-Integration-Infrastructure board.
hashar added a subtask: T75521: Jenkins: Implement hhvm based voting jobs for mediawiki and extensions.
Quiddity closed subtask T70655: Flow: Fix phpunit failure in SpamBlacklistTest::testSpam as Resolved.Jan 28 2015, 7:40 PM
JanZerebecki subscribed.Feb 5 2015, 12:51 PM
greg mentioned this in T88263: Clean #Wiki-Release-Team project.Feb 5 2015, 11:36 PM
greg mentioned this in T536: Take DumpHTML as a use case of 3rd party extension that MediaWiki maintainers cannot ignore.Feb 12 2015, 5:58 PM
greg added a subtask: T90647: Create Jenkins builds for Editing across repositories (MobileFrontend, VisualEditor etc).Feb 25 2015, 12:02 AM
Tpt closed subtask T70644: phpunit failure: Pages::testParserTest: Failed asserting that two strings are equal. as Invalid.Mar 25 2015, 1:18 PM
Legoktm closed subtask T67895: OAI does not pass Jenkins tests (sqlite compatibility) as Resolved.Apr 29 2015, 2:41 AM
hashar removed hashar as the assignee of this task.Jun 9 2015, 2:31 PM
hashar moved this task from In-progress to Next on the Continuous-Integration-Infrastructure board.

Not actively working on it. We surely need more extensions to be added in.

hashar added a subtask: T86930: Add i18n related extensions to the CI gate.Jun 10 2015, 1:33 PM
Nemo_bis added a comment.Jun 13 2015, 11:42 PM

I still have unreviewed patches from my latest testing sprint almost an year ago... once those are all closed I'll reclone everything and repeat.

Jdforrester-WMF closed subtask T70956: VisualEditor: ResourcesTest::testUnsatisfiableDependencies: The module 'mobile.editor.ve' must not have target 'desktop' as Resolved.Jul 31 2015, 11:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 31 2015, 11:29 PM
Nemo_bis mentioned this in T69302: MultimediaViewer thumbnailBeforeProduceHTML hook breaks other extensions parser tests.Sep 24 2015, 4:13 PM
hashar lowered the priority of this task from High to Medium.Oct 6 2015, 11:27 AM
hashar set Security to None.
hashar closed subtask T90647: Create Jenkins builds for Editing across repositories (MobileFrontend, VisualEditor etc) as Invalid.Oct 6 2015, 12:33 PM
Legoktm closed subtask T75521: Jenkins: Implement hhvm based voting jobs for mediawiki and extensions as Resolved.Feb 10 2016, 8:48 AM
Krinkle unsubscribed.Feb 23 2016, 4:49 PM
• Mattflaschen-WMF reopened subtask T69300: GlobalBlocking is lacking a database installer/updater as Open.Mar 24 2016, 2:45 AM
• Mattflaschen-WMF closed subtask T69300: GlobalBlocking is lacking a database installer/updater as Resolved.Mar 31 2016, 12:07 AM
Quiddity removed a parent task: T4007: [DO NOT USE] Tracking bug [superseded by #Tracking].Jul 14 2016, 2:06 AM
MBinder_WMF closed subtask T69302: MultimediaViewer thumbnailBeforeProduceHTML hook breaks other extensions parser tests as Declined.Aug 4 2016, 8:23 PM
Legoktm reopened subtask T69302: MultimediaViewer thumbnailBeforeProduceHTML hook breaks other extensions parser tests as Open.Aug 4 2016, 8:24 PM
Nikerabbit closed subtask T70955: ResourcesTest::testMissingDependencies: Failed asserting that an array contains 'schema.UniversalLanguageSelector'. as Resolved.Aug 18 2016, 12:46 PM
Krinkle closed subtask T70629: PHP Notice: Undefined index: REQUEST_TIME in extensions/EventLogging/includes/EventLogging.php on line 56 as Invalid.Oct 5 2016, 10:21 PM
hashar closed subtask T69608: EducationProgram test fail with permission error caused by WikimediaIncubator as Declined.Nov 24 2016, 2:10 PM
Krinkle closed subtask T70231: wfWikiID() returns different values from test and data provider as Declined.Jan 13 2017, 2:14 AM
Umherirrender mentioned this in T134090: fix or mark as inactive extensions currently failing CI.Jan 19 2017, 8:47 PM
Hydriz closed subtask T69615: PageTriage: test SpecialNewPagesFeedTest::testAddArticles has 'Unknown error: "wminc-error-unprefixed"' as Declined.Apr 8 2017, 1:54 PM
hashar closed this task as Resolved.Aug 23 2019, 2:26 PM
hashar claimed this task.

I have filled this task to be able to run PHPUnit tests of all Wikimedia extensions in the same run. There are a few extensions that are incompatible, but overall the task has served its purpose. There is no point in keeping it, the bulk of the work has been done in 2015 / 2016.

Jdforrester-WMF added a comment.Aug 23 2019, 2:30 PM
In T69216#5433920, @hashar wrote:

I have filled this task to be able to run PHPUnit tests of all Wikimedia extensions in the same run. There are a few extensions that are incompatible, but overall the task has served its purpose. There is no point in keeping it, the bulk of the work has been done in 2015 / 2016.

Surely the incompatible extensions' tests are going to be work we have to do as part of the pipeline work?

Jdforrester-WMF unsubscribed.Aug 23 2019, 2:30 PM
Nikerabbit changed the status of subtask T69295: Fatal error: Call to a member function getMessage() on a non-object in /mediawiki/extensions/Translate/tag/TranslatablePage.php on line 253 from Open to Stalled.Dec 22 2020, 8:49 AM
hashar closed subtask T69295: Fatal error: Call to a member function getMessage() on a non-object in /mediawiki/extensions/Translate/tag/TranslatablePage.php on line 253 as Declined.Apr 28 2021, 8:31 AM
hashar moved this task from Next to Done on the Continuous-Integration-Infrastructure board.Jun 4 2021, 9:52 AM
Daimona closed subtask T70700: phpunit failure: TitlePermissionTest as Invalid.Aug 19 2023, 5:01 PM
Daimona closed subtask T70652: phpunit failure: Magic Word: {{NUMBEROFFILES}} Failed asserting that two strings are equal. as Invalid.Aug 19 2023, 5:14 PM
Daimona closed subtask T70651: phpunit failure: UserTest::testAllRightsWithMessage with some additional rights as Invalid.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL