Page MenuHomePhabricator

Chameleon test failures on WMF CI
Closed, DeclinedPublic

Event Timeline

Reedy created this task.May 21 2020, 4:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 21 2020, 4:01 PM
Reedy triaged this task as High priority.May 21 2020, 4:27 PM
CCicalese_WMF lowered the priority of this task from High to Medium.
CCicalese_WMF added a comment.EditedMay 26 2020, 7:59 PM

There is a notable difference between running phpunit locally (which works) and in Wikimedia CI. https://github.com/ProfessionalWiki/chameleon/blob/master/docs/testing.md require adding -c phpunit.xml.dist, which may have inconsistent options for phpunit. Removing that run the local command results in multiple instances of the error:

Use of undefined constant USE_EXTERNAL_HTML_VALIDATOR - assumed 'USE_EXTERNAL_HTML_VALIDATOR' (this will throw an Error in a future version of PHP)

This is different from the error seen multiple times in Wikimedia CI:

InvalidArgumentException: Key "CanonicalNamespaceNames" not found in input sources
/workspace/src/includes/config/ServiceOptions.php:52

But, in both cases, it is a single error that is repeated in multiple tests.

Reedy added a comment.May 26 2020, 9:05 PM

USE_EXTERNAL_HTML_VALIDATOR is only defined in phpunit.xml.dist. But as that's not failing in CI, I can presume we're including/using the phpunit config file.

Then they use this file to run phpunit via composer.

I don't think you need to worry about that; if CI was failing with that error, we'd have something wrong with the CI config etc.

Looking at the first error..

16:58:28 1) Skins\Chameleon\Tests\Unit\ChameleonTemplateTest::testCanConstruct
16:58:28 Undefined index: egChameleonLayoutFile
16:58:28 
16:58:28 /workspace/src/skins/chameleon/tests/phpunit/Unit/ChameleonTemplateTest.php:55

Something is seemingly going amiss with the config here, as it's defined by default in skin.json

	"config_prefix": "eg",
	"config": {
		"ChameleonLayoutFile": {
			"value": "layouts/standard.xml"
		},
		"ChameleonEnableVisualEditor": {
			"value": true
		}
	},
Reedy added a comment.May 26 2020, 9:45 PM

Also, the tests don't pass locally for me...

reedy@ubuntu64-web-esxi:/var/www/wiki/mediawiki/core/skins/chameleon$ php /var/www/wiki/mediawiki/core/tests/phpunit/phpunit.php --group skins-chameleon -c phpunit.xml.dist
#!/usr/bin/env php
Using PHP 7.3.11-0ubuntu0.19.10.4

Using the MediaWiki vendor autoloader ...

PHPUnit 8.5.5 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.3.11-0ubuntu0.19.10.4 with Xdebug 2.7.2
Configuration: /var/www/wiki/mediawiki/skins/chameleon/phpunit.xml.dist

  Warning - The configuration file did not pass validation!
  The following problems have been detected:

  Line 40:
  - Element 'phpunit', attribute 'checkForUnintentionallyCoveredCode': The attribute 'checkForUnintentionallyCoveredCode' is not allowed.

  Test results may not be as expected.


S..............................................................  63 / 285 ( 22%)
............................................................... 126 / 285 ( 44%)
.............................................F................. 189 / 285 ( 66%)
............................................................... 252 / 285 ( 88%)
.................................                               285 / 285 (100%)

Time: 1.47 seconds, Memory: 54.50 MB

There was 1 failure:

1) Skins\Chameleon\Tests\Unit\Components\NavbarHorizontal\PersonalToolsTest::testGetHtml_LoggedInUserHasNewMessages with data set #0 (DOMElement Object ())
Failed asserting that the given fragment contained the described node.
Failed asserting that false is true.

/var/www/wiki/mediawiki/skins/chameleon/tests/phpunit/Unit/Components/GenericComponentTestCase.php:284
/var/www/wiki/mediawiki/skins/chameleon/tests/phpunit/Unit/Components/NavbarHorizontal/PersonalToolsTest.php:62
/var/www/wiki/mediawiki/core/maintenance/doMaintenance.php:105

--

There was 1 skipped test:

1) Skins\Chameleon\Tests\Integration\StylesCompileTest::testStylesCompile
Test does not work on recent MW+PHP

/var/www/wiki/mediawiki/skins/chameleon/tests/phpunit/Integration/StylesCompileTest.php:52
/var/www/wiki/mediawiki/core/maintenance/doMaintenance.php:105

FAILURES!
Tests: 285, Assertions: 343, Failures: 1, Skipped: 1.

This is what I'm getting testing locally:

root@b4e039095739:/var/www/mediawiki/skins/Chameleon# php /var/www/mediawiki/tests/phpunit/phpunit.php --group skins-chameleon -c phpunit.xml.dist
Using PHP 7.3.17

Using the MediaWiki vendor autoloader ...

PHPUnit 8.5.5 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.3.17
Configuration: /var/www/wikifarm/skins/Chameleon/phpunit.xml.dist

  Warning - The configuration file did not pass validation!
  The following problems have been detected:

  Line 40:
  - Element 'phpunit', attribute 'checkForUnintentionallyCoveredCode': The attribute 'checkForUnintentionallyCoveredCode' is not allowed.

  Test results may not be as expected.


S..............................................................  63 / 285 ( 22%)
............................................................... 126 / 285 ( 44%)
............................................................... 189 / 285 ( 66%)
............................................................... 252 / 285 ( 88%)
.................................                               285 / 285 (100%)

Time: 2.92 seconds, Memory: 30.00 MB

There was 1 skipped test:

1) Skins\Chameleon\Tests\Integration\StylesCompileTest::testStylesCompile
Test does not work on recent MW+PHP

/var/www/wikifarm/skins/Chameleon/tests/phpunit/Integration/StylesCompileTest.php:52
/var/www/mediawiki/maintenance/doMaintenance.php:105

OK, but incomplete, skipped, or risky tests!
Tests: 285, Assertions: 343, Skipped: 1.

I removed the line causing the 'checkForUnintentionallyCoveredCode', and the warning went away, but everything else stayed the same.

I'm slightly irked by this... Why not check it's defined at all?

	public function assertValidHTML( $actual, $message = 'HTML text is not valid. ' ) {
		if ( !USE_EXTERNAL_HTML_VALIDATOR ) {

			$doc = $this->loadXML( $actual, true );
			$this->assertNotFalse( $doc, $message );

			return;
		}

Gonna change this to

	public function assertValidHTML( $actual, $message = 'HTML text is not valid. ' ) {
		if ( !defined( 'USE_EXTERNAL_HTML_VALIDATOR' ) || !USE_EXTERNAL_HTML_VALIDATOR ) {

			$doc = $this->loadXML( $actual, true );
			$this->assertNotFalse( $doc, $message );

			return;
		}

I removed the line causing the 'checkForUnintentionallyCoveredCode', and the warning went away, but everything else stayed the same.

I pushed a fix for that upstream. It's been deprecated since PHPUnit 5.2, removed in 6.0. No value lost

https://github.com/ProfessionalWiki/chameleon/issues/160 filed for the test failure I see on my version with PHP

Reedy added a comment.May 27 2020, 1:04 AM

This is getting ridiculous.

I half wonder if it's because it's not using the proper MW test classes/bootstrapping

https://gerrit.wikimedia.org/r/#/c/mediawiki/skins/chameleon/+/597812/8/tests/phpunit/Unit/Components/GenericComponentTestCase.php

01:39:31 2) Skins\Chameleon\Tests\Unit\Components\LogoTest::testGetHtmlwithEmptyElement
01:39:31 Error: Call to a member function getIP() on null
01:39:31 
01:39:31 /workspace/src/includes/user/User.php:2149
01:39:31 /workspace/src/includes/session/SessionBackend.php:734
01:39:31 /workspace/src/includes/session/SessionBackend.php:207
01:39:31 /workspace/src/includes/session/Session.php:73
01:39:31 /workspace/src/includes/session/SessionManager.php:865
01:39:31 /workspace/src/includes/session/SessionManager.php:310
01:39:31 /workspace/src/includes/session/SessionManager.php:244
01:39:31 /workspace/src/includes/session/SessionManager.php:194
01:39:31 /workspace/src/includes/WebRequest.php:830
01:39:31 /workspace/src/includes/user/User.php:1165
01:39:31 /workspace/src/includes/user/User.php:340
01:39:31 /workspace/src/includes/user/User.php:2121
01:39:31 /workspace/src/includes/user/User.php:3227
01:39:31 /workspace/src/includes/user/UserOptionsManager.php:547
01:39:31 /workspace/src/includes/user/UserOptionsManager.php:455
01:39:31 /workspace/src/includes/user/UserOptionsManager.php:124
01:39:31 /workspace/src/includes/user/User.php:2756
01:39:31 /workspace/src/includes/context/RequestContext.php:351
01:39:31 /workspace/src/includes/language/Message.php:402
01:39:31 /workspace/src/includes/language/Message.php:1305
01:39:31 /workspace/src/includes/language/Message.php:1000
01:39:31 /workspace/src/includes/Linker.php:2114
01:39:31 /workspace/src/includes/Linker.php:2305
01:39:31 /workspace/src/skins/chameleon/src/Components/Logo.php:85
01:39:31 /workspace/src/skins/chameleon/src/Components/Logo.php:72
01:39:31 /workspace/src/skins/chameleon/src/Components/Logo.php:50
01:39:31 /workspace/src/skins/chameleon/tests/phpunit/Unit/Components/GenericComponentTestCase.php:85
01:39:31 
01:39:31 Caused by
01:39:31 Error: Call to a member function getIP() on null
01:39:31 
01:39:31 /workspace/src/includes/user/User.php:2149
01:39:31 /workspace/src/includes/session/SessionBackend.php:734
01:39:31 /workspace/src/includes/session/SessionBackend.php:618
01:39:31 /workspace/src/vendor/wikimedia/scoped-callback/src/ScopedCallback.php:96
01:39:31 /workspace/src/vendor/wikimedia/scoped-callback/src/ScopedCallback.php:56
01:39:31 /workspace/src/includes/session/SessionManager.php:865
01:39:31 /workspace/src/includes/session/SessionManager.php:310
01:39:31 /workspace/src/includes/session/SessionManager.php:244
01:39:31 /workspace/src/includes/session/SessionManager.php:194
01:39:31 /workspace/src/includes/WebRequest.php:830
01:39:31 /workspace/src/includes/user/User.php:1165
01:39:31 /workspace/src/includes/user/User.php:340
01:39:31 /workspace/src/includes/user/User.php:2121
01:39:31 /workspace/src/includes/user/User.php:3227
01:39:31 /workspace/src/includes/user/UserOptionsManager.php:547
01:39:31 /workspace/src/includes/user/UserOptionsManager.php:455
01:39:31 /workspace/src/includes/user/UserOptionsManager.php:124
01:39:31 /workspace/src/includes/user/User.php:2756
01:39:31 /workspace/src/includes/context/RequestContext.php:351
01:39:31 /workspace/src/includes/language/Message.php:402
01:39:31 /workspace/src/includes/language/Message.php:1305
01:39:31 /workspace/src/includes/language/Message.php:1000
01:39:31 /workspace/src/includes/Linker.php:2114
01:39:31 /workspace/src/includes/Linker.php:2305
01:39:31 /workspace/src/skins/chameleon/src/Components/Logo.php:85
01:39:31 /workspace/src/skins/chameleon/src/Components/Logo.php:72
01:39:31 /workspace/src/skins/chameleon/src/Components/Logo.php:50
01:39:31 /workspace/src/skins/chameleon/tests/phpunit/Unit/Components/GenericComponentTestCase.php:85
01:39:31 
01:39:31 3) Skins\Chameleon\Tests\Unit\Components\NavMenuTest::testGetHtmlwithEmptyElement
01:39:31 MWException: Unable to determine IP.
01:39:31 
01:39:31 /workspace/src/includes/WebRequest.php:1263
01:39:31 /workspace/src/includes/user/User.php:2149
01:39:31 /workspace/src/includes/user/UserOptionsManager.php:547
01:39:31 /workspace/src/includes/user/UserOptionsManager.php:455
01:39:31 /workspace/src/includes/user/UserOptionsManager.php:124
01:39:31 /workspace/src/includes/user/User.php:2756
01:39:31 /workspace/src/includes/context/RequestContext.php:351
01:39:31 /workspace/src/includes/language/Message.php:402
01:39:31 /workspace/src/includes/language/Message.php:1305
01:39:31 /workspace/src/includes/language/Message.php:1000
01:39:31 /workspace/src/skins/chameleon/src/Components/NavMenu.php:201
01:39:31 /workspace/src/skins/chameleon/src/Components/NavMenu.php:58
01:39:31 /workspace/src/skins/chameleon/tests/phpunit/Unit/Components/GenericComponentTestCase.php:85
01:39:31 
01:39:31 4) Skins\Chameleon\Tests\Unit\Components\SearchBarTest::testGetHtmlwithEmptyElement
01:39:31 MWException: Unable to determine IP.
01:39:31 
01:39:31 /workspace/src/includes/WebRequest.php:1263
01:39:31 /workspace/src/includes/user/User.php:2149
01:39:31 /workspace/src/includes/user/UserOptionsManager.php:547
01:39:31 /workspace/src/includes/user/UserOptionsManager.php:455
01:39:31 /workspace/src/includes/user/UserOptionsManager.php:124
01:39:31 /workspace/src/includes/user/User.php:2756
01:39:31 /workspace/src/includes/context/RequestContext.php:351
01:39:31 /workspace/src/includes/language/Message.php:402
01:39:31 /workspace/src/includes/language/Message.php:1305
01:39:31 /workspace/src/includes/language/Message.php:1000
01:39:31 /workspace/src/includes/Linker.php:2114
01:39:31 /workspace/src/includes/Linker.php:2325
01:39:31 /workspace/src/skins/chameleon/src/Components/SearchBar.php:58
01:39:31 /workspace/src/skins/chameleon/tests/phpunit/Unit/Components/GenericComponentTestCase.php:85
01:39:31 
01:39:31 5) Skins\Chameleon\Tests\Unit\Menu\MenuFromLinesTest::testBuildEmptyMenu
01:39:31 Error: Call to a member function getVal() on null
01:39:31 
01:39:31 /workspace/src/includes/Title.php:2247
01:39:31 /workspace/src/skins/chameleon/src/Menu/MenuFromLines.php:211
01:39:31 /workspace/src/skins/chameleon/src/Menu/MenuFromLines.php:198
01:39:31 /workspace/src/skins/chameleon/src/Menu/MenuFromLines.php:165
01:39:31 /workspace/src/skins/chameleon/src/Menu/MenuFromLines.php:130
01:39:31 /workspace/src/skins/chameleon/src/Menu/MenuFromLines.php:88
01:39:31 /workspace/src/skins/chameleon/src/Menu/MenuFromLines.php:68
01:39:31 /workspace/src/skins/chameleon/tests/phpunit/Unit/Menu/MenuFromLinesTest.php:121

I created a pull request for both Bootstrap and Chameleon to convert to using MediaWiki test classes and bootstrapping: https://github.com/ProfessionalWiki/Bootstrap/pull/36 and https://github.com/ProfessionalWiki/chameleon/pull/163. The tests pass on travis. It would be good to see if https://gerrit.wikimedia.org/r/c/mediawiki/skins/chameleon/+/597812 would pass CI with the changes.

Reedy added a comment.EditedMay 30 2020, 2:33 AM

Fwiw, you can push them to gerrit too if you add a remote for gerrit

git push gerrit HEAD:refs/for/master

And then see the patches running in our CI

Obviously we won’t merge them, but helps test it here too

Thanks. I will try that next time. Unfortunately, this did not fix the errors in CI. Are you still getting errors testing locally? I am not getting any errors locally.

Or did you not push the update to gerrit? If not, I will do so.

Fwiw, you can push them to gerrit too if you add a remote for gerrit

git push gerrit HEAD:refs/for/master

When I try that I get the following error message:

remote: error: commit d30920b: email address noreply@github.com is not registered in your account, and you lack 'forge committer' permission.

Am I missing some permission on gerrit that I need in order to do this?

Maybe it is as simple as that I am not in the skin-chameleon and extension-Bootstrap gerrit groups.

Reedy added a comment.May 31 2020, 5:18 PM

Maybe it is as simple as that I am not in the skin-chameleon and extension-Bootstrap gerrit groups.

Yeah, probably. But I can't add you unfortunately, needs a gerrit admin

I'm not sure why it's seemingly trying to push a noreply@github.com commit though...

Right, the "noreply@github.com" bit has me confused as well. I'm obviously not pulling the changes from the github repo correctly into my local clone of the gerrit repo. What steps do you use to pull from github?

Reedy added a comment.May 31 2020, 5:37 PM
$ git remote -v
gerrit	ssh://reedy@gerrit.wikimedia.org:29418/mediawiki/skins/chameleon (fetch)
gerrit	ssh://reedy@gerrit.wikimedia.org:29418/mediawiki/skins/chameleon (push)
origin	https://github.com/ProfessionalWiki/chameleon.git (fetch)
origin	https://github.com/ProfessionalWiki/chameleon.git (push)
reedy	git@github.com:reedy/chameleon.git (fetch)
reedy	git@github.com:reedy/chameleon.git (push)

master tracks origin/master, so I've just been doing git push gerrit master for mirroring purposes, not trying to push a changeset to gerrit though

Have you done git fetch --all after adding the remote?

But if I try and push to gerrit for normal... (I generated the Change-Id on another repo rather than installing the hook)

https://gerrit.wikimedia.org/r/#/c/mediawiki/skins/chameleon/+/600952/

reedy@ubuntu64-web-esxi:~/chameleon$ nano README.md 
reedy@ubuntu64-web-esxi:~/chameleon$ git commit -a -m "Test"
[master 919c9ba] Test
 1 file changed, 2 insertions(+)
reedy@ubuntu64-web-esxi:~/chameleon$ git push noreply@github.com^C
reedy@ubuntu64-web-esxi:~/chameleon$ git push gerrit HEAD:refs/for/master
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 2 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 289 bytes | 289.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2)
remote: Processing changes: refs: 1, done    
remote: commit 919c9ba: ERROR: missing Change-Id in message footer
remote: 
remote: Hint: to automatically insert a Change-Id, install the hook:
remote:   gitdir=$(git rev-parse --git-dir); scp -p -P 29418 reedy@gerrit.wikimedia.org:hooks/commit-msg ${gitdir}/hooks/
remote: and then amend the commit:
remote:   git commit --amend
remote: 
To ssh://gerrit.wikimedia.org:29418/mediawiki/skins/chameleon
 ! [remote rejected] HEAD -> refs/for/master (commit 919c9ba: missing Change-Id in message footer)
error: failed to push some refs to 'ssh://reedy@gerrit.wikimedia.org:29418/mediawiki/skins/chameleon'
reedy@ubuntu64-web-esxi:~/chameleon$ git rev-parse --git-dir
.git
reedy@ubuntu64-web-esxi:~/chameleon$ git commit --amend -m "Test
> 
> Change-Id: Id0fccf6e13c702a1a0aa5225ccbec065226f752a"
[master 787b530] Test
 Date: Sun May 31 18:34:07 2020 +0100
 1 file changed, 2 insertions(+)
reedy@ubuntu64-web-esxi:~/chameleon$ git log -n 1
commit 787b53013f684bd2eb533943c9b013a0254dd721 (HEAD -> master)
Author: Reedy <reedy@wikimedia.org>
Date:   Sun May 31 18:34:07 2020 +0100

    Test
    
    Change-Id: Id0fccf6e13c702a1a0aa5225ccbec065226f752a
reedy@ubuntu64-web-esxi:~/chameleon$ git push gerrit HEAD:refs/for/master
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 2 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 333 bytes | 333.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2)
remote: Processing changes: new: 1, done    
remote: 
remote: New Changes:
remote:   https://gerrit.wikimedia.org/r/#/c/mediawiki/skins/chameleon/+/600952 Test
remote: 
To ssh://gerrit.wikimedia.org:29418/mediawiki/skins/chameleon
 * [new branch]      HEAD -> refs/for/master
reedy@ubuntu64-web-esxi:~/chameleon$

I'm wondering if the "noreply@github.com" is just a red herring since I don't have appropriate access to the repo for pushing a change set using HEAD:refs/for/master. I don't see anything significantly different in what I'm am doing from above. I don't get the Change-Id error, but that's probably because I get the other error before it gets that far.

I had not tried pushing directly for mirroring. Since Bootstrap on gerrit is behind github, I tried git push gerrit master and got the expected access denied correctly listing my username.

I added T254106 to request permissions, if you'd like to comment.

I half wonder if it's because it's not using the proper MW test classes/bootstrapping

I updated the code to use MW test classes/bootstrappping (see https://gerrit.wikimedia.org/r/c/mediawiki/skins/chameleon/+/600986), but it does not fix the problem.

Given that T243006 and T237032 are exhibiting the same issue, this does not appear to be a Chameleon issue. Interesting that phpunit has no issues in my local environment, but fails in yours and in Wikimedia CI.

Reedy closed this task as Declined.Tue, Jun 9, 2:04 PM

We don't need Chameleon going forward