Page MenuHomePhabricator

mediawiki-extensions-qunit-jessie tests are failing
Closed, ResolvedPublic

Description

Steps to reproduce

Actual results

  • It is failing with
16:56:04 18 12 2016 16:56:03.129:DEBUG [launcher]: Chrome (id 41038593) captured in 1.288 secs
16:56:04 18 12 2016 16:56:03.149:DEBUG [middleware:karma]: custom files null null
16:56:04 18 12 2016 16:56:03.149:DEBUG [middleware:karma]: Serving static request /context.html
16:56:04 18 12 2016 16:56:03.154:DEBUG [web-server]: serving: /home/jenkins/workspace/mediawiki-extensions-qunit-jessie/src/node_modules/karma/static/favicon.ico
16:56:04 18 12 2016 16:56:03.156:DEBUG [web-server]: serving: /home/jenkins/workspace/mediawiki-extensions-qunit-jessie/src/node_modules/karma/static/context.html
16:56:04 18 12 2016 16:56:03.169:DEBUG [middleware:source-files]: Requesting /base/node_modules/qunitjs/qunit/qunit.js?28e1c466df9848ed4d3a9fe432636b5a198b6c3e /
16:56:04 18 12 2016 16:56:03.169:DEBUG [middleware:source-files]: Fetching /home/jenkins/workspace/mediawiki-extensions-qunit-jessie/src/node_modules/qunitjs/qunit/qunit.js
16:56:04 18 12 2016 16:56:03.172:DEBUG [middleware:source-files]: Requesting /base/node_modules/karma-qunit/lib/adapter.js?2cc5e3594a3b8ae53e8218929324788866ad223b /
16:56:04 18 12 2016 16:56:03.173:DEBUG [middleware:source-files]: Fetching /home/jenkins/workspace/mediawiki-extensions-qunit-jessie/src/node_modules/karma-qunit/lib/adapter.js
16:56:04 18 12 2016 16:56:03.175:DEBUG [web-server]: serving (cached): /home/jenkins/workspace/mediawiki-extensions-qunit-jessie/src/node_modules/qunitjs/qunit/qunit.js
16:56:04 18 12 2016 16:56:03.178:DEBUG [web-server]: serving (cached): /home/jenkins/workspace/mediawiki-extensions-qunit-jessie/src/node_modules/karma-qunit/lib/adapter.js
16:56:04 18 12 2016 16:56:03.180:DEBUG [web-server]: serving: /home/jenkins/workspace/mediawiki-extensions-qunit-jessie/src/node_modules/karma/static/context.js

16:56:07 Chrome 55.0.2883 (Linux 0.0.0) ERROR
16:56:07   Script error.
16:56:07 
16:56:07 
16:56:07 18 12 2016 16:56:06.141:DEBUG [karma]: Run complete, exiting.
16:56:07 18 12 2016 16:56:06.142:DEBUG [launcher]: Disconnecting all browsers

16:56:07 18 12 2016 16:56:06.251:DEBUG [launcher]: Process Chrome exited with code 0
16:56:07 18 12 2016 16:56:06.251:DEBUG [temp-dir]: Cleaning temp dir /home/jenkins/tmpfs/jenkins-0/karma-41038593
16:56:07 18 12 2016 16:56:06.292:DEBUG [launcher]: Finished all browsers
16:56:07 Warning: Task "karma:main" failed. Use --force to continue.

Note: Strange the last merged patch https://gerrit.wikimedia.org/r/#/c/324959/ passed but doing recheck's on other patches are failing.

Looking at the build history:

$ grep  'Script error'  /srv/jenkins/builds/*qunit*/*/log|cut -d\/  -f5|uniq -c
      3 mediawiki-core-qunit-jessie
     70 mediawiki-extensions-qunit-jessie

mediawiki-core-qunit-jessie runs against mediawiki/core and has no dependencies beside mediawiki/vendor. The failing builds are:

TimeGerritConsoleChromium
Dec 12 16:05 UTC325049/139842Chrome 53.0.2785
Dec 12 16:14:41 UTC325049/149843Chrome 53.0.2785
Dec 16 21:47:08 UTC325049/1510154Chrome 55.0.2883

Note that Gerrit change 325049 hasn't been merged. I have rebuild 9842 twice (in Jenkins UI with the Rebuild link in the web interface) and it failed with the same error. So it seems those patches manage to reproduce the issue at hand.

I have triggered the build against the master branch and it passed three times  10304  10305  10306.

Details

Related Gerrit Patches:
mediawiki/extensions/Wikidata : REL1_27Remove warning about missing site ID
integration/config : masterqunit: Remove obsolete 'tac|tac' hack
mediawiki/extensions/Wikibase : REL1_28Remove warning about missing site ID
mediawiki/extensions/Wikidata : REL1_28Remove warning about missing site ID
mediawiki/extensions/Wikibase : wmf/1.29.0-wmf.7Remove warning about missing site ID
mediawiki/extensions/Wikidata : wmf/1.29.0-wmf.5.1Remove warning about missing site ID
mediawiki/extensions/Wikidata : wmf/1.29.0-wmf.7Remove warning about missing site ID
mediawiki/extensions/Wikibase : masterRemove warning about missing site ID
mediawiki/extensions/Wikidata : masterRemove warning about missing site ID
mediawiki/core : master[TEST] test for CI qunit breakage
integration/config : masterTemporarily make mediawiki-extensions-qunit-jessie non voting
operations/puppet : productionContint: Downgrade Chromium to 53.0.2785
integration/config : masterCapture qunit curl commands + log JavaScriptTest output
mediawiki/core : master(DO NOT SUBMIT) karma without browser security

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

The error is different, but symptom might be the same. Please read https://sentry.io/answers/javascript-script-error/ and my reply:

The ajax calls fail with code http and {responseText: '', status: 0, statusText: 'error'}. Googling on the internet people say CSR issues can cause this, but we are using ForeingApi in anonymous mode.

Oh thanks, I guess something big changed between chrome 53 and 54 to cause the cross origin error to spread.

Should we set Access-Control-Allow-Origin: * in apache?

Should we make the qunit test non voting for now?

Is this a bug in chrome or does something need to be updated in mw to fix this?

Another user reported this problem on irc

<SMalyshev> is something wrong with qunit tests? I'm getting failures on patches where everything worked before, e.g. https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit-jessie/19472/console

He says it breaks all merges in CirrusSearch

<SMalyshev> paladox: thank you! looks like it breaks all merges for CirrusSearch :( will wait for resolution then

Change 328217 had a related patch set (by Paladox) published:
Downgrade Chromium to 53.0.2785

https://gerrit.wikimedia.org/r/328217

Paladox added a comment.EditedDec 19 2016, 8:37 PM

Oh wow stretch is using an old version of chromium https://packages.debian.org/stretch/chromium should apt-get from that source and then pin it.

Change 328238 had a related patch set uploaded (by Paladox):
Temporarily make mediawiki-extensions-qunit-jessie non voting

https://gerrit.wikimedia.org/r/328238

Tgr added a subscriber: Tgr.EditedDec 20 2016, 12:21 AM

karma/static/context.js was removed in v1.1.0. Chances are karma is serving up some HTML error page and Chrome is trying to execute it as HTML an failing (and giving this super unhelpful message - why are the assets loaded cross-domain?)

Tgr added a comment.Dec 20 2016, 1:05 AM

Probably unrelated Wikidata errors in the CI log filed as T153729: CI tests throw warning "Cannot find site mywiki in sites table".

Tgr added a comment.Dec 20 2016, 3:51 AM

Apparently not unrelated after all.

This is still breaking. https://gerrit.wikimedia.org/r/#/c/328027/ failed to merge twice today, both times due to "Script error" from the qunit job even though the change doesn't touch any JS. https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit-jessie/19522/console

hashar added a subscriber: hashar.Dec 20 2016, 9:46 AM

I haven't noticed that task yesterday, will try to reproduce. That looks like a race condition in one of the test suite :( Gotta find out whether it is a regression in Chromium, a change in the CI infra, an issue with mediawiki/core or one of the extensions :-/

@Tgr the "Cannot find site mywiki in sites table" has been going on for ages. I don't think it is related at all.

Looking at the build history:

$ grep  'Script error'  /srv/jenkins/builds/*qunit*/*/log|cut -d\/  -f5|uniq -c
      3 mediawiki-core-qunit-jessie
     70 mediawiki-extensions-qunit-jessie

mediawiki-core-qunit-jessie runs against mediawiki/core and has no dependencies beside mediawiki/vendor. The failing builds are:

TimeGerritConsoleChromium
Dec 12 16:05 UTC325049/139842Chrome 53.0.2785
Dec 12 16:14:41 UTC325049/149843Chrome 53.0.2785
Dec 16 21:47:08 UTC325049/1510154Chrome 55.0.2883

Note that Gerrit change 325049 hasn't been merged. I have rebuild 9842 twice (in Jenkins UI with the Rebuild link in the web interface) and it failed with the same error. So it seems those patches manage to reproduce the issue at hand.

I have triggered the build against the master branch and it passed three times  10304  10305  10306.

Looking at the build history:

$ grep  'Script error'  /srv/jenkins/builds/*qunit*/*/log|cut -d\/  -f5|uniq -c
      3 mediawiki-core-qunit-jessie
     70 mediawiki-extensions-qunit-jessie

mediawiki-core-qunit-jessie runs against mediawiki/core and has no dependencies beside mediawiki/vendor. The failing builds are:

TimeGerritConsoleChromium
Dec 12 16:05 UTC325049/139842Chrome 53.0.2785
Dec 12 16:14:41 UTC325049/149843Chrome 53.0.2785
Dec 16 21:47:08 UTC325049/1510154Chrome 55.0.2883

Note that Gerrit change 325049 hasn't been merged. I have rebuild 9842 twice (in Jenkins UI with the Rebuild link in the web interface) and it failed with the same error. So it seems those patches manage to reproduce the issue at hand.
I have triggered the build against the master branch and it passed three times  10304  10305  10306.

Yep I find that strange as some of my recheck's a few days ago failed but then doing a recheck on one patch worked. So could the problem be when you touch a specific file?

Looking at the build history:

$ grep  'Script error'  /srv/jenkins/builds/*qunit*/*/log|cut -d\/  -f5|uniq -c
      3 mediawiki-core-qunit-jessie
     70 mediawiki-extensions-qunit-jessie

mediawiki-core-qunit-jessie runs against mediawiki/core and has no dependencies beside mediawiki/vendor. The failing builds are:

TimeGerritConsoleChromium
Dec 12 16:05 UTC325049/139842Chrome 53.0.2785
Dec 12 16:14:41 UTC325049/149843Chrome 53.0.2785
Dec 16 21:47:08 UTC325049/1510154Chrome 55.0.2883

Note that Gerrit change 325049 hasn't been merged. I have rebuild 9842 twice (in Jenkins UI with the Rebuild link in the web interface) and it failed with the same error. So it seems those patches manage to reproduce the issue at hand.
I have triggered the build against the master branch and it passed three times  10304  10305  10306.

I eventually broke my Apache/PHP local config and fought with karma/qunit configuration for the rest of the day. I am still not able to run the qunit tests locally to reproduce :-(

@hashar what about setting up a vm or using labs?

hashar updated the task description. (Show Details)Dec 20 2016, 4:07 PM

I have reproduced the Script error locally by using https://gerrit.wikimedia.org/r/#/c/325049/15

git fetch https://gerrit.wikimedia.org/r/mediawiki/core refs/changes/49/325049/15
git checkout -b 325049/15 FETCH_HEAD
grunt karma:main
20 12 2016 17:09:31.691:INFO [Chrome 55.0.2883 (Mac OS X 10.11.4)]: Connected on socket ...
20 12 2016 17:09:31.762:DEBUG [web-server]: serving (cached): /Users/amusso/projects/mediawiki/core/node_modules/karma-qunit/lib/adapter.js
Chrome 55.0.2883 (Mac OS X 10.11.4) ERROR
  Script error.
20 12 2016 17:09:35.247:DEBUG [karma]: Run complete, exiting.

Change 328396 had a related patch set uploaded (by Hashar):
(DO NOT SUBMIT) karma without browser security

https://gerrit.wikimedia.org/r/328396

Change 328396 abandoned by Hashar:
(DO NOT SUBMIT) karma without browser security

https://gerrit.wikimedia.org/r/328396

Eventually I made a new karma configuration that pass to Chrome --disable-web-security https://gerrit.wikimedia.org/r/#/c/328396/ . That let it interpret faulty javascript and get Karma to report the error:

Chrome 55.0.2883 (Mac OS X 10.11.4) ERROR
  Uncaught SyntaxError: Unexpected token <
  at http://localhost/index.php?title=Special:JavaScriptTest/qunit/export:1

Using https://gerrit.wikimedia.org/r/#/c/325049/15 , the request for Special:JavaScriptTest/qunit/export spurts a PHP notice:

<br />
<font size='1'><table class='xdebug-error xe-notice' dir='ltr' border='1' cellspacing='0' cellpadding='1'>
<tr><th align='left' bgcolor='#f57900' colspan="5"><span style='background-color: #cc0000; color: #fce94f; font-size: x-large;'>( ! )</span> Notice: Undefined index: rcdays in mediawiki/core/includes/Setup.php on line <i>285</i></th></tr>
...

Which is well, invalid Javascript. I have no idea why Karma does not manage to capture the error from Chrome console unless one pass --disable-web-security.


So my best assumption is that sometime a notice ends up being thrown in the qunit export. And maybe the wikibase sites misconfiguration could cause it. Maybe we can dig in the debug log attached to https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit-jessie/ and see whether some specific error are emitted on the failing builds :/

Change 328401 had a related patch set uploaded (by Hashar):
Capture qunit curl commands log JavaScriptTest output

https://gerrit.wikimedia.org/r/328401

https://gerrit.wikimedia.org/r/328401 Capture qunit curl commands + log JavaScriptTest output

That will do a curl on JavaScriptTest and attach the output to the build. The place I am working from is closing down though, so I can't not deploy it now :-\

Anomie added a subscriber: Anomie.Dec 20 2016, 8:09 PM

Change 328401 merged by jenkins-bot:
Capture qunit curl commands log JavaScriptTest output

https://gerrit.wikimedia.org/r/328401

I have updated the job mediawiki-extensions-qunit-jessie which now captures the output of:

curl --include /index.php/Special:BlankPage
curl --include /load.php?debug=true&modules=startup&only=scripts
curl --include /index.php?title=Special:JavaScriptTest/qunit/export

Probably unrelated Wikidata errors in the CI log filed as T153729: CI tests throw warning "Cannot find site mywiki in sites table".

You were right after all! I have seen at least one occurrence of /index.php?title=Special:JavaScriptTest/qunit/export that had "Cannot find site mywiki in sites table". And I guess that causes the web browser to bails out with the script error.

@Smalyshev hi, the patches @hashar did is for debugging purposes. You can force merge for now if you are sure your patch dosent break anything. 

I see in the error log this:

2016-12-20 23:29:55 ci-jessie-wikimedia-462382 jenkins_u0_mw: [43362324fd2c4695892da59f] [no req] ErrorException from line 309 of /home/jenkins/workspace/mediawiki-extensions-qunit-jessie/src/includes/debug/MWDebug.php: PHP Notice: Cannot find site mywiki in sites table [Called from Wikibase\Client\WikibaseClient::newSiteGroup in /home/jenkins/workspace/mediawiki-extensions-qunit-jessie/src/extensions/Wikidata/extensions/Wikibase/client/includes/WikibaseClient.php at line 734]

That is filled as T153729

But that has been warning has been there for 1+ year.

Tgr added a comment.Dec 20 2016, 11:40 PM

@Smalyshev https://gerrit.wikimedia.org/r/#/c/328326/3 will probably fix this, I am not familiar with the WMDE build setup though (how does a Wikibase patch get into the Wikidata repo?)

Tgr added a comment.Dec 20 2016, 11:44 PM

Well I guess if it works with Depends-On then it will work after merging it, too...

Tgr added a comment.EditedDec 20 2016, 11:49 PM

But that has been warning has been there for 1+ year.

It was there in the update.php run but not in wiki pages (you can see it in Special:BlankPage as well now). Apparently some change caused WikibaseClient::newSiteGroup() to be triggered on (some?) pageviews as well.

Change 328326 had a related patch set uploaded (by Gergő Tisza):
Remove warning about missing site ID

https://gerrit.wikimedia.org/r/328326

Change 328445 had a related patch set (by Paladox) published:
Remove warning about missing site ID

https://gerrit.wikimedia.org/r/328445

Change 328326 merged by jenkins-bot:
Remove warning about missing site ID

https://gerrit.wikimedia.org/r/328326

Change 328445 merged by jenkins-bot:
Remove warning about missing site ID

https://gerrit.wikimedia.org/r/328445

Change 328217 abandoned by Paladox:
Contint: Downgrade Chromium to 53.0.2785

https://gerrit.wikimedia.org/r/328217

Change 328238 abandoned by Paladox:
Temporarily make mediawiki-extensions-qunit-jessie non voting

https://gerrit.wikimedia.org/r/328238

Tgr closed this task as Resolved.Dec 21 2016, 1:02 AM
Tgr claimed this task.

Seems fixed. Not sure about the mediawiki-core-qunit-jessie tests mentioned in T153597#2889448 but that issue does not affect all core patches.

Change 328327 abandoned by Gergő Tisza:
[TEST] test for CI qunit breakage

https://gerrit.wikimedia.org/r/328327

It seems this still fails in https://gerrit.wikimedia.org/r/#/c/328113/
But passes in other patches.

Tgr added a comment.Dec 21 2016, 1:08 AM

(I still don't understand why https://gerrit.wikimedia.org/r/#/c/328327/ passed the tests, even though it depended on Wikibase and not Wikidata, but don't care sufficiently to spend time on it.)

Change 329774 had a related patch set uploaded (by Paladox):
Remove warning about missing site ID

https://gerrit.wikimedia.org/r/329774

Change 329775 had a related patch set uploaded (by Paladox):
Remove warning about missing site ID

https://gerrit.wikimedia.org/r/329775

Paladox reopened this task as Open.Dec 31 2016, 1:24 PM

Re opening as now REL1_28 and REL1_27 and possibly REL1_23 are failing now with the same error.

hashar lowered the priority of this task from Unbreak Now! to Normal.Jan 4 2017, 3:57 PM

(I still don't understand why https://gerrit.wikimedia.org/r/#/c/328327/ passed the tests, even though it depended on Wikibase and not Wikidata, but don't care sufficiently to spend time on it.)

That is worth investigating. Will give it a try eventually.

Re opening as now REL1_28 and REL1_27 and possibly REL1_23 are failing now with the same error.

I guess on release branches we need another kind of job. The mediawiki-extensions* jobs that includes Wikibase should only be run against master and wmf branches. For REL branches, I guess we would just run it with whatever extensions are bundled in the tarball. That is a different task though.

Change 331533 had a related patch set uploaded (by Reedy):
Remove warning about missing site ID

https://gerrit.wikimedia.org/r/331533

Change 331533 merged by jenkins-bot:
Remove warning about missing site ID

https://gerrit.wikimedia.org/r/331533

Change 331542 had a related patch set uploaded (by Reedy):
Remove warning about missing site ID

https://gerrit.wikimedia.org/r/331542

Change 331542 merged by jenkins-bot:
Remove warning about missing site ID

https://gerrit.wikimedia.org/r/331542

Change 331548 had a related patch set uploaded (by Reedy):
Remove warning about missing site ID

https://gerrit.wikimedia.org/r/331548

Change 331548 merged by jenkins-bot:
Remove warning about missing site ID

https://gerrit.wikimedia.org/r/331548

Change 331571 had a related patch set uploaded (by Reedy):
Remove warning about missing site ID

https://gerrit.wikimedia.org/r/331571

Change 329774 merged by jenkins-bot:
Remove warning about missing site ID

https://gerrit.wikimedia.org/r/329774

Change 331571 merged by jenkins-bot:
Remove warning about missing site ID

https://gerrit.wikimedia.org/r/331571

This proposal is selected for the Developer-Wishlist voting round and will be added to a MediaWiki page very soon. To the subscribers, or proposer of this task: please help modify the task description: add a brief summary (10-12 lines) of the problem that this proposal raises, topics discussed in the comments, and a proposed solution (if there is any yet). Remember to add a header with a title "Description," to your content. Please do so before February 5th, 12:00 pm UTC.

Tgr updated the task description. (Show Details)Feb 4 2017, 10:45 AM

This task is about a specific issue that has been fixed. I don't see much point in adding it to the developer wishlist.

Should this be closed then?

Tgr closed this task as Resolved.Feb 21 2017, 7:34 PM

Change 345392 had a related patch set uploaded (by Krinkle):
[integration/config@master] qunit: Remove obsolete 'tac|tac' hack

https://gerrit.wikimedia.org/r/345392

Change 345392 merged by jenkins-bot:
[integration/config@master] qunit: Remove obsolete 'tac|tac' hack

https://gerrit.wikimedia.org/r/345392

Change 329775 merged by Hashar:
[mediawiki/extensions/Wikidata@REL1_27] Remove warning about missing site ID

https://gerrit.wikimedia.org/r/329775