Page MenuHomePhabricator

Run automated browser tests across set of Wikimedia wiki pages to ensure sane JavaScript
Open, LowPublic

Description

Ideas of things to scan for:

  • Any request resulting in an error response (404 Not Found, 500 Internal Error, etc.).
  • Uncaught JavaScript exceptions.
  • Network requests made to third-party domains (images, stylesheets, scripts, xhr, etc.)

From comment 13 at T71445: Implement a sane code-review process for MediaWiki JS/CSS pages on Wikimedia sites :

However it would be great to have automated tests running after changes in common.js and gadgets. Such general tests can at least validate no external resource is loaded, or no invalid syntax is added.
The simplest option is to run it outside MW (such as a script in labs that runs X times a day), but an extension to do it while editing would be even better...
BTW
I just runned a simple phantomjs script on all wikipedia languages to catch simple errors:
testing als
ERROR: ReferenceError: Can't find variable: ta
TRACE:
-> https://als.wikipedia.org/w/index.php?title=MediaWiki:If-pt-login.js&action=raw&ctype=text/javascript&dontcountme=s: 9
testing kk
ERROR: TypeError: 'null' is not an object (evaluating 'document.getElementById('mw-vector-current-variant').innerHTML=wgULS("Кирил","Latın","توتە")')
TRACE:
-> https://bits.wikimedia.org/kk.wikipedia.org/load.php?debug=false&lang=kk&modules=site&only=scripts&skin=vector&*: 9
anyone wants to fix them? :)

From comment 14 at T71445:

Example for errors across wikis
Maybe it is a bit off-topic but here are some cases of errors across wikis in simple read/edit transaction using phantomJS

See also:

Details

Reference
bz69519
Related Gerrit Patches:
integration/config : masterAdd docker image to run audit-resources
integration/audit-resources : masterInitial commit

Event Timeline

bzimport raised the priority of this task from to Low.Nov 22 2014, 3:28 AM
bzimport added a project: Quality-Assurance.
bzimport set Reference to bz69519.
bzimport added a subscriber: Unknown Object (MLST).

The idea here is to use a tool such as PhantomJS (a headless WebKit browser) to hit a bunch of pre-defined URLs and report JavaScript errors.

In addition to strictly checking errors, checking fetches of content on domains outside of Wikimedia's control (e.g., Google Analytics) would also be nice.

http://tools.wmflabs.org/legobot/wmf_js_scanner.js checks for any JS errors being thrown on the main page (as an anonymous user) as well as the inclusion of any external resources.

Ideally such a tool would be able to login, visit a few pages (including ?action=edit), enable and disable individual gadgets and change skins.

(In reply to MZMcBride from comment #0)

...
testing als
ERROR: ReferenceError: Can't find variable: ta
TRACE:
->
https://als.wikipedia.org/w/index.php?title=MediaWiki:If-pt-login.
js&action=raw&ctype=text/javascript&dontcountme=s: 9
...

FIXED:
https://als.wikipedia.org/wiki/MediaWiki:If-pt-login.js?diff=584752

Seems we have two versions :-]  If you guys could unify the effort and have the resulting code hosted in Gerrit, I would be more than happy to write a Jenkins job that runs the script once per hour and spam some place (mail/IRC...).

There's also the VE production browser test that ensures that all wikis with VisualEditor installed actually let you get to VE successfully:

https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-VisualEditor-production-linux-firefox-sauce/

(The fy.wikipedia.org one is just a regular test suite failure; the kk.wikipedia.org one looks like a local sysop has broken JS there.)

Krinkle updated the task description. (Show Details)Jul 9 2015, 2:45 PM
Krinkle added a subscriber: Krinkle.EditedJul 9 2015, 4:13 PM

The webperf project has an asset-check deamon (see "Webperf" on Wikitech) that frequently (every 5 minutes) views a bunch of different endpoints in a PhantomJS browser and collects metrics. We can extend this to cover any metrics needed here.

It currently publishes the metrics to Ganglia only which isn't very accessible. See T105354 for moving this to Graphite so we can build a dashboard for it and just in general to allow people to query the data more easily.

See the checkAssets() in webperf/asset-check.js for an overview of the data currently collected.

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJul 9 2015, 4:14 PM
hashar removed a subscriber: hashar.Jul 9 2015, 6:37 PM
Peter added a subscriber: Peter.Aug 19 2015, 8:18 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 19 2015, 8:18 PM

Looks like this is more something for Jenkins to run than something that needs production server access...

Krenair edited projects, added Continuous-Integration-Config; removed Jenkins.
hashar added a subscriber: hashar.Aug 24 2015, 1:41 PM

@Krinkle mentioned that PhantomJS is based on a completely outdated Webkit version that no more reflect the current browsers hitting the sites. So I guess we can rephrase this task as being a smoke test for the production sites using modern browsers (chromium/firefox).

We have a bunch of browser tests hitting production already ( https://integration.wikimedia.org/ci/view/BrowserTests/view/-Dashboard/ ) and even a test suite for MediaWiki core although it is only hitting beta https://integration.wikimedia.org/ci/view/BrowserTests/view/Core/

Potentially we could come up with a set of tests defined somewhere (maybe integration/config) that are a shared responsibility, run continuously and spam everyone whenever something fails. Will need to describe what kind of smoke tests we are interested to exercise though.

@Krinkle mentioned that PhantomJS is based on a completely outdated Webkit version

Yeah, we're rewriting asset-check on top of Chromium for that reason. Similar to what I did with our QUnit infrastructure in Jenkins. The APIs are similar though, so it's not that much work. See T106770.

hashar removed a subscriber: hashar.
Aklapper renamed this task from Run PhantomJS across set of Wikimedia wiki pages to ensure sane JavaScript to Run automated browser tests across set of Wikimedia wiki pages to ensure sane JavaScript.May 1 2017, 12:29 PM
Aklapper removed a subscriber: wikibugs-l-list.

Replaced "PhantomJS" by "automated browser tests" in the task summary.

Reading through the comments here and trying to summarize the perfect world:

  • run tests on all Wikimedia sites (gather from sitematrix)
  • run tests on mainpage, editing page etc (via predefined urls), with debug=true
  • allow test to be logged in
  • enable / cover all available gadgets on the site
  • cover MediaWiki:common.js on the site
  • cover skins on the site
  • maybe cover stuff like MediaWiki:Mobile.js or MediaWiki:Minerva.js or MediaWiki:Vector.js ?) on the site
  • cover detecting code that calls 3rd party domains

I assume there is consensus to run such a test as a Jenkins job?

Also wondering if this could potentially be a Hackathon project. If e.g. legoktm's script can be used as a base.

For the records, aforementioned https://integration.wikimedia.org/ci/view/BrowserTests/view/Core/ and https://integration.wikimedia.org/ci/view/BrowserTests/view/-Dashboard/ are 404s nowadays. No idea if similar things are available on https://integration.wikimedia.org/ci/view/Default/ ...

...and mentioning T106770: Rewrite asset-check on top of the Chromium (instead of PhantomJS) here.
Not sure if that is a hard requirement and should be a dependency (subtask) here though

Legoktm claimed this task.Feb 17 2018, 5:19 PM

Change 412618 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/audit-resources@master] Initial commit

https://gerrit.wikimedia.org/r/412618

Change 412618 merged by Legoktm:
[integration/audit-resources@master] Initial commit

https://gerrit.wikimedia.org/r/412618

OK, so I opted for the initial solution that was proposed, using phantomjs to go over a bunch of URLs. Except instead of phantomjs, it's using slimerjs which has roughly the same API, but uses Firefox. It currently generates a per-wiki debug log, and then a single "error" log that contains any external resource loading violations plus JavaScript errors. I'd like to split those into two logs. We can have jenkins run this daily, and email to lists of people depending on what type of error it is (external resource or JS exception).

Using 8 threads for each wiki's home page on the default skin took about 15 minutes total to run.

Thanks @Legoktm well done and daily jenkins job is good idea to track it regularly. you can mark it as resolved.

(and some words to end this phab task from where it started T71445 : we should think how to use this utility as part of community "code review" process - so if we can trigger the job on wiki edit in .js files or document it somewhere as part of "community code review" process )

Change 413684 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/config@master] Add docker image to run audit-resources

https://gerrit.wikimedia.org/r/413684

Thanks @Legoktm well done and daily jenkins job is good idea to track it regularly. you can mark it as resolved.
(and some words to end this phab task from where it started T71445 : we should think how to use this utility as part of community "code review" process - so if we can trigger the job on wiki edit in .js files or document it somewhere as part of "community code review" process )

Integrate with https://www.mediawiki.org/wiki/Review_queue? (not sure how official/followed that is.)

Thanks @Legoktm well done and daily jenkins job is good idea to track it regularly. you can mark it as resolved.
(and some words to end this phab task from where it started T71445 : we should think how to use this utility as part of community "code review" process - so if we can trigger the job on wiki edit in .js files or document it somewhere as part of "community code review" process )

Integrate with https://www.mediawiki.org/wiki/Review_queue? (not sure how official/followed that is.)

IMO this page is more targeted for extension developers with different skills (git/gerrit/php). I added documentation here: https://www.mediawiki.org/w/index.php?title=Gadget_kitchen&type=revision&diff=2725392&oldid=2718404 I

@Legoktm - I couldn't find the relevant job in https://integration.wikimedia.org/ci - can you please add a link there ("where??")

Masti added a subscriber: Masti.Mar 15 2018, 10:16 PM
Krinkle updated the task description. (Show Details)Mar 29 2018, 1:02 AM

Change 413684 merged by jenkins-bot:
[integration/config@master] Add docker image to run audit-resources

https://gerrit.wikimedia.org/r/413684

IMO this page is more targeted for extension developers with different skills (git/gerrit/php). I added documentation here: https://www.mediawiki.org/w/index.php?title=Gadget_kitchen&type=revision&diff=2725392&oldid=2718404 I
@Legoktm - I couldn't find the relevant job in https://integration.wikimedia.org/ci - can you please add a link there ("where??")

It wasn't actually being run yet...the jenkins job is now https://integration.wikimedia.org/ci/job/audit-resources/ and I just kicked off a test run.