The idea here is to use a tool such as PhantomJS (a headless WebKit browser) to hit a bunch of pre-defined URLs and report JavaScript errors.

In addition to strictly checking errors, checking fetches of content on domains outside of Wikimedia's control (e.g., Google Analytics) would also be nice.

http://tools.wmflabs.org/legobot/wmf_js_scanner.js checks for any JS errors being thrown on the main page (as an anonymous user) as well as the inclusion of any external resources.

Ideally such a tool would be able to login, visit a few pages (including ?action=edit), enable and disable individual gadgets and change skins.

(In reply to MZMcBride from comment #0)

...
testing als
ERROR: ReferenceError: Can't find variable: ta
TRACE:
->
https://als.wikipedia.org/w/index.php?title=MediaWiki:If-pt-login.
js&action=raw&ctype=text/javascript&dontcountme=s: 9
...

FIXED:
https://als.wikipedia.org/wiki/MediaWiki:If-pt-login.js?diff=584752

here is my version used for error checking:
https://github.com/eranroz/wiki-js-error-log/blob/master/testwiki.js

Seems we have two versions :-]  If you guys could unify the effort and have the resulting code hosted in Gerrit, I would be more than happy to write a Jenkins job that runs the script once per hour and spam some place (mail/IRC...).

There's also the VE production browser test that ensures that all wikis with VisualEditor installed actually let you get to VE successfully:

https://integration.wikimedia.org/ci/view/BrowserTests/job/browsertests-VisualEditor-production-linux-firefox-sauce/

(The fy.wikipedia.org one is just a regular test suite failure; the kk.wikipedia.org one looks like a local sysop has broken JS there.)

The webperf project has an asset-check deamon (see "Webperf" on Wikitech) that frequently (every 5 minutes) views a bunch of different endpoints in a PhantomJS browser and collects metrics. We can extend this to cover any metrics needed here.

It currently publishes the metrics to Ganglia only which isn't very accessible. See T105354 for moving this to Graphite so we can build a dashboard for it and just in general to allow people to query the data more easily.

See the checkAssets() in webperf/asset-check.js for an overview of the data currently collected.

Looks like this is more something for Jenkins to run than something that needs production server access...

@Krinkle mentioned that PhantomJS is based on a completely outdated Webkit version that no more reflect the current browsers hitting the sites. So I guess we can rephrase this task as being a smoke test for the production sites using modern browsers (chromium/firefox).

We have a bunch of browser tests hitting production already ( https://integration.wikimedia.org/ci/view/BrowserTests/view/-Dashboard/ ) and even a test suite for MediaWiki core although it is only hitting beta https://integration.wikimedia.org/ci/view/BrowserTests/view/Core/

Potentially we could come up with a set of tests defined somewhere (maybe integration/config) that are a shared responsibility, run continuously and spam everyone whenever something fails. Will need to describe what kind of smoke tests we are interested to exercise though.

See also T101497: [Story] Run browsertests regularly on test.wikidata.org via Jenkins

In T71519#1567266, @hashar wrote:

@Krinkle mentioned that PhantomJS is based on a completely outdated Webkit version

Yeah, we're rewriting asset-check on top of Chromium for that reason. Similar to what I did with our QUnit infrastructure in Jenkins. The APIs are similar though, so it's not that much work. See T106770.

Replaced "PhantomJS" by "automated browser tests" in the task summary.

Reading through the comments here and trying to summarize the perfect world:

• run tests on all Wikimedia sites (gather from sitematrix)
• run tests on mainpage, editing page etc (via predefined urls), with debug=true
• allow test to be logged in
• enable / cover all available gadgets on the site
• cover MediaWiki:common.js on the site
• cover skins on the site
• maybe cover stuff like MediaWiki:Mobile.js or MediaWiki:Minerva.js or MediaWiki:Vector.js ?) on the site
• cover detecting code that calls 3rd party domains

I assume there is consensus to run such a test as a Jenkins job?

Also wondering if this could potentially be a Hackathon project. If e.g. legoktm's script can be used as a base.

For the records, aforementioned https://integration.wikimedia.org/ci/view/BrowserTests/view/Core/ and https://integration.wikimedia.org/ci/view/BrowserTests/view/-Dashboard/ are 404s nowadays. No idea if similar things are available on https://integration.wikimedia.org/ci/view/Default/ ...

...and mentioning T106770: Rewrite asset-check on top of the Chromium (instead of PhantomJS) here.
Not sure if that is a hard requirement and should be a dependency (subtask) here though

Change 412618 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/audit-resources@master] Initial commit

https://gerrit.wikimedia.org/r/412618

Change 412618 merged by Legoktm:
[integration/audit-resources@master] Initial commit

https://gerrit.wikimedia.org/r/412618

OK, so I opted for the initial solution that was proposed, using phantomjs to go over a bunch of URLs. Except instead of phantomjs, it's using slimerjs which has roughly the same API, but uses Firefox. It currently generates a per-wiki debug log, and then a single "error" log that contains any external resource loading violations plus JavaScript errors. I'd like to split those into two logs. We can have jenkins run this daily, and email to lists of people depending on what type of error it is (external resource or JS exception).

Using 8 threads for each wiki's home page on the default skin took about 15 minutes total to run.

Thanks @Legoktm well done and daily jenkins job is good idea to track it regularly. you can mark it as resolved.

(and some words to end this phab task from where it started T71445 : we should think how to use this utility as part of community "code review" process - so if we can trigger the job on wiki edit in .js files or document it somewhere as part of "community code review" process )

Change 413684 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/config@master] Add docker image to run audit-resources

https://gerrit.wikimedia.org/r/413684

In T71519#3982243, @eranroz wrote:

Thanks @Legoktm well done and daily jenkins job is good idea to track it regularly. you can mark it as resolved.

(and some words to end this phab task from where it started T71445 : we should think how to use this utility as part of community "code review" process - so if we can trigger the job on wiki edit in .js files or document it somewhere as part of "community code review" process )

Integrate with https://www.mediawiki.org/wiki/Review_queue? (not sure how official/followed that is.)

In T71519#4000631, @Elitre wrote:

In T71519#3982243, @eranroz wrote:

Thanks @Legoktm well done and daily jenkins job is good idea to track it regularly. you can mark it as resolved.

(and some words to end this phab task from where it started T71445 : we should think how to use this utility as part of community "code review" process - so if we can trigger the job on wiki edit in .js files or document it somewhere as part of "community code review" process )

Integrate with https://www.mediawiki.org/wiki/Review_queue? (not sure how official/followed that is.)

IMO this page is more targeted for extension developers with different skills (git/gerrit/php). I added documentation here: https://www.mediawiki.org/w/index.php?title=Gadget_kitchen&type=revision&diff=2725392&oldid=2718404 I

@Legoktm - I couldn't find the relevant job in https://integration.wikimedia.org/ci - can you please add a link there ("where??")

Change 413684 merged by jenkins-bot:
[integration/config@master] Add docker image to run audit-resources

https://gerrit.wikimedia.org/r/413684

In T71519#4002895, @eranroz wrote:

IMO this page is more targeted for extension developers with different skills (git/gerrit/php). I added documentation here: https://www.mediawiki.org/w/index.php?title=Gadget_kitchen&type=revision&diff=2725392&oldid=2718404 I

@Legoktm - I couldn't find the relevant job in https://integration.wikimedia.org/ci - can you please add a link there ("where??")

It wasn't actually being run yet...the jenkins job is now https://integration.wikimedia.org/ci/job/audit-resources/ and I just kicked off a test run.

So https://integration.wikimedia.org/ci/job/audit-resources/ has been running for 2 years now. What else do we need/want?

I think in terms of implementation that's probably good enough.

Does its output go to something we can alert on? And/or does it tell someone/something when it fails?

In the meanwhile I noticed that the current Jenkins job cited here is failing for all the wikis:

aa.wikibooks.org.log

ERROR: aa.wikibooks.org: JSON errorGecko error: it seems /usr/bin/firefox is not compatible with SlimerJS.
See Gecko version compatibility. If version is correct, launch slimerjs
with --debug=true to see Firefox error message
from (https://aa.wikibooks.org/wiki/Main_Page)

en.wikipedia.org.log

ERROR: en.wikipedia.org: JSON errorGecko error: it seems /usr/bin/firefox is not compatible with SlimerJS.
See Gecko version compatibility. If version is correct, launch slimerjs
with --debug=true to see Firefox error message
 from (https://en.wikipedia.org/wiki/Main_Page)

...
https://integration.wikimedia.org/ci/job/audit-resources/lastSuccessfulBuild/artifact/log/

:( unfortunately slimerjs is basically dead at this point. I should rewrite it on top of selenium.