Page MenuHomePhabricator

Jenkins: Assert no PHP errors (notices, warnings) were raised or exceptions were thrown
Closed, ResolvedPublic

Description

We currently already set up a web-facing instance of MediaWiki (during the qunit build). Let's extend our test suite by checking the error log before and after and consider any new entries a fatal build failure.

PHP notices, warnings, errors, exceptions etc.

The primary motivation here is to catch errors triggered during load.php requests because those are almost invisible in general, and even more so when all we see is the QUnit callback through PhantomJS.

There've been a few commits lately in master that messed something up in the registration of javascript modules but nothing caught it. They are only reliably detectable at run-time.

And of course it helps catching lots of other errors.

Since we already have a policy of not tolerating php errors, this should be trivial to enable. Though we do have a few known php errors, none of them trigger on regular usage (open wiki page, request load.php, run qunit tests etc.).

Related Objects

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 1:32 AM
bzimport set Reference to bz48002.
Krinkle created this task.May 2 2013, 3:28 PM

Recommended approach:

  • Rename mediawiki-core-qunit to mediawiki-core-browsertest
  • Build steps:
    • Install MediaWiki
    • curl /wiki/Special:BlankPage (anything not HTTP 200 is error)
    • grunt qunit /wiki/Special:JavaScriptTest/qunit
    • assert apache/php error log is empty or unchanged
hashar added a comment.May 6 2013, 8:50 AM

Another way would be to override the PHP handler to log all the exceptions, format them in a nice format (such as HTML?) and then use a post build action in Jenkins to publish the report.

HTML is difficult to work with and format from plain bash (or most programming languages for that matter). Plus we're striving towards getting rid of Jenkins formatters to only use plain text reporters (with regards to build reports static caching).

I'd say plain text is more than clear enough. We're all familiar with PHP's error_log format.

We just need:

  • php error log to go to a separate file for the qunit.localhost requests
  • Jenkins needs to have permission to truncate this file
  • Apache and/or PHP needs to not keep the handle open on this file, so that when each job truncates the file, it really does truncates it.

(In reply to comment #3)

HTML is difficult to work with and format from plain bash (or most
programming
languages for that matter). Plus we're striving towards getting rid of
Jenkins
formatters to only use plain text reporters (with regards to build reports
static caching).
I'd say plain text is more than clear enough. We're all familiar with PHP's
error_log format.
We just need:

  • php error log to go to a separate file for the qunit.localhost requests
  • Jenkins needs to have permission to truncate this file
  • Apache and/or PHP needs to not keep the handle open on this file, so that

when each job truncates the file, it really does truncates it.

Also, even if we'd make a custom HTML reporter or something, we still have these same problems. So let's deal with that first.

The important part in my reply was using a custom PHP error/exception handler. It seems to be easier to tweak and more robust.

How would you handle several jobs running QUnit tests in parallel ?

Parsing the apache log is a smart idea to catch all issues that happens during run time and we can't catch using static analysis. But since all QUnit jobs are hitting the same virtualhost, you will end up with erros from different extensions in the apache log.

Instead we could use php auto_prepend_file that would let you inject whatever PHP at the top of any PHP file being included (such as index.php / api.php ... )

http://www.php.net/manual/fr/ini.core.php#ini.auto-prepend-file

That let you write a small PHP File that fetch the Jenkins env variable override the error_log and have something a bit more robust and easy to tweak.

Will have to make sure the injected code is only executed when PHP require a MediaWiki entry point.

To clarify, we'd change the php init error_log setting to point to a file inside the env.WORKSPACE directory from Jenkins so that we have the errors on a per job base.

PS2: Depending on when we get to implementation of this, if we have jobs running in Vagrant we don't need any of that hackery since the VM would be dedicated to that one job only.

So we need to:

  1. Add configuration to the MediaWiki install to enable logging of errors extensively to the log/ directory in the workspace.
  1. Find a way to either have Apache send its error log (which includes PHPs stderr) to a local directory. Assuming that's too complicated as long as we're using the same Apache and vhost for all jobs, lets forget about Apache issues and instead find a way to hijack php's error log.

Antoine has started on #1 by setting up logging. We're not yet asserting them, but at least we have them available now from the build artifacts.

Antoine suggested we implement #2 by prepending php ini_set (or something like that) to all web entry points. For maintenance scripts we don't need to since we already have stderr there, and for any mediawiki internal things, those are logged to log/ which applies to both web and cli runs.

See also PHP ini setting auto_prepend_file I mentioned in comment #6. Could be pointed to some custom error handler under /srv/slave-scripts/mediawiki (which is integration/jenkins.git).

Since I added a MediaWiki log group for php errors (notices and warnings) we no longer need to get them via the apache2 log or from php sterr. We can capture them directly, and assert them to be empty.

Let's set up logGroup[error] and logGroup[exception] for -qunit (www, frontend) and -phpunit (cli, installer, backend) jobs and assert them to be empty.

The only thing this still won't catch is notices and warnings very early on in MediaWiki run time. Those would still be seen (for CLI scripts, these would show up in the Jenkins job console output, and for index.php/load.php the qunit job output would show these), but not detected automatically.

But that's an edge case we can try and catch later. Right now we're not catching any of them.

Change 174810 had a related patch set uploaded by Krinkle:
mediawiki: Configure 'error' and 'exception' log groups

https://gerrit.wikimedia.org/r/174810

Krinkle set Security to None.
demon removed a subscriber: demon.Dec 16 2014, 6:05 PM
Krinkle added a comment.EditedFeb 21 2015, 3:13 AM

Per T45086, Reedy and I implemented ability to log PHP warnings and notices.

They've been enabled for Jenkins builds as of f65721e16c009 and are available for viewing via Build artefacts.

Next step here is to assert that (some of?) these logs are empty at the end of a build.

$wgDBerrorLog = "$wmgMwLogDir/mw-dberror.log";
$wgDebugLogGroups['exception'] = "$wmgMwLogDir/mw-exception.log";
$wgDebugLogGroups['error'] = "$wmgMwLogDir/mw-error.log";
Krinkle updated the task description. (Show Details)Mar 4 2015, 8:58 PM

Known errors to fix first:

SqlBagOStuff::set/single-row 5 database is locked REPLACE INTO objectcache (T89180)
Found every run of mediawiki-core-qunit

Flow\Exception\InvalidDataException: Invalid metadata for revision (T89691)
Found 36x in mw-exception.log from mediawiki-extensions-hhvm and mediawiki-extensions-zend

DatabaseSqlite::replace/single-row 19 NOT NULL constraint failed: unittest_module_deps.md_module (T91567)
Found in mw-dberror.log on every run of mediawiki-phpunit-hhvm

DatabaseSqlite::replace/single-row 19 unittest_module_deps.md_module may not be NULL REPLACE INTO unittest_module_deps (T91567)
Found in mw-dberror.log in every run of mediawiki-phpunit-zend

Krinkle removed Krinkle as the assignee of this task.Jun 4 2015, 12:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 17 2015, 5:30 AM
Krinkle added a comment.EditedOct 1 2016, 12:08 AM

It took three years, but we're error-log-free on MediaWiki core!

mw-dberror.log, mw-exception.log and mw-error.log (src) are empty (not lazy-created) as of current master MediaWiki core - since rMW094105f017a1: Fix wfLogDBError channel redirect logic

The core runs are finally clear. Let's enforce it there ASAP to avoid new regressions.

greg added a subscriber: greg.Feb 13 2017, 11:41 PM

It took three years, but we're error-log-free on MediaWiki core!
mw-dberror.log, mw-exception.log and mw-error.log (src) are empty (not lazy-created) as of current master MediaWiki core - since rMW094105f017a1: Fix wfLogDBError channel redirect logic

The core runs are finally clear. Let's enforce it there ASAP to avoid new regressions.

They still seem clear, what's stopping this? Writing an (quick email) announcement would be good, @hashar ?

Recent runs:

Change 364935 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] resourceloader: Avoid mw-exception log spam from PHPUnit tests

https://gerrit.wikimedia.org/r/364935

Change 364935 merged by jenkins-bot:
[mediawiki/core@master] resourceloader: Avoid mw-exception log spam from PHPUnit tests

https://gerrit.wikimedia.org/r/364935

OK. mw-error.log and mw-exception.log are empty again for phpunit and qunit jobs. Let's stay on top of this and enforce it soon? (@hashar, @greg)

@Krinkle definitely. Will probably want to limit it to the master branch for a while.

Would it be all about asserting the log files are empty and if not fail the build? If so that sounds easy to implement.

@Krinkle definitely. Will probably want to limit it to the master branch for a while.
Would it be all about asserting the log files are empty and if not fail the build? If so that sounds easy to implement.

Yep, LGTM. For convenience we may also want to cat the file in the console output after asserting it is non-empty, for convenience. (As opposed to failing and requiring the user to browse up to the Job Status page and find the artefact.)

Change 369663 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/jenkins@master] Add a note about mw debug file name

https://gerrit.wikimedia.org/r/369663

Change 369676 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Assert MediaWiki does not generate error logs

https://gerrit.wikimedia.org/r/369676

hashar claimed this task.Aug 2 2017, 3:20 PM
hashar moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.
greg added a comment.Aug 7 2017, 5:33 PM

Tyler will try to take a look at this while Antoine is out.

Change 369676 merged by jenkins-bot:
[integration/config@master] Assert MediaWiki does not generate error logs

https://gerrit.wikimedia.org/r/369676

Change 370691 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[integration/config@master] Remove mediawiki-extension-* assert-no-mediawiki-errors

https://gerrit.wikimedia.org/r/370691

Change 370691 merged by jenkins-bot:
[integration/config@master] Remove mediawiki-extension-* assert-no-mediawiki-errors

https://gerrit.wikimedia.org/r/370691

greg added a comment.Aug 9 2017, 3:29 PM

dang (re ^ )

This assertion is now live for:

mediawiki-core-qunit-selenium-jessie
mediawiki-phpunit-hhvm-composer-jessie
mediawiki-phpunit-hhvm-jessie
mediawiki-phpunit-php55-trusty
mediawiki-phpunit-php70-jessie

The initial patch made the assertion active for:

mediawiki-extensions-hhvm-jessie
mediawiki-extensions-php55-trusty
mediawiki-extensions-php70-jessie
mediawiki-extensions-qunit-jessie

as well; however, I removed the assertion from those tests as all tests were failing :(

thcipriani closed this task as Resolved.Aug 23 2017, 4:56 PM

The assertion has been active for these jobs for 2 weeks:

mediawiki-extensions-hhvm-jessie
mediawiki-extensions-php55-trusty
mediawiki-extensions-php70-jessie
mediawiki-extensions-qunit-jessie

Calling this complete

Krinkle removed a project: Patch-For-Review.

Change 369663 merged by jenkins-bot:
[integration/jenkins@master] Add a note about mw debug file name

https://gerrit.wikimedia.org/r/369663