Page MenuHomePhabricator

Qunit test always fail in Wikidata Jenkins jobs.
Closed, ResolvedPublic

Description

Jenkins currently fails for all Wikidata changes. The error is in the QUnit job, e.g. https://integration.wikimedia.org/ci/job/mwext-Wikibase-qunit/9786/console:

09:49:11 Chromium 41.0.2272 (Ubuntu) wikibase.api.RepoApi Create an empty entity FAILED
09:49:11 	API request failed returning code: "http". See console for details.

Event Timeline

daniel created this task.May 20 2015, 10:04 AM
daniel assigned this task to Krinkle.
daniel raised the priority of this task from to Unbreak Now!.
daniel updated the task description. (Show Details)
daniel added a subscriber: daniel.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 20 2015, 10:04 AM
Manybubbles added a subscriber: zeljkofilipin.
Manybubbles added a subscriber: Manybubbles.

+@zeljkofilipin because he's the last jenkins expert that is still online.

This is strange:

00:00:37.193 (23) Failed writing body
hashar added a subscriber: thcipriani.

It might or not be an issue with CI infra. @thcipriani @zeljkofilipin and I are taking this as an opportunity to do some knowledge transfer. The curl error reported by Zeljko is definitely not obvious at all :-/

The mwext-Wikibase-qunit job started failing between May 18th 12:39:11 UTC and May 18th 13:36:51 UTC.

Looking at the first failing build, there is an mw-error.log file attached to it which is $wgDebugLogGroups['error'] with:

2015-05-19 13:38:25 integration-slave-trusty-1014 jenkins_u1_mw: [a97629d3] [no req]   ErrorException from line 419 of includes/jobqueue/JobQueueGroup.php: PHP Notice: JobQueueGroup::__destruct: 1 buffered job(s) never inserted.
#0 [internal function]: MWExceptionHandler::handleError(1024, 'JobQueueGroup::...', '/mnt/jenkins-wo...', 419, Array)
#1 includes/jobqueue/JobQueueGroup.php(419): trigger_error('JobQueueGroup::...')
#2 [internal function]: JobQueueGroup->__destruct()
#3 {main}
2015-05-19 13:38:25 integration-slave-trusty-1014 jenkins_u1_mw: [30a22e70] [no req]   ErrorException from line 419 of includes/jobqueue/JobQueueGroup.php: PHP Notice: JobQueueGroup::__destruct: 1 buffered job(s) never inserted.
#0 [internal function]: MWExceptionHandler::handleError(1024, 'JobQueueGroup::...', '/mnt/jenkins-wo...', 419, Array)
#1 includes/jobqueue/JobQueueGroup.php(419): trigger_error('JobQueueGroup::...')
#2 [internal function]: JobQueueGroup->__destruct()
#3 {main}
2015-05-19 13:38:26 integration-slave-trusty-1014 jenkins_u1_mw: [2cd5481b] [no req]   ErrorException from line 419 of includes/jobqueue/JobQueueGroup.php: PHP Notice: JobQueueGroup::__destruct: 1 buffered job(s) never inserted.
#0 [internal function]: MWExceptionHandler::handleError(1024, 'JobQueueGroup::...', '/mnt/jenkins-wo...', 419, Array)
#1 includes/jobqueue/JobQueueGroup.php(419): trigger_error('JobQueueGroup::...')
#2 [internal function]: JobQueueGroup->__destruct()
#3 {main}
2015-05-19 13:38:28 integration-slave-trusty-1014 jenkins_u1_mw: [b61c63d2] [no req]   ErrorException from line 419 of includes/jobqueue/JobQueueGroup.php: PHP Notice: JobQueueGroup::__destruct: 1 buffered job(s) never inserted.
#0 [internal function]: MWExceptionHandler::handleError(1024, 'JobQueueGroup::...', '/mnt/jenkins-wo...', 419, Array)
#1 includes/jobqueue/JobQueueGroup.php(419): trigger_error('JobQueueGroup::...')
#2 [internal function]: JobQueueGroup->__destruct()
#3 {main}

So it seems some HTTP queries fails, and the curl error would potentially be caused by it.

A few JobQueue related changes landed in MediaWiki core that day which might be related:

The first actually adds the __destruct method to JobQueueGroup so that would explain it.

So it might just be a breaking change or a flaw in https://gerrit.wikimedia.org/r/#/c/211320/ which we might well want to revert to unreal the Wikidata job.

hashar removed Krinkle as the assignee of this task.May 20 2015, 10:41 PM
hashar added a project: MediaWiki-JobQueue.
hashar added a subscriber: Krinkle.

Removed @Krinkle from assignee, doesn't make sense to force assign someone without priori acknowledgement.

Seems to me the issue is in MediaWiki core job queue system so + MediaWiki-JobQueue.

1 buffered job(s) never inserted.

I have seen this locally, screwing up API responses. Seems broken. I wonder why it doesn't happen with core tests, though.

aude added a subscriber: aude.May 20 2015, 10:47 PM

@hashar but strange that even with pulling master core, I don't get the failures locally. could be my settings or something though and the Exceptions are odd.

also note that the RepoApi qunit tests succeed when run alone in the WikibaseJavaScriptApi component (https://gerrit.wikimedia.org/r/#/c/212284/) I think it's been some time since we submitted patches there, until today.

JanZerebecki added a comment.EditedMay 20 2015, 10:58 PM

Regarding the curl error: curl --fail only ommits any output in the HTTP status failure case. In sh false | true will be true. Bash has the option pipefail to change this to more expected behaviour. However in this case I think just omitting the tail is fine.

The curl error was also happening when all tests pass, see e.g.: https://integration.wikimedia.org/ci/job/mwext-Wikibase-qunit/9770/consoleFull

Change 212473 had a related patch set uploaded (by Aaron Schulz):
Flush lazy jobs in load.php and api.php too

https://gerrit.wikimedia.org/r/212473

1 buffered job(s) never inserted.

I have seen this locally, screwing up API responses. Seems broken. I wonder why it doesn't happen with core tests, though.

That is unclear to me. I guess none of the PHPUnit tests are exercising that code path and the mediawiki core QUnit tests do not exercise that code path either. But the api.php queries made by Wikibase ends up exercising the code and dieing.

One sure thing the feature introduced by https://gerrit.wikimedia.org/r/#/c/211320/ does not have any test :-D

Related to T99849?

Yeah seems to be the exact same issue :-}

Regarding the curl error: curl --fail only ommits any output in the HTTP status failure case. In sh false | true will be true. Bash has the option pipefail to change this to more expected behaviour. However in this case I think just omitting the tail is fine.
The curl error was also happening when all tests pass, see e.g.: https://integration.wikimedia.org/ci/job/mwext-Wikibase-qunit/9770/consoleFull

Filled as T99854: JJB qunit macro ignores curl error exit code

Change 212473 merged by jenkins-bot:
Flush lazy jobs in load.php and api.php too

https://gerrit.wikimedia.org/r/212473

Change 212477 had a related patch set uploaded (by Aaron Schulz):
Flush lazy jobs in load.php and api.php too

https://gerrit.wikimedia.org/r/212477

Change 212477 merged by Ori.livneh:
Flush lazy jobs in load.php and api.php too

https://gerrit.wikimedia.org/r/212477

Change 212486 had a related patch set uploaded (by Aaron Schulz):
Flush lazy jobs in load.php and api.php too

https://gerrit.wikimedia.org/r/212486

Change 212486 merged by jenkins-bot:
Flush lazy jobs in load.php and api.php too

https://gerrit.wikimedia.org/r/212486

That did it, thanks a lot, guys!