+@zeljkofilipin because he's the last jenkins expert that is still online.

This is strange:

00:00:37.193 (23) Failed writing body

It might or not be an issue with CI infra. @thcipriani @zeljkofilipin and I are taking this as an opportunity to do some knowledge transfer. The curl error reported by Zeljko is definitely not obvious at all :-/

A tweak that did not help.

https://integration.wikimedia.org/ci/job/mwext-Wikibase-qunit/jobConfigHistory/showDiffFiles?timestamp1=2015-05-11_20-56-08&timestamp2=2015-05-20_17-17-24

The mwext-Wikibase-qunit job started failing between May 18th 12:39:11 UTC and May 18th 13:36:51 UTC.

Looking at the first failing build, there is an mw-error.log file attached to it which is $wgDebugLogGroups['error'] with:

2015-05-19 13:38:25 integration-slave-trusty-1014 jenkins_u1_mw: [a97629d3] [no req]   ErrorException from line 419 of includes/jobqueue/JobQueueGroup.php: PHP Notice: JobQueueGroup::__destruct: 1 buffered job(s) never inserted.
#0 [internal function]: MWExceptionHandler::handleError(1024, 'JobQueueGroup::...', '/mnt/jenkins-wo...', 419, Array)
#1 includes/jobqueue/JobQueueGroup.php(419): trigger_error('JobQueueGroup::...')
#2 [internal function]: JobQueueGroup->__destruct()
#3 {main}
2015-05-19 13:38:25 integration-slave-trusty-1014 jenkins_u1_mw: [30a22e70] [no req]   ErrorException from line 419 of includes/jobqueue/JobQueueGroup.php: PHP Notice: JobQueueGroup::__destruct: 1 buffered job(s) never inserted.
#0 [internal function]: MWExceptionHandler::handleError(1024, 'JobQueueGroup::...', '/mnt/jenkins-wo...', 419, Array)
#1 includes/jobqueue/JobQueueGroup.php(419): trigger_error('JobQueueGroup::...')
#2 [internal function]: JobQueueGroup->__destruct()
#3 {main}
2015-05-19 13:38:26 integration-slave-trusty-1014 jenkins_u1_mw: [2cd5481b] [no req]   ErrorException from line 419 of includes/jobqueue/JobQueueGroup.php: PHP Notice: JobQueueGroup::__destruct: 1 buffered job(s) never inserted.
#0 [internal function]: MWExceptionHandler::handleError(1024, 'JobQueueGroup::...', '/mnt/jenkins-wo...', 419, Array)
#1 includes/jobqueue/JobQueueGroup.php(419): trigger_error('JobQueueGroup::...')
#2 [internal function]: JobQueueGroup->__destruct()
#3 {main}
2015-05-19 13:38:28 integration-slave-trusty-1014 jenkins_u1_mw: [b61c63d2] [no req]   ErrorException from line 419 of includes/jobqueue/JobQueueGroup.php: PHP Notice: JobQueueGroup::__destruct: 1 buffered job(s) never inserted.
#0 [internal function]: MWExceptionHandler::handleError(1024, 'JobQueueGroup::...', '/mnt/jenkins-wo...', 419, Array)
#1 includes/jobqueue/JobQueueGroup.php(419): trigger_error('JobQueueGroup::...')
#2 [internal function]: JobQueueGroup->__destruct()
#3 {main}

So it seems some HTTP queries fails, and the curl error would potentially be caused by it.

A few JobQueue related changes landed in MediaWiki core that day which might be related:

• https://gerrit.wikimedia.org/r/#/c/211320/ Added JobQueueGroup::lazyPush method which also does clean ups
• https://gerrit.wikimedia.org/r/#/c/211748/ Use lazyPush() instead of push() in a few places for jobs

The first actually adds the __destruct method to JobQueueGroup so that would explain it.

So it might just be a breaking change or a flaw in https://gerrit.wikimedia.org/r/#/c/211320/ which we might well want to revert to unreal the Wikidata job.

1 buffered job(s) never inserted.

I have seen this locally, screwing up API responses. Seems broken. I wonder why it doesn't happen with core tests, though.

@hashar but strange that even with pulling master core, I don't get the failures locally. could be my settings or something though and the Exceptions are odd.

also note that the RepoApi qunit tests succeed when run alone in the WikibaseJavaScriptApi component (https://gerrit.wikimedia.org/r/#/c/212284/) I think it's been some time since we submitted patches there, until today.

Related to T99849?

Regarding the curl error: curl --fail only ommits any output in the HTTP status failure case. In sh false | true will be true. Bash has the option pipefail to change this to more expected behaviour. However in this case I think just omitting the tail is fine.

The curl error was also happening when all tests pass, see e.g.: https://integration.wikimedia.org/ci/job/mwext-Wikibase-qunit/9770/consoleFull

Change 212473 had a related patch set uploaded (by Aaron Schulz):
Flush lazy jobs in load.php and api.php too

https://gerrit.wikimedia.org/r/212473

In T99775#1300343, @daniel wrote:

1 buffered job(s) never inserted.

I have seen this locally, screwing up API responses. Seems broken. I wonder why it doesn't happen with core tests, though.

That is unclear to me. I guess none of the PHPUnit tests are exercising that code path and the mediawiki core QUnit tests do not exercise that code path either. But the api.php queries made by Wikibase ends up exercising the code and dieing.

One sure thing the feature introduced by https://gerrit.wikimedia.org/r/#/c/211320/ does not have any test :-D

In T99775#1300379, @Legoktm wrote:

Related to T99849?

Yeah seems to be the exact same issue :-}

In T99775#1300392, @JanZerebecki wrote:

Regarding the curl error: curl --fail only ommits any output in the HTTP status failure case. In sh false | true will be true. Bash has the option pipefail to change this to more expected behaviour. However in this case I think just omitting the tail is fine.

The curl error was also happening when all tests pass, see e.g.: https://integration.wikimedia.org/ci/job/mwext-Wikibase-qunit/9770/consoleFull

Filled as T99854: JJB qunit macro ignores curl error exit code

Change 212473 merged by jenkins-bot:
Flush lazy jobs in load.php and api.php too

https://gerrit.wikimedia.org/r/212473

Change 212477 had a related patch set uploaded (by Aaron Schulz):
Flush lazy jobs in load.php and api.php too

https://gerrit.wikimedia.org/r/212477

Change 212477 merged by Ori.livneh:
Flush lazy jobs in load.php and api.php too

https://gerrit.wikimedia.org/r/212477

Change 212486 had a related patch set uploaded (by Aaron Schulz):
Flush lazy jobs in load.php and api.php too

https://gerrit.wikimedia.org/r/212486

Change 212486 merged by jenkins-bot:
Flush lazy jobs in load.php and api.php too

https://gerrit.wikimedia.org/r/212486

That did it, thanks a lot, guys!