So for integration-publishing, its used from contint1001 which is a production machine. The server thus can not resolve the DNS entry integration-publishing.integration.eqiad.wmflabs. and we have hardcoded the instance IP address in the Jenkins jobs.

The gerrit. metrics are actually reported by the Zuul service on contint1001. It corresponds to events receive from Gerrit such as patchsets, changes, comments. The related Grafana board is https://grafana.wikimedia.org/dashboard/db/releng-gerrit

Zuul now emits stats to localhost which has statsite running :)

CI no more rely on the service that is hosted on labnodepool1001.eqiad.wmnet.

1.33.0-wmf.4 has been deployed everywhere

I have migrated the last jobs still using Nodepool. Ready to phase out Nodepool :-]

All jobs have been migrated. There are still some sub tasks opened but they are no more blockers for phasing out Nodepool.

The jobs to test Wikibase client and repo have been migrated. They were the last ones still relying on Nodepool.

The repo/client jobs are passing , "just" had to get Quibble to run build/jenkins/mw-apply-wb-settings.sh before running phpunit tests.

That is an issue with the serialization of a variable reference (that is the r in the serialized format). Example:

<?php
$var = "hello";
$arr = [&$var, &$var];
var_dump($arr);
var_dump(serialize($arr));

I have forgot to update the task. group1 went with 1.33.0-wmf.4 during the Americans train window (8pm UTC yesterday).

Git protocol 2 support has been added.

The skip-if are not merged (just like branch), the last one in the config wins:

for config_job in data.get('jobs', []):
    job = layout.getJob(config_job['name'])

For release-quibble-vendor-mysql-php72-docker, @Legoktm mentioned it has:

- name: ^release-quibble-
  branch: ^REL

I went with a skip-if. If a job is matched by multiple jobs: filters each having a branch: , I think only the last one stick. The skip-if ones get merged.

Sqlite got filtered out already.

Thank you Brad. There is no more errors in logstash.

I thought about a link in the left side bar such as Search Gerrit. Some tasks have lot of patchsets and it is a bit hard to keep track of which are still pending. I then end up manually searching in Gerrit for bug:T12345 when potentially I could just have clicked a link to bring up the search.

For monitoring it seems we monitor the JVM with JMX. There is a task for Gerrit at: T184086 The CI Jenkins used to have Java melody but that is apparently no more enabled.

I ran a full sync for deployment, hence the delay :(

Train on hold based on T209429 . I have posted to wikitech-l about it

We had a lot of stack overflow errors on random pages such as gittiles, that is due to some pages being very long to prettify. Then at 5:36:43 UTC we had a lot of:

[2018-11-14 05:36:43,643] [HTTP-8409] WARN  org.eclipse.jetty.util.thread.QueuedThreadPool : HTTP{STARTED,10<=60<=60,i=0,q=500} rejected org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@3e829144
[2018-11-14 05:36:43,643] [HTTP-8409] WARN  org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy : Rejected execution of org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@3e829144
[2018-11-14 05:36:43,666] [HTTP-8401] WARN  org.eclipse.jetty.util.thread.QueuedThreadPool : HTTP{STARTED,10<=60<=60,i=0,q=500} rejected org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@4a74a19e
[2018-11-14 05:36:43,669] [HTTP-8401] WARN  org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy : Rejected execution of org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@4a74a19e

That might be related to Gerrit 2.15.6 upgrade T205784 . I am not familiar with Jetty though but we can at least dig in the logs on the cobalt server.

@fgiunchedi can we add the statsd proxy for the servers running Zuul? My previous comment above T88997#4676750 has all the relevant bits. It should be straightforward to setup.

I do not have any available bandwidth for the next 4 weeks at least, sorry. I am running the MediaWiki train from 11/12 to 11/16 and from 11/26 to 11/30. We then have an offsite first week of December which I have to prepare next week.

For the record, the phpunit failures under php 7.1 were:

There were 12 errors:

Does not seem to harmful, I will attempt to find the root cause tomorrow.

T209429 is about memcached errors appearing with 1.33.0-wmf.4 on group0

We went with Gerrit 2.15.6 since all the preparation work has been done already and it was deemed stable for production deployment.

Deployment notes:

Status: I need a few patches for Wikibase to be merged in in order to have the custom configuration to be prepended in LocalSettings.php, in effect before doing require_once( __DIR__ . 'mediawiki/extensions/Wikibase.php' ). That is https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/470798/ which has two parent changes.

I have manually created the GitHub repository at https://github.com/wikimedia/blubber and the replication worked:

$ ssh -p 29418 gerrit.wikimedia.org replication start blubber --wait
Replicate blubber ref ..all.. to gerrit2001.wikimedia.org, Succeeded! (OK)
Replicate blubber ref ..all.. to github.com, Succeeded! (OK)
Replication of blubber ref ..all.. completed to 2 nodes, 
----------------------------------------------
Replication completed successfully!

All four hosts updated https://debmonitor.wikimedia.org/packages/jenkins

Ha @Volans nailing it: the search is not implemented :) thanks!

I went to https://debmonitor.wikimedia.org/ , there is a search box in the menu bar on the top right, I have simply filled search terms there (jenkins or contint1001) but no luck, I am simply redirected to https://debmonitor.wikimedia.org/? (with a trailling question mark).
The direct links work:

• https://debmonitor.wikimedia.org/packages/jenkins
• https://debmonitor.wikimedia.org/hosts/contint1001.wikimedia.org

In T209264#4739185, @MoritzMuehlenhoff wrote:

I've uploaded 2.138.3 for jessie and stretch. The component names are correct, we've only introduced the concept of dedicated components for thirdparty with stretch. So for jessie it went into "thirdparty" and for stretch into "thirdparty/ci".

I think I got the low hanging fruits solved:

The apps-android-wikipedia-test job generates several apk for the android app, each being roughly 62MBytes:

62	./prod
120	./alpha
63	./dev
62	./fdroid
56	./androidTest
62	./beta
62	./custom
62	./amazon

I am not sure whether we need to capture them.

The worst offenders on contint1001 under /srv/jenkins/builds in MBytes:

The check pipeline is used for php -l, composer validate and the commit message validator. I think we can drop support for them.

The Zuul check pipeline is for users which have not been whitelisted to have code running for them, it only runs jobs that are fairly simple and do not execute code from the patch (eg php -l). Eventually the pipeline has to be removed entirely T192217 , most of the repositories now use CI entry point (composer test, npm test ..) which make it easy to reproduce on a local machine.

@Paladox mentioned that 2.15.7 is to be cut this Monday, the first day of Gerrit hackathon (ref: https://gerrit-review.googlesource.com/c/homepage/+/203612/2#message-5970b893ff591c8906bebff7d50045c671106065 )

Guten Tag Moritz, would you be kind enough to refresh the Jenkins Debian package in reprepro? https://wikitech.wikimedia.org/wiki/Jenkins#Updating

2.15.7 has not be cut yet. We will proceed with 2.15.6 (have to schedule it)