hashar (Antoine "hashar" Musso (WMF))
WMF Software developer - Release Engineering

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
Oct 3 2014, 2:31 PM (215 w, 1 d)
Availability
Available
IRC Nick
hashar
LDAP User
Hashar
MediaWiki User
Unknown

https://www.mediawiki.org/wiki/User:Hashar

Based in Nantes, France CET/CEST (UTC+1, UTC+2)

Main IRC channel is #wikimedia-releng

antoine-approve

Recent Activity

Fri, Nov 16

mmodell awarded T190097: Migrate all CI jobs from Nodepool, deprecate its use a Orange Medal token.
Fri, Nov 16, 5:48 PM · Patch-For-Review, Release-Engineering-Team, Epic
hashar updated the task description for T209361: Phase out Nodepool from production.
Fri, Nov 16, 5:35 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
greg awarded T209361: Phase out Nodepool from production a Barnstar token.
Fri, Nov 16, 5:21 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar updated subscribers of T208803: Migrate the Integration cloud project to eqiad1-r.

So for integration-publishing, its used from contint1001 which is a production machine. The server thus can not resolve the DNS entry integration-publishing.integration.eqiad.wmflabs. and we have hardcoded the instance IP address in the Jenkins jobs.

Fri, Nov 16, 3:47 PM · Patch-For-Review, Release-Engineering-Team (Kanban), Epic, Cloud-Services
hashar updated the task description for T205870: Provision >= 50% of statsd/Graphite-only metrics in Prometheus.
Fri, Nov 16, 3:32 PM · Patch-For-Review, monitoring, Operations
hashar added a comment to T205870: Provision >= 50% of statsd/Graphite-only metrics in Prometheus.

The gerrit. metrics are actually reported by the Zuul service on contint1001. It corresponds to events receive from Gerrit such as patchsets, changes, comments. The related Grafana board is https://grafana.wikimedia.org/dashboard/db/releng-gerrit

Fri, Nov 16, 3:31 PM · Patch-For-Review, monitoring, Operations
hashar added a comment to T88997: Improve graphite failover.

Zuul now emits stats to localhost which has statsite running :)

Fri, Nov 16, 3:28 PM · Performance-Team (Radar), Patch-For-Review, Zuul, Operations, Graphite
hashar updated the task description for T88997: Improve graphite failover.
Fri, Nov 16, 3:27 PM · Performance-Team (Radar), Patch-For-Review, Zuul, Operations, Graphite
hashar committed rEQSdc8164cf7f7f: Update ffi and rubyzip gems (authored by hashar).
Update ffi and rubyzip gems
Fri, Nov 16, 2:26 PM
hashar updated the task description for T88997: Improve graphite failover.
Fri, Nov 16, 12:49 PM · Performance-Team (Radar), Patch-For-Review, Zuul, Operations, Graphite

Thu, Nov 15

hashar set the color for Nodepool to Red.
Thu, Nov 15, 9:32 PM
hashar archived Nodepool.
Thu, Nov 15, 9:32 PM
hashar placed T209642: Remove labnodepool1001.eqiad.wmnet up for grabs.

CI no more rely on the service that is hosted on labnodepool1001.eqiad.wmnet.

Thu, Nov 15, 9:31 PM · decommission, cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar updated the task description for T209642: Remove labnodepool1001.eqiad.wmnet.
Thu, Nov 15, 9:29 PM · decommission, cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar updated the task description for T209642: Remove labnodepool1001.eqiad.wmnet.
Thu, Nov 15, 9:28 PM · decommission, cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar updated the task description for T209361: Phase out Nodepool from production.
Thu, Nov 15, 9:18 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar created T209644: Delete contintcloud WMCS project.
Thu, Nov 15, 9:17 PM · cloud-services-team
hashar updated the task description for T209361: Phase out Nodepool from production.
Thu, Nov 15, 9:12 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar updated the task description for T209361: Phase out Nodepool from production.
Thu, Nov 15, 9:03 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar updated the task description for T209361: Phase out Nodepool from production.
Thu, Nov 15, 8:57 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar updated the task description for T209642: Remove labnodepool1001.eqiad.wmnet.
Thu, Nov 15, 8:56 PM · decommission, cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar created T209642: Remove labnodepool1001.eqiad.wmnet.
Thu, Nov 15, 8:42 PM · decommission, cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar updated the task description for T209361: Phase out Nodepool from production.
Thu, Nov 15, 8:39 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar removed a subtask for T209361: Phase out Nodepool from production: T190097: Migrate all CI jobs from Nodepool, deprecate its use.
Thu, Nov 15, 8:38 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar removed a parent task for T190097: Migrate all CI jobs from Nodepool, deprecate its use: T209361: Phase out Nodepool from production.
Thu, Nov 15, 8:38 PM · Patch-For-Review, Release-Engineering-Team, Epic
thcipriani awarded T209361: Phase out Nodepool from production a Yellow Medal token.
Thu, Nov 15, 8:08 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar updated the task description for T209361: Phase out Nodepool from production.
Thu, Nov 15, 8:07 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar closed T206658: 1.33.0-wmf.4 deployment blockers as Resolved.

1.33.0-wmf.4 has been deployed everywhere

Thu, Nov 15, 7:57 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
hashar added a comment to T209361: Phase out Nodepool from production.

I have migrated the last jobs still using Nodepool. Ready to phase out Nodepool :-]

Thu, Nov 15, 7:28 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar closed T190097: Migrate all CI jobs from Nodepool, deprecate its use as Resolved.

All jobs have been migrated. There are still some sub tasks opened but they are no more blockers for phasing out Nodepool.

Thu, Nov 15, 7:28 PM · Patch-For-Review, Release-Engineering-Team, Epic
hashar closed T190097: Migrate all CI jobs from Nodepool, deprecate its use, a subtask of T209361: Phase out Nodepool from production, as Resolved.
Thu, Nov 15, 7:28 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar closed T183512: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker as Resolved.

The jobs to test Wikibase client and repo have been migrated. They were the last ones still relying on Nodepool.

Thu, Nov 15, 7:26 PM · MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)), Patch-For-Review, Epic, Release-Engineering-Team (Kanban), releng-201718-q3, Continuous-Integration-Infrastructure (shipyard)
hashar closed T183512: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker, a subtask of T190097: Migrate all CI jobs from Nodepool, deprecate its use, as Resolved.
Thu, Nov 15, 7:25 PM · Patch-For-Review, Release-Engineering-Team, Epic
hashar closed T188717: Move Wikibase to using the normal mediawiki extension (quibble) jobs as Resolved.

The repo/client jobs are passing , "just" had to get Quibble to run build/jenkins/mw-apply-wb-settings.sh before running phpunit tests.

Thu, Nov 15, 7:25 PM · MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Release-Engineering-Team (Kanban), wikidata-tech-focus, Patch-For-Review, User-Addshore, Continuous-Integration-Config, Wikidata
hashar closed T188717: Move Wikibase to using the normal mediawiki extension (quibble) jobs, a subtask of T183512: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker, as Resolved.
Thu, Nov 15, 7:24 PM · MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)), Patch-For-Review, Epic, Release-Engineering-Team (Kanban), releng-201718-q3, Continuous-Integration-Infrastructure (shipyard)
hashar closed T209582: Unable to serialize r:46: Id 46 out of range. as Invalid.

That is an issue with the serialization of a variable reference (that is the r in the serialized format). Example:

<?php
$var = "hello";
$arr = [&$var, &$var];
var_dump($arr);
var_dump(serialize($arr));
Thu, Nov 15, 1:52 PM · Wikimedia-production-error, MediaWiki-General-or-Unknown
hashar created T209582: Unable to serialize r:46: Id 46 out of range..
Thu, Nov 15, 1:25 PM · Wikimedia-production-error, MediaWiki-General-or-Unknown
hashar added a comment to T206658: 1.33.0-wmf.4 deployment blockers.

I have forgot to update the task. group1 went with 1.33.0-wmf.4 during the Americans train window (8pm UTC yesterday).

Thu, Nov 15, 1:06 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
hashar added a comment to T200739: Upgrade to Gerrit 2.16.

Git protocol 2 support has been added.

Thu, Nov 15, 7:17 AM · Patch-For-Review, Gerrit

Wed, Nov 14

Aklapper awarded T209429: memcached error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE a Like token.
Wed, Nov 14, 10:14 PM · MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), Patch-For-Review, MediaWiki-JobQueue, Wikimedia-production-error
hashar added a comment to T192702: Skip Quibble "composer" and "sqlite" jobs on wmf branches.

The skip-if are not merged (just like branch), the last one in the config wins:

zuul/scheduler.py
for config_job in data.get('jobs', []):
    job = layout.getJob(config_job['name'])
Wed, Nov 14, 8:46 PM · Patch-For-Review, Release-Engineering-Team (Next), Continuous-Integration-Config, Quibble
hashar created P7806 Zuul hold following change.
Wed, Nov 14, 8:29 PM
hashar added a comment to T192702: Skip Quibble "composer" and "sqlite" jobs on wmf branches.

For release-quibble-vendor-mysql-php72-docker, @Legoktm mentioned it has:

- name: ^release-quibble-
  branch: ^REL
Wed, Nov 14, 8:07 PM · Patch-For-Review, Release-Engineering-Team (Next), Continuous-Integration-Config, Quibble
hashar added a comment to T192702: Skip Quibble "composer" and "sqlite" jobs on wmf branches.

I went with a skip-if. If a job is matched by multiple jobs: filters each having a branch: , I think only the last one stick. The skip-if ones get merged.

Wed, Nov 14, 8:01 PM · Patch-For-Review, Release-Engineering-Team (Next), Continuous-Integration-Config, Quibble
hashar added a comment to T192702: Skip Quibble "composer" and "sqlite" jobs on wmf branches.

Sqlite got filtered out already.

Wed, Nov 14, 7:56 PM · Patch-For-Review, Release-Engineering-Team (Next), Continuous-Integration-Config, Quibble
hashar updated the task description for T192702: Skip Quibble "composer" and "sqlite" jobs on wmf branches.
Wed, Nov 14, 7:55 PM · Patch-For-Review, Release-Engineering-Team (Next), Continuous-Integration-Config, Quibble
hashar closed T209429: memcached error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE as Resolved.

Thank you Brad. There is no more errors in logstash.

Wed, Nov 14, 7:21 PM · MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), Patch-For-Review, MediaWiki-JobQueue, Wikimedia-production-error
hashar closed T209429: memcached error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE , a subtask of T206658: 1.33.0-wmf.4 deployment blockers, as Resolved.
Wed, Nov 14, 7:21 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
hashar added a comment to T209463: Add a link from Phabricator task to a Gerrit search for bug:TXXXXX.

I thought about a link in the left side bar such as Search Gerrit. Some tasks have lot of patchsets and it is a bit hard to keep track of which are still pending. I then end up manually searching in Gerrit for bug:T12345 when potentially I could just have clicked a link to bring up the search.

Wed, Nov 14, 7:06 PM · Phabricator
hashar added a comment to T209456: Gerrit is down "502 Proxy Error".

For monitoring it seems we monitor the JVM with JMX. There is a task for Gerrit at: T184086 The CI Jenkins used to have Java melody but that is apparently no more enabled.

Wed, Nov 14, 6:57 PM · Operations, Gerrit
hashar added a comment to T209429: memcached error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE .

I ran a full sync for deployment, hence the delay :(

Wed, Nov 14, 6:37 PM · MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), Patch-For-Review, MediaWiki-JobQueue, Wikimedia-production-error
hashar added a comment to T206658: 1.33.0-wmf.4 deployment blockers.

Train on hold based on T209429 . I have posted to wikitech-l about it

Wed, Nov 14, 2:30 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
Addshore awarded T209463: Add a link from Phabricator task to a Gerrit search for bug:TXXXXX a Goat token.
Wed, Nov 14, 10:50 AM · Phabricator
hashar added a comment to T209456: Gerrit is down "502 Proxy Error".

We had a lot of stack overflow errors on random pages such as gittiles, that is due to some pages being very long to prettify. Then at 5:36:43 UTC we had a lot of:

[2018-11-14 05:36:43,643] [HTTP-8409] WARN  org.eclipse.jetty.util.thread.QueuedThreadPool : HTTP{STARTED,10<=60<=60,i=0,q=500} rejected org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@3e829144
[2018-11-14 05:36:43,643] [HTTP-8409] WARN  org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy : Rejected execution of org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@3e829144
[2018-11-14 05:36:43,666] [HTTP-8401] WARN  org.eclipse.jetty.util.thread.QueuedThreadPool : HTTP{STARTED,10<=60<=60,i=0,q=500} rejected org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@4a74a19e
[2018-11-14 05:36:43,669] [HTTP-8401] WARN  org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy : Rejected execution of org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@4a74a19e
Wed, Nov 14, 10:48 AM · Operations, Gerrit
hashar updated subscribers of T209456: Gerrit is down "502 Proxy Error".

That might be related to Gerrit 2.15.6 upgrade T205784 . I am not familiar with Jetty though but we can at least dig in the logs on the cobalt server.

Wed, Nov 14, 10:33 AM · Operations, Gerrit
hashar added a comment to T88997: Improve graphite failover.

@fgiunchedi can we add the statsd proxy for the servers running Zuul? My previous comment above T88997#4676750 has all the relevant bits. It should be straightforward to setup.

Wed, Nov 14, 10:30 AM · Performance-Team (Radar), Patch-For-Review, Zuul, Operations, Graphite
hashar committed rERSLed577b8e8be1: Update rubyzip gem (authored by hashar).
Update rubyzip gem
Wed, Nov 14, 10:23 AM
hashar created T209463: Add a link from Phabricator task to a Gerrit search for bug:TXXXXX.
Wed, Nov 14, 10:16 AM · Phabricator
hashar added a comment to T208101: Migrate deployment-prep to eqiad1.

I do not have any available bandwidth for the next 4 weeks at least, sorry. I am running the MediaWiki train from 11/12 to 11/16 and from 11/26 to 11/30. We then have an offsite first week of December which I have to prepare next week.

Wed, Nov 14, 10:07 AM · Release-Engineering-Team, Beta-Cluster-Infrastructure, Epic, Cloud-Services
hashar added a comment to T188585: PhpTags tests fail on PHP 7.0.

For the record, the phpunit failures under php 7.1 were:

There were 12 errors:
Wed, Nov 14, 9:54 AM · Patch-For-Review, Core Platform Team (PHP7 (TEC4)), Core Platform Team Kanban (Doing), PHP 7.0 support, MediaWiki-extensions-PhpTags

Tue, Nov 13

hashar added a comment to T209429: memcached error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE .

Does not seem to harmful, I will attempt to find the root cause tomorrow.

Tue, Nov 13, 11:05 PM · MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), Patch-For-Review, MediaWiki-JobQueue, Wikimedia-production-error
hashar added a comment to T206658: 1.33.0-wmf.4 deployment blockers.

T209429 is about memcached errors appearing with 1.33.0-wmf.4 on group0

Tue, Nov 13, 11:02 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
hashar added a parent task for T209429: memcached error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE : T206658: 1.33.0-wmf.4 deployment blockers.
Tue, Nov 13, 11:01 PM · MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), Patch-For-Review, MediaWiki-JobQueue, Wikimedia-production-error
hashar added a subtask for T206658: 1.33.0-wmf.4 deployment blockers: T209429: memcached error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE .
Tue, Nov 13, 11:01 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
hashar updated the task description for T209429: memcached error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE .
Tue, Nov 13, 11:00 PM · MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), Patch-For-Review, MediaWiki-JobQueue, Wikimedia-production-error
hashar updated the task description for T209429: memcached error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE .
Tue, Nov 13, 10:58 PM · MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), Patch-For-Review, MediaWiki-JobQueue, Wikimedia-production-error
hashar created T209429: memcached error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE .
Tue, Nov 13, 10:56 PM · MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), Patch-For-Review, MediaWiki-JobQueue, Wikimedia-production-error
hashar closed T205784: Upgrade gerrit to 2.15.6 as Resolved.

We went with Gerrit 2.15.6 since all the preparation work has been done already and it was deemed stable for production deployment.

Tue, Nov 13, 5:43 PM · Patch-For-Review, Release-Engineering-Team, Gerrit
hashar renamed T205784: Upgrade gerrit to 2.15.6 from Upgrade gerrit to 2.15.7 to Upgrade gerrit to 2.15.6.
Tue, Nov 13, 5:41 PM · Patch-For-Review, Release-Engineering-Team, Gerrit
hashar added a comment to T206658: 1.33.0-wmf.4 deployment blockers.

Deployment notes:

Tue, Nov 13, 2:49 PM · Release-Engineering-Team (Kanban), Release, Train Deployments
chasemp awarded T209361: Phase out Nodepool from production a Love token.
Tue, Nov 13, 2:30 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar updated the task description for T209361: Phase out Nodepool from production.
Tue, Nov 13, 2:29 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar added a subtask for T209361: Phase out Nodepool from production: T190097: Migrate all CI jobs from Nodepool, deprecate its use.
Tue, Nov 13, 2:23 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar added a parent task for T190097: Migrate all CI jobs from Nodepool, deprecate its use: T209361: Phase out Nodepool from production.
Tue, Nov 13, 2:23 PM · Patch-For-Review, Release-Engineering-Team, Epic
hashar created T209361: Phase out Nodepool from production.
Tue, Nov 13, 2:22 PM · cloud-services-team (Kanban), Patch-For-Review, Operations, Nodepool, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure (shipyard)
hashar added a comment to T188717: Move Wikibase to using the normal mediawiki extension (quibble) jobs.

Status: I need a few patches for Wikibase to be merged in in order to have the custom configuration to be prepended in LocalSettings.php, in effect before doing require_once( __DIR__ . 'mediawiki/extensions/Wikibase.php' ). That is https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/470798/ which has two parent changes.

Tue, Nov 13, 12:38 PM · MW-1.33-notes (1.33.0-wmf.6; 2018-11-27), Wikidata-Campsite (Wikidata-Campsite-Iteration-∞), Release-Engineering-Team (Kanban), wikidata-tech-focus, Patch-For-Review, User-Addshore, Continuous-Integration-Config, Wikidata
hashar created P7797 T188717 parse-testsuites.py.
Tue, Nov 13, 12:33 PM
hashar created P7796 T188717 parse-testcases.py.
Tue, Nov 13, 12:32 PM
hashar closed T209280: https://gerrit.wikimedia.org/r/#/projects/blubber should be mirrored to github as Resolved.

I have manually created the GitHub repository at https://github.com/wikimedia/blubber and the replication worked:

$ ssh -p 29418 gerrit.wikimedia.org replication start blubber --wait
Replicate blubber ref ..all.. to gerrit2001.wikimedia.org, Succeeded! (OK)
Replicate blubber ref ..all.. to github.com, Succeeded! (OK)
Replication of blubber ref ..all.. completed to 2 nodes, 
----------------------------------------------
Replication completed successfully!
Tue, Nov 13, 10:14 AM · Gerrit, GitHub-Mirrors, Release Pipeline (Blubber)
hashar added a comment to T209280: https://gerrit.wikimedia.org/r/#/projects/blubber should be mirrored to github.
$ ssh -p 29418 gerrit.wikimedia.org replication start blubber --wait
Error: Cannot replicate to git@github.com:wikimedia/blubber; repository not found
Tue, Nov 13, 10:12 AM · Gerrit, GitHub-Mirrors, Release Pipeline (Blubber)
hashar updated the task description for T209280: https://gerrit.wikimedia.org/r/#/projects/blubber should be mirrored to github.
Tue, Nov 13, 10:09 AM · Gerrit, GitHub-Mirrors, Release Pipeline (Blubber)
hashar added a project to T209280: https://gerrit.wikimedia.org/r/#/projects/blubber should be mirrored to github: Gerrit.
Tue, Nov 13, 10:08 AM · Gerrit, GitHub-Mirrors, Release Pipeline (Blubber)

Mon, Nov 12

hashar closed T209264: Upgrade Jenkins instances to 2.138.3 as Resolved.

All four hosts updated https://debmonitor.wikimedia.org/packages/jenkins

Mon, Nov 12, 2:04 PM · Jenkins, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure
hashar added a comment to T209279: debmonitor search yields nothing.

Ha @Volans nailing it: the search is not implemented :) thanks!

Mon, Nov 12, 2:02 PM · Operations
hashar added a comment to T209279: debmonitor search yields nothing.

I went to https://debmonitor.wikimedia.org/ , there is a search box in the menu bar on the top right, I have simply filled search terms there (jenkins or contint1001) but no luck, I am simply redirected to https://debmonitor.wikimedia.org/? (with a trailling question mark).
The direct links work:

Mon, Nov 12, 2:02 PM · Operations
hashar created T209279: debmonitor search yields nothing.
Mon, Nov 12, 1:53 PM · Operations
hashar added a comment to T209264: Upgrade Jenkins instances to 2.138.3.

I've uploaded 2.138.3 for jessie and stretch. The component names are correct, we've only introduced the concept of dedicated components for thirdparty with stretch. So for jessie it went into "thirdparty" and for stretch into "thirdparty/ci".

Mon, Nov 12, 1:26 PM · Jenkins, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure
hashar closed T208138: Give several access to l10n-bot and l10n-bot-watcher on labs/tools/* as Resolved.
Mon, Nov 12, 1:24 PM · Patch-For-Review, Map-of-monuments, Wikinity, Weapon of Mass Description, Commons-Mass-Description, Repository-Ownership-Requests, Gerrit
hashar closed T208138: Give several access to l10n-bot and l10n-bot-watcher on labs/tools/*, a subtask of T208190: Translatewiki.net translation update commit failures, as Resolved.
Mon, Nov 12, 1:24 PM · translatewiki.net
hashar added a comment to T209123: contint1001 build cleanup.

I think I got the low hanging fruits solved:

Mon, Nov 12, 12:53 PM · Patch-For-Review, Release-Engineering-Team, Continuous-Integration-Infrastructure
hashar added a comment to T209123: contint1001 build cleanup.

The apps-android-wikipedia-test job generates several apk for the android app, each being roughly 62MBytes:

62	./prod
120	./alpha
63	./dev
62	./fdroid
56	./androidTest
62	./beta
62	./custom
62	./amazon

I am not sure whether we need to capture them.

Mon, Nov 12, 11:54 AM · Patch-For-Review, Release-Engineering-Team, Continuous-Integration-Infrastructure
hashar added a comment to T209123: contint1001 build cleanup.

The worst offenders on contint1001 under /srv/jenkins/builds in MBytes:

Mon, Nov 12, 11:48 AM · Patch-For-Review, Release-Engineering-Team, Continuous-Integration-Infrastructure
hashar updated subscribers of T192217: Remove the "check" pipeline and Zuul's user-filter.

The check pipeline is used for php -l, composer validate and the commit message validator. I think we can drop support for them.

Mon, Nov 12, 11:04 AM · Release-Engineering-Team, Continuous-Integration-Infrastructure
hashar closed T207685: Run lint in check pipeline for Urbanecm's tools as Declined.

The Zuul check pipeline is for users which have not been whitelisted to have code running for them, it only runs jobs that are fairly simple and do not execute code from the patch (eg php -l). Eventually the pipeline has to be removed entirely T192217 , most of the repositories now use CI entry point (composer test, npm test ..) which make it easy to reproduce on a local machine.

Mon, Nov 12, 10:28 AM · WMCZ-Events, WMCZ-General, WMCZ-Tracker, Weapon of Mass Description, Commons-Mass-Description, Map-of-monuments, Wikinity, Continuous-Integration-Config, User-Urbanecm
hashar awarded T209176: Add Mukunda to releasers-mediawiki a Like token.
Mon, Nov 12, 10:07 AM · Patch-For-Review, SRE-Access-Requests, Operations
hashar added a comment to T205784: Upgrade gerrit to 2.15.6.

@Paladox mentioned that 2.15.7 is to be cut this Monday, the first day of Gerrit hackathon (ref: https://gerrit-review.googlesource.com/c/homepage/+/203612/2#message-5970b893ff591c8906bebff7d50045c671106065 )

Mon, Nov 12, 9:49 AM · Patch-For-Review, Release-Engineering-Team, Gerrit
hashar updated subscribers of T209264: Upgrade Jenkins instances to 2.138.3.

Guten Tag Moritz, would you be kind enough to refresh the Jenkins Debian package in reprepro? https://wikitech.wikimedia.org/wiki/Jenkins#Updating

Mon, Nov 12, 9:25 AM · Jenkins, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure
hashar created T209264: Upgrade Jenkins instances to 2.138.3.
Mon, Nov 12, 9:20 AM · Jenkins, Release-Engineering-Team (Kanban), Continuous-Integration-Infrastructure
hashar added a comment to T205784: Upgrade gerrit to 2.15.6.

2.15.7 has not be cut yet. We will proceed with 2.15.6 (have to schedule it)

Mon, Nov 12, 9:05 AM · Patch-For-Review, Release-Engineering-Team, Gerrit
hashar renamed T205784: Upgrade gerrit to 2.15.6 from Upgrade gerrit to 2.15.7 to Upgrade gerrit to 2.15.6.
Mon, Nov 12, 9:01 AM · Patch-For-Review, Release-Engineering-Team, Gerrit