Excellent report Dan. I really envy your excellent sense of humor!

In T218233#5036593, @Krinkle wrote:

Looks like all the Debian channels (jessie, stretch, buster, sid) are still on 1.8.13. Last update was last year (1.8.13-10) indicating that the package no longer has a Debian maintainer (discussion).

From: Antoine Musso
Date: Wed, 20 Mar 2019, 22:24

From: Paolo Greppi
Date: Wed, 20 Mar 2019 18:52:04 +0100

When scap clean is issued, on Gerrit sshd_log we eventually see:

13:44:00 Started prune-git-branches
Received disconnect from 2620:0:861:3:208:80:154:85 port 29418:2: Too many authentication failures: 7
Authentication failed.

13:44:00 Started prune-git-branches
Received disconnect from 2620:0:861:3:208:80:154:85 port 29418:2: Too many authentication failures: 7
Authentication failed.

208.80.154.85 would be the IPv4 address of the host
It is gerrit.wikimedia.org

We have a Jenkins job T97513 which has been made to recognizes Hosts: in commit message to passe a list of nodes to the PPC. 91b70b4612abc6443a571dcc397622c5ddbe82af

We need a dedicated policy. Granting Gerrit administrative rights is a lot more responsibilities and require a lot of trust. It is the same level as granting rights to the puppet repository.

Thank you for the fix and for the backports to REL1_31 and REL1_32.

Thank you @Krinkle and sorry for the revert earlier :-\

T218388#5036149 debug session by @kostajh is exactly how I used to track those kind of regressions. Namely:

• clone as few repositories as possible
• run tests with --filter until a repro is find then either revert/rollback/git bisect or insert print/return/continue statements.
• log thing and use colordiff (wrapper around diff with ansi coloring) / wdiff (word by word differences, colordiff supports colorizing it). Or: wdiff bad.log good.log | colordiff

From a CI perspective, we need to keep supporting older branches of MediaWiki that we run tests against, so the "migration" of php-ast from my perspective only ends once the last branch using the older version is phased out.

Tentatively that has been solved already?

Sorry, I am definitely not working on this one :-(

There are patch aboves, including one to update the composer merge plugin. I haven't had any review or feedback on those though.

Completely forgot about this one or got distracted with too many things. Anyway I am not actively working on this sorry.

The patch made it in v0.20.0 and it has been / is being deployed via T212774

Pending the puppet patch to be merged. Otherwise it is essentially solved :]

I have cherry picked https://gerrit.wikimedia.org/r/495681/3 on the CI puppetmaster and I now have:

Ok sury.org apt fetch seems fixed for now.

In Puppet we have a key added via T144872

$ gpg --list-options show-keyring ./modules/contint/files/sury-php.gpg
gpg: WARNING: no command supplied.  Trying to guess what you mean ...
pub   rsa4096 2014-09-09 [SC]
      DF3D585DB8F0EB658690A554AC0E47584A7A714D
uid           CZ.NIC Labs Archive Automatic Signing Key <ftpmaster@labs.nic.cz>
sub   rsa4096 2014-09-09 [E]

Sorry I have lost track of this ticket since March 7th but seems it had ample activity.

that is somehow due to ::profile::openstack::main::clientpackages. I have reverted https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/495757/ / ff5d0e075eb7ae9900d89dd5e865c8f1e7e8d971 and that fixed the apt issue

I have looked at the bots doc on:

My local email server configuration was screwed up. The bug is: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=925068

I have send a bug to Debian but I have not received the automatic acknowledgement yet :(

https://github.com/doxygen/doxygen/issues/3545

PHP was not parsed properly when it appeared in a <script language="php"> section.

Yes from http://php.net/manual/en/language.basic-syntax.phptags.php

At least PHP works since that generates https://doc.wikimedia.org/

The check pipeline is gone!

Superseded by https://gerrit.wikimedia.org/r/#/c/integration/quibble/+/497474

A LocalSettings.php file has been detected. To upgrade this installation, please run update.php instead

Note: you can skp Selenium tests with: quibble --skip selenium

This is intentional for now. The PHPUnit marked with @group Database are usually way slower than the Selenium tests and the intent is to fail fast. That being said we can revisit later.

I would rather have a single canonical documentation https://doc.wikimedia.org/quibble/ But surely we should have some introduction on https://www.mediawiki.org/wiki/Continuous_integration/Quibble

Gerrit is not reachable but here is the patch:

At least with git 2.11, the git submodule update seems to exit 1 when I kill the underlying git-remote-https subprocess.

I do not have any idea how to detect the submodule fetching failed. Maybe by running git submodule status and processing its output :/

Indeed, it did try to process the submodules but since Gerrit was not responsive/down the clones failed:

INFO:quibble.cmd:Updating git submodules of extensions and skins
extensions/VisualEditor/.gitmodules
+ git submodule foreach git clean -xdff -q
+ git submodule update --init --recursive
Submodule 'lib/ve' (https://gerrit.wikimedia.org/r/p/VisualEditor/VisualEditor.git) registered for path 'lib/ve'
Cloning into '/workspace/src/extensions/VisualEditor/lib/ve'...
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: clone of 'https://gerrit.wikimedia.org/r/p/VisualEditor/VisualEditor.git' into submodule path '/workspace/src/extensions/VisualEditor/lib/ve' failed
Failed to clone 'lib/ve'. Retry scheduled
Cloning into '/workspace/src/extensions/VisualEditor/lib/ve'...
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: clone of 'https://gerrit.wikimedia.org/r/p/VisualEditor/VisualEditor.git' into submodule path '/workspace/src/extensions/VisualEditor/lib/ve' failed
Failed to clone 'lib/ve' a second time, aborting
...
INFO:backend.MySQL:Initializing MySQL data directory

For the IOS application, the development moved to Github to be able to use a third party that provides MacOS / Iphone etc as a service. That has been deemed easier and cheaper than having to figure out how to maintain Mac OS X system in a production environment or in the WMF Office It environment.

The Docker container is now based on Stretch and has java 8:

tox-pyspark (0.3.0) wikimedia; urgency=medium

I do not know whether it still occurs, I haven't checked. Maybe the SQL database is too slow :(

Sorry for lack of follow up, I have missed the task creation notification.

Looking again at top8 -1months highest average for tools.*webgrid-lighttpd*.cpu.total.user

I have filled this task when I have migrated the CI job for MediaWiki toward Docker. MariaDB came with the default configuration from the Debian packages:

8b45a9a4df8b2e56d4f262a23ea94d677d64dd58 introduced the job in Feb 2018 with:

  # Only run patch coverage if a PHP file was
  # changed on master.
- name: mediawiki-phpunit-coverage-patch
  branch: master
  files:
   - '^.*\.php$'

We have filled this task following Release-Engineering-Team May 2018 offsite. One of the need is for CI to be able to **optionally* install dependent extensions. The current system is centrally managed in integration/config.git and suffers from several issues:

• requires review/merge/deploy from one of the CI maintainer
• is not aware of branches

That makes total sense. Thank you!

Fixed by Upstream with https://secure.phabricator.com/D19967

For the record, that was due to an issue in libc6/libpthread which got fixed in a more recent Debian package than the one we had in the Docker container.

In T216689#5027159, @hashar wrote:

The job has been updated around 2019-03-14T21:21:20Z

The last failing build has been #39845 on March 14th at 21:16:00Z

The libc package update fixed it, the container was simply faulty.

https://gerrit.wikimedia.org/r/#/c/integration/config/+/496887/ is a CI change to replace "jsduck" with "npm doc". I have reverted it since Wikibase is broken.

The change got merged on 03/18 at 09:29 UTC. From Zuul debug logs

The configuration is generated via a python script modules/shinken/files/shinkengen in operations/puppet. It seems to query OpenStack for a list of instances on the project.

This one is resolved. There are a few pending changes but nothing any interesting or urgent for the service.

tests/phpunit/DiscussionParserTest.php makes expectations which are fine when just running Echo tests, though that should be verified for example by sending a dummy change to Echo and see whether the job passes. I cant tell what is wrong really, might be in Echo or in mediawiki/core, or worse some code added in another extension that end up triggering a condition that this test does not expect :-/

The job has been updated around 2019-03-14T21:21:20Z

@santhosh that is unrelated to this task.