Changed to Gerrit project owners for the MwEmbedSupport extension :)

A very basic attempt at reproducing the issue on my local setup:

$ php maintenance/shell.php

>>> require_once 'extensions/Translate/Translate.php';
=> 1

>>> $registry = new ExtensionRegistry();
=> ExtensionRegistry {#1082}

>>> $queue = [ 'extensions/FundraisingTranslateWorkflow/extension.json' => 1 ];
=> [
     "extensions/FundraisingTranslateWorkflow/extension.json" => 1,
   ]

>>> $registry->readFromQueue( $queue );
ExtensionDependencyError with message 'FundraisingTranslateWorkflow requires Translate to be installed.

>>> $registry->isLoaded( 'Translate' );
=> false
>>>

$wmgFundraisingTranslateWorkflow is only enabled on meta. What I guess is happening is that maintenance/mergeMessageFileList.php loads extensions from wmf-config/extension-list.json in the order listed to the file. The script does roughly:

# Add setup files contained in file passed to --list-file
if ( $this->hasOption( 'list-file' ) ) {
    $extensionPaths = $this->readFile( $this->getOption( 'list-file' ) );
    $mmfl['setupFiles'] = array_merge( $mmfl['setupFiles'], $extensionPaths );
}

That also points to https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/FundraisingTranslateWorkflow/+/486345/

Tentatively a blocker is mergeMessageList.php breaking on the beta cluster:

Note beta cluster can not update T218860

Wondering if that would cause the train to break when we next regenerate the l10n messages? T206676: 1.33.0-wmf.22 deployment blockers

So it is broken and I have no idea what is going on sorry :(

The first broken build was on March 20th 2019 at 20:59:13 which is before that Gerrit change :/

Might be related to https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/497956/ / a0553288d5902ae49e84489b89edc5740d5606e4 for T213943.

I have published a quick post about the debugging session I did last week. That might be helpful as a reference in the future:
J152 : Blog Post: Help my CI job fails with exit status -11.

Excellent report Dan. I really envy your excellent sense of humor!

In T218233#5036593, @Krinkle wrote:

Looks like all the Debian channels (jessie, stretch, buster, sid) are still on 1.8.13. Last update was last year (1.8.13-10) indicating that the package no longer has a Debian maintainer (discussion).

From: Antoine Musso
Date: Wed, 20 Mar 2019, 22:24

From: Paolo Greppi
Date: Wed, 20 Mar 2019 18:52:04 +0100

When scap clean is issued, on Gerrit sshd_log we eventually see:

13:44:00 Started prune-git-branches
Received disconnect from 2620:0:861:3:208:80:154:85 port 29418:2: Too many authentication failures: 7
Authentication failed.

13:44:00 Started prune-git-branches
Received disconnect from 2620:0:861:3:208:80:154:85 port 29418:2: Too many authentication failures: 7
Authentication failed.

208.80.154.85 would be the IPv4 address of the host
It is gerrit.wikimedia.org

We have a Jenkins job T97513 which has been made to recognizes Hosts: in commit message to passe a list of nodes to the PPC. 91b70b4612abc6443a571dcc397622c5ddbe82af

We need a dedicated policy. Granting Gerrit administrative rights is a lot more responsibilities and require a lot of trust. It is the same level as granting rights to the puppet repository.

Thank you for the fix and for the backports to REL1_31 and REL1_32.

Thank you @Krinkle and sorry for the revert earlier :-\

T218388#5036149 debug session by @kostajh is exactly how I used to track those kind of regressions. Namely:

• clone as few repositories as possible
• run tests with --filter until a repro is find then either revert/rollback/git bisect or insert print/return/continue statements.
• log thing and use colordiff (wrapper around diff with ansi coloring) / wdiff (word by word differences, colordiff supports colorizing it). Or: wdiff bad.log good.log | colordiff

From a CI perspective, we need to keep supporting older branches of MediaWiki that we run tests against, so the "migration" of php-ast from my perspective only ends once the last branch using the older version is phased out.

Tentatively that has been solved already?

Sorry, I am definitely not working on this one :-(

There are patch aboves, including one to update the composer merge plugin. I haven't had any review or feedback on those though.

Completely forgot about this one or got distracted with too many things. Anyway I am not actively working on this sorry.

The patch made it in v0.20.0 and it has been / is being deployed via T212774

Pending the puppet patch to be merged. Otherwise it is essentially solved :]

I have cherry picked https://gerrit.wikimedia.org/r/495681/3 on the CI puppetmaster and I now have:

Ok sury.org apt fetch seems fixed for now.

In Puppet we have a key added via T144872

$ gpg --list-options show-keyring ./modules/contint/files/sury-php.gpg
gpg: WARNING: no command supplied.  Trying to guess what you mean ...
pub   rsa4096 2014-09-09 [SC]
      DF3D585DB8F0EB658690A554AC0E47584A7A714D
uid           CZ.NIC Labs Archive Automatic Signing Key <ftpmaster@labs.nic.cz>
sub   rsa4096 2014-09-09 [E]

Sorry I have lost track of this ticket since March 7th but seems it had ample activity.

that is somehow due to ::profile::openstack::main::clientpackages. I have reverted https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/495757/ / ff5d0e075eb7ae9900d89dd5e865c8f1e7e8d971 and that fixed the apt issue

I have looked at the bots doc on:

My local email server configuration was screwed up. The bug is: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=925068

I have send a bug to Debian but I have not received the automatic acknowledgement yet :(

https://github.com/doxygen/doxygen/issues/3545

PHP was not parsed properly when it appeared in a <script language="php"> section.

Yes from http://php.net/manual/en/language.basic-syntax.phptags.php

At least PHP works since that generates https://doc.wikimedia.org/

The check pipeline is gone!

Superseded by https://gerrit.wikimedia.org/r/#/c/integration/quibble/+/497474

A LocalSettings.php file has been detected. To upgrade this installation, please run update.php instead

Note: you can skp Selenium tests with: quibble --skip selenium

This is intentional for now. The PHPUnit marked with @group Database are usually way slower than the Selenium tests and the intent is to fail fast. That being said we can revisit later.

I would rather have a single canonical documentation https://doc.wikimedia.org/quibble/ But surely we should have some introduction on https://www.mediawiki.org/wiki/Continuous_integration/Quibble

Gerrit is not reachable but here is the patch:

At least with git 2.11, the git submodule update seems to exit 1 when I kill the underlying git-remote-https subprocess.

I do not have any idea how to detect the submodule fetching failed. Maybe by running git submodule status and processing its output :/

Indeed, it did try to process the submodules but since Gerrit was not responsive/down the clones failed:

INFO:quibble.cmd:Updating git submodules of extensions and skins
extensions/VisualEditor/.gitmodules
+ git submodule foreach git clean -xdff -q
+ git submodule update --init --recursive
Submodule 'lib/ve' (https://gerrit.wikimedia.org/r/p/VisualEditor/VisualEditor.git) registered for path 'lib/ve'
Cloning into '/workspace/src/extensions/VisualEditor/lib/ve'...
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: clone of 'https://gerrit.wikimedia.org/r/p/VisualEditor/VisualEditor.git' into submodule path '/workspace/src/extensions/VisualEditor/lib/ve' failed
Failed to clone 'lib/ve'. Retry scheduled
Cloning into '/workspace/src/extensions/VisualEditor/lib/ve'...
fatal: could not read Username for 'https://gerrit.wikimedia.org': No such device or address
fatal: clone of 'https://gerrit.wikimedia.org/r/p/VisualEditor/VisualEditor.git' into submodule path '/workspace/src/extensions/VisualEditor/lib/ve' failed
Failed to clone 'lib/ve' a second time, aborting
...
INFO:backend.MySQL:Initializing MySQL data directory

For the IOS application, the development moved to Github to be able to use a third party that provides MacOS / Iphone etc as a service. That has been deemed easier and cheaper than having to figure out how to maintain Mac OS X system in a production environment or in the WMF Office It environment.

The Docker container is now based on Stretch and has java 8:

tox-pyspark (0.3.0) wikimedia; urgency=medium

I do not know whether it still occurs, I haven't checked. Maybe the SQL database is too slow :(

Sorry for lack of follow up, I have missed the task creation notification.

Looking again at top8 -1months highest average for tools.*webgrid-lighttpd*.cpu.total.user

I have filled this task when I have migrated the CI job for MediaWiki toward Docker. MariaDB came with the default configuration from the Debian packages:

8b45a9a4df8b2e56d4f262a23ea94d677d64dd58 introduced the job in Feb 2018 with:

  # Only run patch coverage if a PHP file was
  # changed on master.
- name: mediawiki-phpunit-coverage-patch
  branch: master
  files:
   - '^.*\.php$'

We have filled this task following Release-Engineering-Team May 2018 offsite. One of the need is for CI to be able to **optionally* install dependent extensions. The current system is centrally managed in integration/config.git and suffers from several issues:

• requires review/merge/deploy from one of the CI maintainer
• is not aware of branches

That makes total sense. Thank you!

Fixed by Upstream with https://secure.phabricator.com/D19967