I have deployed all the patches (thank you for the refs/meta/config patch) and deleted the GitHub repository. I could not find a repository in Diffusion, maybe it was never created?

Thanks! So the summary (for my later self) is:

Translate had a similar one at T322439 and was fixed by https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Translate/+/852934/1/TranslateHooks.php

(blocked on T345695: Deal with tox 4 upgrading cause the new images are now using tox v4 and we gotta migrate the repositories before being able to add new packages)

Looks like pypy has always been in the CI tox image. It got created in 2017 based on Jessie and the original change did install the pypy package ( https://gerrit.wikimedia.org/r/c/integration/config/+/377337 ). That got carried in the releng/tox-buster image created in https://gerrit.wikimedia.org/r/c/integration/config/+/643776 and the package is still present:

$ docker run --pull --rm -it --entrypoint=pypy docker-registry.wikimedia.org/releng/tox-buster:latest --version
Python 2.7.13 (7.0.0+dfsg-3, Mar 24 2019, 15:30:18)
[PyPy 7.0.0 with GCC 8.3.0]

Read/writes should be faster now that the instance has the 4xiops flavor (which quadruples the number of IO that can be done).

I have updated all the Jenkins jobs to use Quibble 1.5.6 which now spawns MariaDB with --innodb-print-all-deadlocks. Thus if I got it right, when a deadlock occurs details should be available in the log attached to the builds as log/mysql-error.log.

The Docker images have been built and published:

Successfully published image docker-registry.wikimedia.org/releng/mediawiki-phan-php74:0.2.2-s1
Successfully published image docker-registry.wikimedia.org/releng/composer-package-php73:0.5.1-s1
Successfully published image docker-registry.wikimedia.org/releng/composer-scratch:2.6.4
Successfully published image docker-registry.wikimedia.org/releng/quibble-buster-php81:1.5.6-s1
Successfully published image docker-registry.wikimedia.org/releng/composer-test-php80:0.1.1-s1
Successfully published image docker-registry.wikimedia.org/releng/composer-package-php81:0.0.2-s1
Successfully published image docker-registry.wikimedia.org/releng/quibble-buster:1.5.6-s1
Successfully published image docker-registry.wikimedia.org/releng/composer-test-php81:0.0.2-s1
Successfully published image docker-registry.wikimedia.org/releng/composer-security-check:1.2.2-s1
Successfully published image docker-registry.wikimedia.org/releng/composer-php81:0.0.2-s1
Successfully published image docker-registry.wikimedia.org/releng/composer-php82:0.1.0-s3
Successfully published image docker-registry.wikimedia.org/releng/quibble-buster-php74:1.5.6-s1
Successfully published image docker-registry.wikimedia.org/releng/composer-package-php80:0.0.5-s1
Successfully published image docker-registry.wikimedia.org/releng/mediawiki-phan-php80:0.1.2-s12
Successfully published image docker-registry.wikimedia.org/releng/quibble-buster-php82:1.5.6-s1
Successfully published image docker-registry.wikimedia.org/releng/node18-test-browser-php80-composer:0.1.0-s1
Successfully published image docker-registry.wikimedia.org/releng/quibble-buster-php80:1.5.6-s1
Successfully published image docker-registry.wikimedia.org/releng/mediawiki-phan-taint-check-demo:0.2.0-s1
Successfully published image docker-registry.wikimedia.org/releng/mediawiki-phan-php82:0.0.2-s1
Successfully published image docker-registry.wikimedia.org/releng/composer-package-php82:0.1.1-s1
Successfully published image docker-registry.wikimedia.org/releng/composer-test-php82:0.1.1-s1
Successfully published image docker-registry.wikimedia.org/releng/composer-php73:0.4.1-s1
Successfully published image docker-registry.wikimedia.org/releng/phpmetrics:2.8.1-s5
Successfully published image docker-registry.wikimedia.org/releng/quibble-buster-php74-coverage:1.5.6-s1
Successfully published image docker-registry.wikimedia.org/releng/mediawiki-phan-testrun:0.3.2-s1
Successfully published image docker-registry.wikimedia.org/releng/quibble-buster-php73:1.5.6-s1
Successfully published image docker-registry.wikimedia.org/releng/quibble-fresnel:1.5.6-s1
Successfully published image docker-registry.wikimedia.org/releng/node20-test-browser-php80-composer:0.1.0-s1
Successfully published image docker-registry.wikimedia.org/releng/mediawiki-phan-php73:0.3.2-s1
Successfully published image docker-registry.wikimedia.org/releng/composer-php80:0.0.2-s19
Successfully published image docker-registry.wikimedia.org/releng/composer-test-php73:0.3.1-s1
Successfully published image docker-registry.wikimedia.org/releng/node16-test-browser-php80-composer:0.4.0-s1
Successfully published image docker-registry.wikimedia.org/releng/composer-package-php74:0.4.2-s1
Successfully published image docker-registry.wikimedia.org/releng/composer-php74:0.3.2-s1
Successfully published image docker-registry.wikimedia.org/releng/mediawiki-phan-php81:0.0.3-s1
Successfully published image docker-registry.wikimedia.org/releng/composer-test-php74:0.1.2-s1

SRE had an initiative on that front for their operations/puppet.git repository T148494: Add shell scripts CI validations

I have updated all the Jenkins jobs to use Quibble 1.5.6 which now spawns MariaDB with --innodb-print-all-deadlocks. The log should be attached to the builds as log/mysql-error.log.

Indeed it is a bug in the Gerrit web interface. See my earlier comment T328843#8588537 (nonetheless, thank you to have filed this issue!).

The built-in defaults for reference:

If I look at the Jenkins console HTML, the poor contrast lines have style="color: #E5E5E5" which is a light grey. In the XML configuration that is WHITE from the Normal map (and would be a #FFFFFF in the Bright map).

Then ANSI sequences are rendered by the Jenkins ANSI Color Plugin. I has some several mappings between the ANSI colors and the HTML/CSS rendered color. We use xterm and the defaults are defined at https://github.com/jenkinsci/ansicolor-plugin/blob/master/src/main/java/hudson/plugins/ansicolor/AnsiColorMap.java#L79

The above can probably be added to docker-pkg README.md :)

The Docker daemon runs as root and having access to it effectively grants full root permission on the host (I guess something like docker run --user=root --entrypoint=bash -v /:/rootfs).

And to complete I have added panels to the cloud grafana instance to show the number of disk IO in progress as well as the read/write throughput.

I have recreated the instance with the same hostname integration-castor05. Since the ssh host key changed I had to manually verify the ssh key in Jenkins.

As a followup, I had to recreate the instance and reattach the volume but it was not magically mounted to /srv. Andrew said prepare-cinder-volume creates an fstab entry but since I have recreated the instance it was gone.

TLDR: I don't know what is going on :-]

In both case they run

+ php
-d extension=pcov.so
-d pcov.enabled=1
-d pcov.directory=/workspace/src
-d 'pcov.exclude=@(tests|vendor)@' vendor/bin/phpunit
--exclude-group Dump,Broken,ParserFuzz,Stub
--coverage-clover /workspace/log/clover.xml
--log-junit /workspace/log/junit.xml tests/phpunit/unit

For MediaWiki core, the analysis of the branch happens after a change has been merged (postmerge pipeline in Zuul) by triggering https://integration.wikimedia.org/ci/job/mwcore-codehealth-master-non-voting/

Also have to update the Jenkins job:

jjb/castor-load-sync.bash:19:CASTOR_HOST="${CASTOR_HOST:-integration-castor03.integration.eqiad.wmflabs}"

In T347320#9198371, @Stegmujo wrote:

@taavi How can I flag this task or add a suitable assignee, it seems to be an issue with CI ?

I have cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/961025 on the integration Puppet master:

On an agent that executed a successful build:

hashar@integration-agent-docker-1055:~$ sudo su - jenkins-deploy
jenkins-deploy@integration-agent-docker-1055:~$ ssh -p 29418 gerrit.wikimedia.org
wdqsguibuilder@gerrit.wikimedia.org: Permission denied (publickey).

The Jenkins job clones wikidata/query/gui and shells out to scripts/set-up-git.sh which:

git clone "https://gerrit.wikimedia.org/r/wikidata/query/gui-deploy" ./build
git -C ./build \
    remote set-url --push origin \
    ssh://wdqsguibuilder@gerrit.wikimedia.org:29418/wikidata/query/gui-deploy

In T287900#8588779, @hashar wrote:

git grep integration_composer

php5-sssd/base/Dockerfile.template:{{ macros.integration_composer() }}
php72-sssd/base/Dockerfile.template:{{ macros.integration_composer() }}
php73-sssd/base/Dockerfile.template:{{ macros.integration_composer() }}

Which are based respectively on Debian Jessie, Stretch and Buster. I don't know whether those images are still of any use though.

As far as I can tell, the three affected repositories are

In T346868#9182555, @Daimona wrote:

Indeed, caused by r938387, and also r814359 which caused another failure further down the line.

I have fixed wdio-mediawiki to await browser.saveScreenshot but through Code Search https://codesearch.wmcloud.org/search/?q=saveScreenshot&files=&excludeFiles=&repos= I found a couple more in CirrusSearch and Wikibase

I have restarted Gerrit and the Sign up link now points to https://idm.wikimedia.org/signup/

drop support for Python 3.5 with Pywikibot 9.0

With tox 4, skipsdist = True cause it to always skip the package installation (package=skip) regardless of the value of usedevelop which is not even recognized when skipsdist is set. We should thus remove it. With both tox v3 and v4, usedevelop causes the sdist to be skipped.

The package has been published at https://pypi.org/project/datadraw/ but I haven't been able to find the source code repo though the author pushed a few examples at https://github.com/pepprseed/datadraw-examples/ It looks like a Python API to generate graph and I don't think it supports reading a ploticus script. Then the author being the same, maybe there is a 1/1 mapping between ploticus script and the datadraw API, or at least a close one.

After a quick check on integration-agent-pkgbuilder-1001 and integration-agent-pkgbuilder-1002 it looks like the old packages no more accumulate. I am assuming the same happens on the production build host and thus claiming this issue to be resolved.

I am guessing the stracktrace is generated by PHPUnit itself. Maybe it can be hooked with MediaWiki MWExceptionHandler /MWExceptionRenderer (which supports redacting function parameters).

I am guessing @jbond might be able to build Python 3.10 & 3.11.

I have noticed repositories get migrated but are not archived in the old system so they end up still being maintained despite the Gerrit repository no more being the canonical place.

The g3.cores8.ram36.disk20.4xiops flavor has been created (T345925). Given the current flavor and the new one do not have ephemeral disk, I should be able to resize the instance without having to recreate it.

The flavor name is created by a cookbook; I'm not sure that existing flavors can be easily renamed but I'll poke in the DB.

@NealMcB I have filed this task as a placeholder following a discussion I had with @Quiddity. Since there was a task upstream ( Framadate issue 158 ) and the task got assigned to someone just a week before our chat, I felt that maybe I could try to convince upstream to implement the feature for @Quiddity .

My intent was to let @Mabualruz run a backport during the training which in turns require access to the deployment group hence why I came back to this task. Would be great to have the access granted today so we can do the training tomorrow.

Well done! :)

I can't see the g3.cores8.ram36.disk20.4xiops in Horizon. I guess it has been created but hasn't been made available to the integration project?

Thank you for the fix, that was hitting Quibble integration tests (which I filed as the duplicate task T346604). I have deployed the wmf backport ( https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Flow/+/957872 ).

After rechecking a change to Quibble ( https://gerrit.wikimedia.org/r/c/integration/quibble/+/952016 ) the build passed. I suspect that got solved by https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Flow/+/957332 tests: Do not assume UTSysop exists for T346253

I have removed the cherry pick from the integration Puppet master. I am bailing out on ensuring an initial provisioning works on the first time given we usually rebuild those instances every couple years :)

I have poked #wikimedia-cloud-admin to potentially get the flavor renamed to use a 4xiops suffix rather than integration.

We did some training / presentation on Thursday 15th September. Unfortunately there was nothing worthwhile to deploy and Mohd missed access to the deployments group (T342535). Will try again next week :)

This is still valid. I guess the issue is finding out the right fields to use and crafting the visualization on the Biterg.io instance at https://wikimedia.biterg.io/app/kibana#/dashboard/8c515590-e1de-11e8-8aac-ef7fd4d8cbad :)