After a chat with @TheresNoTime @dancy and @Legoktm on IRC.

The changes in the patch-performance or coverage have a low precedence and are only triggered after anything else.

To be fair after all those years using Gerrit it is probably the first time I encounter the yellow / blue diff. Looking at the DOM, the elements are marked with the CSS class dueToRebase which eventually leads me to:

Thanks for the confirmation @TheresNoTime !

Should be good now after I have restarted Zuul (ssh contint2001.wikimedia.org sudo systemctl restart zuul) which cleared the idling ssh connection.

From the Gerrit log https://logstash.wikimedia.org/app/dashboards#/view/AW1f-0k0ZKA7RpirlnKV

Max connection count for user jenkins-bot exceeded, rejecting new connection. currentSessionCount = 4, maxSessionCount = 4

o/ checking

Somehow the build workspace are belong to root:root:

drwxr-xr-x  3 root           root    4096 May 21 12:19 quibble-composer-mysql-php72-selenium-docker

In T308397#7942861, @jrbs wrote:

In T308397#7941759, @hashar wrote:

https://vote.wikimedia.org/ has been switched to language zh. The only culprit I find is that https://vote.wikimedia.org/ redirects to https://vote.wikimedia.org/wiki/%E9%A6%96%E9%A1%B5 ( 首页 ) which does not exist.

I just added a redirect. We could have a Main Page/zh with a translation but I don't think it's necessary :)

Reassigning to @TheresNoTime who found out the reason above (T308692#7939339). I have merely pushed the button ;)

I believe I have fixed the above issues. I have published the generated schemas at https://people.wikimedia.org/~hashar/T304947/schemas/

Thank you so much @thiemowmde for your assistance and to have taken some extra time to explain the last log messages I have posted. Follow up T308753 is an excellent idea to improve the user reporting, it kind of confused us since there is no message logged on the backend at error level so I was a bit clueless as to where the error was. Turns out it is "normal" behavior and does not deserve a backend error log ;)

Thanks @thiemowmde !

https://vote.wikimedia.org/ has been switched to language zh. The only culprit I find is that https://vote.wikimedia.org/ redirects to https://vote.wikimedia.org/wiki/%E9%A6%96%E9%A1%B5 ( 首页 ) which does not exist.

And for the failed to commit message I believe it is:

| May 19, 2022 @ 13:26:58.182 | FileImporter | mw1369 | commonswiki | FileImporter\Services\Importer::commitImportOperationsFailed to commit operations.
| May 19, 2022 @ 13:26:58.168 | FileImporter | mw1369 | commonswiki | FileImporter\Operations\FileRevisionFromRemoteUrl::commit failed to commit.
| May 19, 2022 @ 13:26:53.640 | FileImporter | mw1369 | commonswiki | Performing submit on ImportPlan for URL: https://zh.wikibooks.org/wiki/File:Wiki.png
| May 19, 2022 @ 13:26:53.640 | FileImporter | mw1369 | commonswiki | FileImporter\Services\Importer::import started
| May 19, 2022 @ 13:26:53.122 | FileImporter | mw1369 | commonswiki | Calculated two-hop interwiki prefix b:zh to zh.wikibooks.org
| May 19, 2022 @ 13:26:52.780 | FileImporter | mw1369 | commonswiki | Getting ImportPlan for URL: https://zh.wikibooks.org/wiki/File:Wiki.png

Looking at log for channel:FileImporter i see message:

Getting ImportPlan for URL: https://zh.wikibooks.org/wiki/File:Wiki.png
Calculated two-hop interwiki prefix b:zh to zh.wikibooks.org
ImportException: This page has been protected to prevent editing or other actions.

And

Getting ImportPlan for URL: https://zh.wikibooks.org/wiki/File:Wiki.png
Calculated two-hop interwiki prefix b:zh to zh.wikibooks.org
ImportException: File already on wiki

Sorry I was confused https://gerrit.wikimedia.org/r/c/mediawiki/extensions/FileImporter/+/793415 is for master and is the long term proper fix.

Thank you Tiemo! Given @Tgr looked at https://gerrit.wikimedia.org/r/c/mediawiki/extensions/FileImporter/+/793415 and that WMDE people are in a meeting this afternoon, I settle on deploying the change you have proposed. The worse case scenario is that FileImporter is as broken as it is currently.

Side track \MediaWiki\User\UserNameUtils::isUsable() has been changed by https://gerrit.wikimedia.org/r/c/mediawiki/core/+/786345 but that got deployed by wmf.10 and it merely replace a string by a constant.

+ @thiemowmde who has proposed the change to FileImporter

Should be good now

@TheresNoTime proposed a patch and fixed a few other links. It is blocked on an unrelated Phan build failure on the repo when being run under php 8.1 which is T308692
and has a fix :)

Confirmed. Thank you very much @Marostegui

With https://gerrit.wikimedia.org/r/791565 deployed , the CI servers have /var/cache/helm owned by helm:contint-admins and Puppet runs fine now :] Thank you @jbond

The agents could not connect:

10:12:07 <hashar> [05/18/22 08:06:32] [SSH] Copying latest remoting.jar...
10:12:07 <hashar> java.io.IOException: Could not copy remoting.jar into '/srv/jenkins/workspace' on agent

The package ship a /etc/default/jenkins which our Puppet erase and also comes with a systemd unit which we might consider using instead of our Puppet one.

My pull request has been merged and the maintainer addressed all the concerns directly via https://github.com/victools/jsonschema-generator/pull/255/ (merged as well).

One occurrence today with 1.39.0-wmf.10, there are two messages shown for the same reqId:

Not sure there is much I can do to help @Mhurd on that front, but I can surely finally read the python script which generates the metrics.db file and I have filed a task to get access to the Phabricator hosts T308478

I have made a bit more progress today and managed to get a Json Schema which validates a comment added event from the Gerrit Java class!

That is due to T307621 . The upstream repo was lagging behind and its master branch did not work with Gerrit 3.

We now ensure Scap test suite works against Debian Stretch, Buster and Bullseye. I even caught a python 3.9 incompatibility while doing so https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/775857 :)

Thank you to have taken the time to file this report. I have looked at the Gerrit server and added the stacktrace to this task description.

The latency is at ~ 100 ms which is good. I don't think there is any specific improvement to be made.

After the deployment of 4 threads for replication to codfw, we can see it is faster:

P27828 is a CommentAdded json event

Automatically generated schema for Gerrit CommentAdded event for T304947

For the archival of Gerrit repository it is done manually via Projects-Cleanup which has a form listing all the steps required. We had a long standing task to automatize that process which is T175499

Puppet fails on the contint* hosts cause /var/cache/helm went from being owned by wikidev to deployment, a group which does not exist on those hosts. Filed as T307740

From the contint1001 /var/log/puppet.log* files, the last good run was:

Apr 27 15:32:34 contint1001 puppet-agent[14311]: Caching catalog for contint1001.wikimedia.org
Apr 27 15:32:34 contint1001 puppet-agent[14311]: Applying configuration version '(891b0a4c36) Giuseppe Lavagetto - varnish: switch to using new-style request filters'
Apr 27 15:32:56 contint1001 puppet-agent[14311]: Applied catalog in 22.20 seconds

For each Gerrit patch set made to Vector, CI runs a command that takes a visual diff of master against that change/changes on top of master.

I first try a manual replication to gerrit2001 but only one thread was processing. I guess the plugin had to be reloaded somehow I have choose to restart Gerrit instead. And:

$ gerrit show-queue -w|grep -v waiting
+ ssh -p 29418 hashar@gerrit.wikimedia.org gerrit show-queue -w
Task     State        StartTime         Command
------------------------------------------------------------------------------
e8eb46b9              19:58:37.654      [a80eae9a] push gerrit2@gerrit2001.wikimedia.org:/srv/gerrit/git/pywikibot/pycolorname.git [..all..]
481132bd              19:58:37.653      [08001a65] push gerrit2@gerrit2001.wikimedia.org:/srv/gerrit/git/operations/software/bernard.git [..all..]
28fb9e88              19:58:37.652      [e8002665] push gerrit2@gerrit2001.wikimedia.org:/srv/gerrit/git/blubber-doc/example/calculator-service.git [..all..]
88340a4d              19:58:37.652      [482a92e5] push gerrit2@gerrit2001.wikimedia.org:/srv/gerrit/git/operations/debs/wikimedia-search-qa.git [..all..]
3dd0eb0f              19:59:07.419      [1dd3a71b] push git@github.com:wikimedia/analytics-log2udp2 [..all..]
...

I am not sure what happened but the deployment group does not exist on contint2001 / contint1001 (which has Helm)

Error: Could not find group deployment
Error: /Stage[main]/Helm/File[/var/cache/helm]/group: change from 'wikidev' to 'deployment' failed: Could not find group deployment

Although PHPUnit integration tests and QUnit are centrally set in mediawiki/core, webdriver.io tests are split in each repositories. That is the model also used for to the linters (which are run via composer test and npm test) and let us use different versions of webdriver.io. It is too challenging if not impossible to force migrate all repositories at the same time.

Indeed looks done.

I guess they can be replaced by:
https://grafana.wikimedia.org/d/000000321/zuul
https://grafana.wikimedia.org/d/000000284/continuous-integration

I have been using the dashboard Cloud VPS Project Board but it retrieves metrics from Graphite that is how I found out the integration instances have vanished.

This is an exact duplicate of T202710. It is due to /tmp being owned by root which mismatch the UID of the Xvfb process (nobody) and it thus spurts an error completely missing the fact it can actually write to /tmp.

I have deployed an experimental job which uses a Bullseye image and triggered it on https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/787694 by commenting check experimental. That leads to:

+ /src/wikimedia/fundraising/crm/bin/ci-create-dbs.sh
ERROR 1238 (HY000) at line 1: Variable 'innodb_file_format' is a read only variable

Before October 1st 2012, the code is my own and per my contract at the time: "source code contributed as part of this contract relationship will be licensed under an applicable open source license" and I hereby place it under Apache License 2 with copyright Antoine Musso <hashar@free.fr>. Then I don't know whether there is a lot left from this era beside the Rake puppet-lint, contint and jenkins modules.

In T225730#7889448, @Tgr wrote:

This is really getting frustrating for the wmf branches. E.g. gerrit 785944 spent an hour in CI, then errored out with Build timed out (after 60 minutes). Marking the build as failed.. gerrit 785941 took 92 minutes to merge. It's basically getting impossible to do an extension backport within the one-hour deploy window; not to mention multiple backports.

When trying the jobs yesterday they failed again. The reason is that the extension dependencies are not injected by Zuul since the change is in progress and not deployed. I have been testing them by manually triggering them on Jenkins, I have added a temp change https://gerrit.wikimedia.org/r/c/integration/config/+/790984 to inject EXT_DEPENDENCIES=mediawiki/skins/MinervaNeue\nmediawiki/extensions/MobileFrontend\nmediawiki/extensions/UniversalLanguageSelector.

Reopening since https://gerrit.wikimedia.org/r/c/mediawiki/services/wikispeech/mishkal/+/774770 still has to be reviewed and merged. I have reached by email @Sebastian_Berlin-WMSE and @kalle from Wikimedia Sweden.

It is finally fully deployed

I guess we want to:

• fix the CheckUser repo or rollback becae0cf2ac909c7ab7ce43ed7e94775ac24c5c3
• add Sqlite based jobs in CI to all those extensions and eventually all of them

The CI change https://gerrit.wikimedia.org/r/c/integration/config/+/653121 enforces Sqlite for AbuseFilter but since CI inject other extensions they should be checked against Sqlite as well. T307990 is about dropping Sqlite support in CheckUser which ends up causing AbuseFilter CI to break.

I have looked at the CI configuration (integration/config), AbuseFilter is apparently the sole extension running Sqlite based jobs on CI. That was done by @Jdforrester-WMF via https://gerrit.wikimedia.org/r/c/integration/config/+/653121 for T251967: quibble-vendor-sqlite-php72-docker is broken by AbuseFilter. The reason is the task about adding Sqlite and PostgreSQL support to AbuseFilter T199544.