I am assuming that one was related to yesterday GitHub outage.

I had a look at the plugin dev instance, created a new job and looked for:

• build steps, it has Conditional Step (single), Conditional Step (multiple).
• post build action (publishers), does not have the conditional step.

I don't remember why I went to use an unconditional execution of the castor-save-workspace-cache job (that was a decade ago T112560).

Reopening since there are still open changes to transition from sonar.login to sonar.token: https://gerrit.wikimedia.org/r/q/bug:T429547 Some can probably be raised to the #wmf-java channel on Slack :)

I have deleted the 3 Jenkins jobs (beta-code-update-eqiad, beta-scap-sync-world, beta-update-databases-eqiad) and the resulting view that became empty: https://integration.wikimedia.org/ci/view/Beta/

@Lars yes that is my suspicion and I will keep using that task for now since it has all the debugging context ;)

I'd like to find how the build is so much slower (15 vs 24 minutes when CPU speeds are not that different: 2.6GHz vs 2.1GHz). Maybe I can try to reproduce the CiviCRM on a fast vs a slow host and see whether I can find something, but maybe it is due the CPU power management.

I will try to reproduce the CiviCRM on a fast vs a slow host and see whether I can find something. I'd like to find how the build is so much slower (15 vs 24 minutes when CPU speeds are 2.9GHz vs 2.6GHz).

As a workaround, instead of using the roles:

Hosts: O:ci
Hosts: O:jenkins

I went to use the hosts I am interesting for:

Hosts: contint1002.wikimedia.org
Hosts: contint1003.wikimedia.org
Hosts: contint2003.wikimedia.org

The hotfix has been deployed. I have promoted wmf.15 to group 1 wikis and the error is no more happening 🎉

The npm cache directory has not reappeared which solves this issue:

$ sudo du -s -h /srv/castor/*/*/mwext-phpunit-coverage*/
314M	/srv/castor/castor-mw-ext-and-skins/master/mwext-phpunit-coverage-publish/

The npm cache directory has not reappeared which solves this issue.

du: cannot access '/srv/castor/*/*/*codehealth*/npm': No such file or directory

The list of CPU we for later reference:

Looking again at https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-bookworm/buildTimeTrend , there are fast builds happening on other hosts than integration-agent-docker-1093:

I went to query all 25 instances for the reported MHz using integration-cumin.integration.eqiad1.wikimedia.cloud:

sudo cumin --force -p 0 'name:docker' 'grep -m 1 MHz /proc/cpuinfo'

I have updated the 3 mwext-phpunit-coverage* jobs and nuked the /cache/npm directory. I am letting this open for a little while so we can verify the cache does not get populated back. That can be done using:

ssh integration-castor06.integration.eqiad1.wikimedia.cloud \
  sudo du -s -h /srv/castor/*/*/mwext-phpunit-coverage*/npm

Which was 11GB!!

Assigning to @Peter who found the root cause.

After bdev_enable_discard got rolled back ( https://gerrit.wikimedia.org/r/c/operations/puppet/+/1322878 ), the disk went well again on cloudcephosd1043 and the latency / disk business metrics I reported on T434024#12196170 look all fine now. Thank you!

@Xqt thank you for the detailed reply!

Back in Spring 2019, I have discovered some hardware hosts were quite slower than others, some fancy Xeon servers were slower than my laptop (the one I still have, but maybe I bought an over powered one and that was a smart choice since I had it for 7/8 years now) or than previous machine, an Intel NUC that was even older than that. Anyway, I digress.

I then looked at the underlying hosts for each instances, their metrics can be found on https://grafana.wikimedia.org/d/000000377/host-overview

I pick 6893 and 6894 because they both started at 7:00 am UTC which should be low traffic and they ran on different hosts. We can look at how busy each hosts was while the builds were running using https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board There is a panel showing the Pressure Stall Information (Kernel.org doc) which in short shows process waiting for CPU/IO/Memory.

That table is amazing. That made me remember Jenkins provides the duration for the last 50 builds which can be found at https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-bookworm/buildTimeTrend with a graph:

I think we need a few builds to check the behavior, at least it is no more timing out so I guess things have improved. May you build a table of how long each steps take now so it can be compared with the Reference (fastest) column in this task description?

I am not working on it, last time I acted on that front was a couple years ago at T371930 in order to add a new ssh key and is a subtask hence why this one is stallen. The other task has fallen under the radar.

And I think the problem is that mwext-codehealth-master-non-voting runs different containers which have those different cache configurations:

Very good finding @Peter ! I have enhanced the task description to give the whole context. The quick solution is to use quibble --skip-npm-install which was crafted by @Mhurd with T427922 :]

@Peter you did a Quibble run on your local machine which reproduced the large delay. May you retry it with https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1324303 applied? (quibble --change 1324303 ...) should download and apply it).

I have rolled Quibble 1.19.0 which now supports --skip-npm-install. @Mhurd can we sync up on deploying the change https://gerrit.wikimedia.org/r/c/integration/config/+/1321626 which updates the mwext-phpunit-coverage* jobs? Due to the timezone difference, maybe it is better done pairing with someone else than me (Release Engineering or Vaugh which I recently trained on the updating Jenkins jobs).

An implementation was proposed as https://gerrit.wikimedia.org/r/c/integration/config/+/1294992 and abandoned. Rather than skipping the cache saving, we should fix the root causes of the slowness. That is T427450 / T427471.

I am cleaning up the task related to mwext-codehealth-master-non-voting. I did the investigation in T427471, see T427471#11964274 and the following comments showing the issue are dupe npm cache and the large cypress files.

T434024#12196170 is an issue with the Ceph filesystem which certainly caused a slowdown. That one started on July 24/25 and has tentatively been fixed on Friday August 7th 20:00 UTC by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1322878 . I haven't looked at the performance of the civicrm job but that change at least got rid of timeouts though it is still slower than it used to be ( T434024#12196806 ). But I'll follow up on that other task. It is entirely possible there is another issue in the underlying infrastructure.

I intentionally did not do this, as I wanted to flush out the remaining Python 2 uses and fix them individually. :-(

I have added the PipelineLib configuration to CI ( https://gerrit.wikimedia.org/r/c/integration/config/+/1323849 ) and did a recheck of the refreshed Blubber config https://gerrit.wikimedia.org/r/c/3d2png/+/516709 which missed a .pipeline/config.yaml file. It now has a single test pipeline. I guess the dev variant should be build as well, and maybe published to the Docker registry?

Thank you for having split the case of 3d2png to a standalone task (T434378)

I have build the images and update the generic-node24 job via https://gerrit.wikimedia.org/r/c/integration/config/+/1323667 . I did not update the other jobs that will be for Monday, hence I am leaving this task open for now.

I went to look at wmcs dashboards, there are several of them for Ceph/OSD, given I know nothing about about the infra, I am pasting below my findings.

And from a Quibble / MediaWiki job, the daily average rose on July 25th (view over last 30 days and civicrm view over last 30 days:

And the changes we made to the docker-registry.wikimedia.org/releng/civicrm image:

civicrm (0.9-s4) wikimedia; urgency=high

i think @Mhurd and @Peter noticed the MediaWiki job became slower fairly recently.

Hello Given this affects PHP CodeSniffer which is the styling tool we use across a thousand of MediaWiki repositories × a few supported branches (releases, wmf, fundraising) that is going to be a few thousands of changes to upload to Gerrit and pass through CI.

Well done, I have approved the change which has been merged. It should deploy automatically on the beta cluster infrastructure over the next 10 minutes :]

Hello @Les4353, I have added you to the Trusted Contributors group in Gerrit, it brings you additional rights such as editing hashtags, topics, rebase or add a patch to another person change (use those two carefully!).

Thank you! I confirm I have managed to send an email via the wikis. The page took a while to process the send (which I guess correspond to the 4 seconds delay)

https://gerrit.wikimedia.org/r/1316125 got deployed and the new template should have been taken in consideration automatically. The template will be carried as we upgrade Gerrit. We will just have to remember to remove it from our Puppet repository once we upgrade to a version of Gerrit that has it, but it is not a big issue.

I gave some links to @diegodlh over IRC:

• https://www.mediawiki.org/wiki/Developer_account (they have one already)
• the tutorial to set up git/gerrit and interact with Gerrit https://www.mediawiki.org/wiki/Gerrit/Tutorial
• the alternative simple https://www.mediawiki.org/wiki/Gerrit_patch_uploader/en

TLDR: given the job running core+extensions together is now way faster (thanks @Peter & all), I think we can decline this and instead add ReadingLists to the gated extensions which is T403560 .

The code we wrote is now placed under GPL2.0 or later

@Paladox has let me know (thank you) that upstream has released 11.11.2 in June 2026 which includes my fix.

The backup is being conducted while the service is actively writing to LFS object files (rarely happens), git packfiles and objects. We also have git garbage collection triggering on a weekly basis.

From my list of action on T411583#12074321

This happens automatically when Gerrit is being stopped (more exactly when the H2 DB Driver is about to disconnect from the db). We had multiple controlled restart of Gerrit over the last few days and the files got shrunk:

-rw-r--r-- 1 gerrit gerrit 762M Jul  9 14:50 git_file_diff.h2.db
-rw-r--r-- 1 gerrit gerrit 767M Jul  9 14:50 gerrit_file_diff.h2.db
-rw-r--r-- 1 gerrit gerrit 774M Jul  9 14:50 conflicts.h2.db
-rw-r--r-- 1 gerrit gerrit 785M Jul  9 14:48 diff_intraline.h2.db
-rw-r--r-- 1 gerrit gerrit 832M Jul  9 14:49 change_kind.h2.db
-rw-r--r-- 1 gerrit gerrit 845M Jul  9 14:50 mergeability.h2.db
-rw-r--r-- 1 gerrit gerrit 1.9G Jul  9 14:49 diff_summary.h2.db
-rw-r--r-- 1 gerrit gerrit 2.2G Jul  9 14:49 comment_context.h2.db

We have historically not backed up the search indices. We had them backed up by mistake as part of some path change and they take a sizeable amount of disk space, they are also ever growing (T411583)

+ doc.wikimedia.org since the repo is shared with https://doc.wikimedia.org/

I am claiming 1.47.0-wmf.10 to have successfully rolled out.

Self note review the two cases in T352319#10288944

I am pretty sure that is the same as T352319 and is an issue within the Jenkins Parameterized Build plugin

🎉 Thank you for having confirmed the resolution. Very much appreciated!

If I try to push an existing image on the old host:

contint1002$ docker --config /etc/docker-pusher push docker-registry.wikimedia.org/releng/release-notes:0.1.0-s1
The push refers to repository [docker-registry.wikimedia.org/releng/release-notes]
...
357ab391beed: Layer already exists 
unauthorized: authentication required

Jul 09 11:17:52 contint1002 dockerd[1160]: time="2026-07-09T11:17:52.996083076Z" level=info msg="Attempting next endpoint for push after error: unauthorized: authentication required"

I have confirmed the credential file has the same auth information on the old host (contint1002) and the new host (contint1003). It is provisioned by Puppet.

The docker daemon has:

Jul 09 09:41:01 contint1003 dockerd[1293]:
time="2026-07-09T09:41:01.925689187Z"
level=error
msg="Upload failed: unauthorized: <html>\r\n<head><title>401 Authorization Required</title></head>\r\n<body>\r\n<center><h1>401 Authorization Required</h1></center>\r\n<hr><center>nginx/1.22.1</center>\r\n</body>\r\n</html>\r\n"

Thanks @kevinbazira! The build ran on contint1003 which is a production agent and as such should have the credentials to push to the registry (to be verified). From the console log, the build does:

+ sudo /usr/local/bin/docker-pusher docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services-tts:2026-07-09-094058-publish
The push refers to repository [docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services-tts]
...
0c28ce2f5fb5: Waiting
unauthorized: <html>
<head><title>401 Authorization Required</title></head>
...

TLDR must_not without Exclude results ends up being an OR filter.

must_not

Logical not operator. All matches are excluded from the results. If must_not has multiple clauses, only documents that do not match any of those clauses are returned.
For example, "must_not":[{clause_A}, {clause_B}] is equivalent to NOT(A OR B).

Good catch, and for the record the filter had the custom label NOT T394850 - JsonConfig: Undefined array key with the OpenSearch Query DSL being:

 {
  "query": {
    "bool": {
      "must_not": [
        {
          "match_phrase": {
            "error.stack_trace.text": "extensions/JsonConfig/includes/GlobalJsonLinksQuery.php(274)"
          }
        },
        {
          "match_phrase": {
            "error.message": "PHP Warning: Undefined array key"
          }
        }
      ]
    }
  }
}