We did a rebuild of quibble-for-mediawiki-core-vendor-postgres-php83 with Quibble 1.17.1 which completed in 14m 09s.

We have confirmed the job is now running the npm install in mediawiki/core which makes mocha available. It is solved!!!

I am closing this since CI is working again. I have mentioned the revert and the error on T420964 for follow up actions.

I have reverted

After a quick check on https://gerrit.wikimedia.org/r/q/project:mediawiki/core+status:open that fails for any patch so I will revert the Vector patch. To be investigated is why CI did not catch it.

Patch recently merged in mediawiki/skins/Vector:

This has been caused by Quibble being upgraded to 1.17.0. Specially due to T418743 Skip extra npm install for browser test only jobs. That is an optimization to prevent running needless npm ci and save building time so that is now only run for qunit.

We have deployed T419567: Splitting GrowthExperiments from gated extensions this morning

The implementation got split in two subtasks:

• T419574 and the job got added in Jenkins via integration/config 1268960.
• T417412 for the Zuul configuration. The workflow logic got implemented in integration/config 1268559.

That is fantastic! Thank you so much Ahmon.

This happened while backporting https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1269334 . The sequence was:

The already cut release branches still have selenium-test and have the selenium tests running. A more recent example is https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/1269414 made for REL1_45 it does run the browser test since the npm script is still selenium-test.

As follows-up to our discussion this morning with Michael, Peter and Zeljko, because the selenium tests are now hidden we need to:

As follows-up to our discussion this morning with Michael, Peter and Zeljko, because the selenium tests are now hidden we need to:

Thank you @dduvall! I have missed PipelineLib treated labels as capabilities when the rest of the Jenkins job use it to differentiate between WMCS and production agents. With PipelineLib now having dedicated tags that clears the mixed use of those labels.

I have filed this task as a follow up to the incident T417497 (CI losts Gerrit events for several days after the switchover behind the CDN). The workaround I found was to bypass the TCP proxy and have Zuul to connect directly to Gerrit.

Stuff to find: where is sdb hardcoded? ;-]

So we have in the labs_lvm module a default of sda:

class labs_lvm( 
    Stdlib::Unixpath $disk      = '/dev/sda',

I asked WMCS admins:

<andrewbogott> hashar: unfortunately the drive label assignment is undefined by the OS, so it's not really possible to control which is which. It might be that puppet could be smarter about detecting the right one...

@dancy you got subscribed because before deleting the integration-agent-qemu-1003 instance I found a script in your home dir from January 2022. I have saved it at P90322. It probably does not have much value, but at least it is not lost :-]

I reran the ci-build-images script and it eventually has completed. I took sometime to enhance the script I wrote the last time. I will verify whether the image works properly.

This was most probably created for T284774 and later became modules/profile/files/ci/ci-build-images.sh in Puppet. See https://gerrit.wikimedia.org/r/c/operations/puppet/+/758514

I have created a new integration-agent-qemu-1004 instance with Puppet class role::ci::agent::qemu. I ran Puppet and the image creation failed:

grub-install: warning: File system `ext2' doesn't support embedding.
grub-install: error: embedding is not possible, but this is required for cross-disk install.
virt-customize: error: grub-install /dev/sda: command exited with an error

Downloading base image from Debian
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

Note, as part of rebuilding the VM (T421114) I have deleted all the agents that had only 2 executors and the 1059 instance that had 16vCPU. The whole fleet now uses 10vCPU and the agents have 3 executors. Some agents can be changed to use 2 executors if you need the data points.

I have created some new virtual machines this morning, most had the ephemeral disk as sda but integration-agent-docker-1089.integration.eqiad1.wikimedia.cloud got it on sdb and successfully provisioned:

lsblk
NAME                     MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda                        8:0    0  100G  0 disk 
├─vd-docker              254:0    0   45G  0 lvm  /var/lib/docker
└─vd-second--local--disk 254:1    0   55G  0 lvm  /srv
sdb                        8:16   0   20G  0 disk 
├─sdb1                     8:17   0 19.9G  0 part /
├─sdb14                    8:30   0    3M  0 part 
└─sdb15                    8:31   0  124M  0 part /boot/efi

Though I rebooted it at some point? Looking at /var/log/kern.log, the first boot had:

kernel: [    1.620977] sd 2:0:0:0: [sda] 41943040 512-byte logical blocks: (21.5 GB/20.0 GiB)
kernel: [    1.623524] sd 2:0:0:1: [sdb] 209715200 512-byte logical blocks: (107 GB/100 GiB)

Thus Puppet ran with the ephemeral disk space at sdb. After reboot they got swapped

kernel: [    1.549002] sd 2:0:0:1: [sda] 209715200 512-byte logical blocks: (107 GB/100 GiB)
kernel: [    1.551251] sd 2:0:0:0: [sdb] 41943040 512-byte logical blocks: (21.5 GB/20.0 GiB)

I have updated PipelineLib to have the jobs to point to the Docker label instead of blubber and removed the label from all the Jenkins agents.

In T421827#11785489, @Jdforrester-WMF wrote:

The git commands are done from the zuul-cloner docker image that's build based on integration/zuul.git (our very dated decade+-old fork of upstream).

@ABran-WMF this T246763 is the canonical issue for the mod_proxy → Jetty path. I have reopened it because although it happens less frequently, it still hasn't been complete fixed. I gave some hypothesis when reopening at T246763#11637703

Awesome thank you for having confirmed it! :]

@Xqt it is deployed, sorry I have should done it earlier but I only noticed quite late in the evening on Wednesday and I got busy :]