Page MenuHomePhabricator
Create Task
Maniphest T422488

Rebuild integration-agent-qemu-1003 from Bullseye to Bookworm
Closed, ResolvedPublic
Actions

Description

Fresh uses a VM for testing docker commands. The CI job runs on integration-agent-qemu-1003 is based on Bullseye, we need to rebuild it with Bookworm in order for Jenkins to connect to it using Java 21 (T421114).

The system was created in early 2020 T250808 using Debian Stretch. It was further rebuild with Debian Bullseye mainly to get access to a newer version of QEmu (T284774).

The instance requires kvm acceleration (T250808#6075448)

The current instance was created Sept. 17, 2021.

There is a build script in operations/puppet:

I have found in my home directory a script ci-create-qemu-image.sh which was last modified January 31st 2022. I have no idea whether I wrote it or whether I have adjusted an existing script:

ci-create-qemu-image.sh3 KBDownload

And a script in @dancy homedir from January 2022, seemingly for T248779

P90322 Qemu provision script by Ahmon for T248779?
1# Found in 2026 as part of T422488 (Rebuild QEmu Jenkins agent
2# Script by Ahmon Dancy, last modified Feb 4th 2022
3
4ssh_key=testkey
5
6if [ ! -f $ssh_key ]; then
7 ssh-keygen -N '' -f $ssh_key
8fi
9
10# Packages to add
11base_pkgs='ssh,git,docker.io'
12# Packages for mw-cli jobs T248779
13mw_cli_job_pkgs='curl,make,python3-dev,libffi-dev,gcc,libc-dev,cargo'
14
15pkgs="$base_pkgs","$mw_cli_job_pkgs"
16
17# testing
18pkgs="ssh"
19
20# the stock virtual disk is 6GB, so no need to resize at this time
21
22time virt-builder \
23 --smp 4 \
24 --ssh-inject "root:file:$ssh_key.pub" \
25 --append-line "/etc/network/interfaces:auto ens3" \
26 --append-line "/etc/network/interfaces:iface ens3 inet dhcp" \
27 --install "$pkgs" \
28 --firstboot-command "dpkg-reconfigure openssh-server" \
29 -o vm.img \
30 debian-11

The provisioning process is documented at https://www.mediawiki.org/wiki/Continuous_integration/Qemu . I did refresh it in February 2022 with the last rebuild ( T284774 ).

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
ci: enhance ci-build-images scriptoperations/puppetproduction+59 -29
Test CI with new nodefreshmaster+0 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Tue, Apr 7, 1:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Apr 7, 1:05 PM
hashar added a parent task: T421114: Rebuild all Jenkins agents VM to Bookworm to support Java 21.Tue, Apr 7, 1:06 PM
hashar mentioned this in T421114: Rebuild all Jenkins agents VM to Bookworm to support Java 21.
hashar added a comment.Tue, Apr 7, 2:00 PM

I have created a new integration-agent-qemu-1004 instance with Puppet class role::ci::agent::qemu. I ran Puppet and the image creation failed:

grub-install: warning: File system `ext2' doesn't support embedding.
grub-install: error: embedding is not possible, but this is required for cross-disk install.
virt-customize: error: grub-install /dev/sda: command exited with an error
full log
Downloading base image from Debian
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

Veryfing checksum of Debian image 'debian-11-nocloud-amd64-20220121-894.qcow2'
debian-11-nocloud-amd64-20220121-894.qcow2: OK
Creating a grown image with 5G disk size
Formatting 'grown.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=5368709120 lazy_refcounts=off refcount_bits=16
[   0.0] Examining debian-11-nocloud-amd64-20220121-894.qcow2
**********

Summary of changes:

virt-resize: /dev/sda14: This partition will be left alone.

virt-resize: /dev/sda15: This partition will be left alone.

virt-resize: /dev/sda1: This partition will be resized from 1.9G to 4.9G.  The filesystem ext4 on /dev/sda1 will be expanded using the ‘resize2fs’ method.

**********
[  41.4] Setting up initial partition table on grown.qcow2
[  67.3] Copying /dev/sda14
[  67.8] Copying /dev/sda15
[  69.7] Copying /dev/sda1
[ 118.6] Expanding /dev/sda1 (now /dev/sda3) using the ‘resize2fs’ method

virt-resize: Resize operation completed with no errors.  Before deleting the old disk, carefully check that the resized disk boots and works correctly.
Generating a ssh key pair 'sshkey_qemu_root_v1'
Generating public/private rsa key pair.
Your identification has been saved in sshkey_qemu_root_v1
Your public key has been saved in sshkey_qemu_root_v1.pub
The key fingerprint is:
SHA256:6UqbR8ngwgEi6xJmwWstNtSf4x80vRb/O8zlJS/L20o root@integration-agent-qemu-1004
The key's randomart image is:
+---[RSA 3072]----+
|. .              |
|o+..             |
|oo+.. . .        |
|oO ..+.o +       |
|*.o..oooS.+      |
|..  o..o+o .  ...|
|.    .o.+   + Eo.|
|     . =.    B.o.|
|      +.     .B=.|
+----[SHA256]-----+
Creating our CI specific qcow2 layer
Formatting 'debian-11-ci.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=5368709120 backing_file=grown.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
[   0.0] Examining the guest ...
[  26.7] Setting a random seed
virt-customize: warning: random seed could not be set for this type of guest
[  27.0] Setting the machine ID in /etc/machine-id
[  27.0] Running: grub-install /dev/sda
Installing for i386-pc platform.
grub-install: warning: File system `ext2' doesn't support embedding.
grub-install: error: embedding is not possible, but this is required for cross-disk install.
virt-customize: error: grub-install /dev/sda: command exited with an error

If reporting bugs, run virt-customize with debugging enabled and include the complete output:

  virt-customize -v -x [...]
Error creating CI image debian-11-ci.qcow2
removed 'debian-11-ci.qcow2'
gerritbot added a comment.Tue, Apr 7, 4:23 PM

Change #1268594 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] ci: enhance ci-build-images script

https://gerrit.wikimedia.org/r/1268594

gerritbot added a project: Patch-For-Review.Tue, Apr 7, 4:23 PM
hashar added a comment.Tue, Apr 7, 4:23 PM

I reran the ci-build-images script and it eventually has completed. I took sometime to enhance the script I wrote the last time. I will verify whether the image works properly.

hashar claimed this task.Tue, Apr 7, 4:23 PM
hashar moved this task from Inbox to Support & Meta on the Fresh board.
gerritbot added a comment.Tue, Apr 7, 8:28 PM

Change #1268655 had a related patch set uploaded (by Hashar; author: Hashar):

[fresh@master] Test CI with new node

https://gerrit.wikimedia.org/r/1268655

hashar updated the task description. (Show Details)Wed, Apr 8, 6:23 AM
hashar added a subscriber: dancy.
gerritbot added a comment.Wed, Apr 8, 6:23 AM

Change #1268655 abandoned by Hashar:

[fresh@master] Test CI with new node

Reason:

It worked!

https://gerrit.wikimedia.org/r/1268655

Stashbot added a comment.Wed, Apr 8, 6:24 AM

Mentioned in SAL (#wikimedia-releng) [2026-04-08T06:24:57Z] <hashar> Deleted integration-agent-qemu-1003 Bullseye image, replaced by integration-agent-qemu-1004 which is on Bookworm # T422488

hashar closed this task as Resolved.Wed, Apr 8, 6:26 AM

@dancy you got subscribed because before deleting the integration-agent-qemu-1003 instance I found a script in your home dir from January 2022. I have saved it at P90322. It probably does not have much value, but at least it is not lost :-]

I have verified last night the fresh-test CI job works properly on the new integration-agent-qemu-1004 Bookworm instance and I have deleted integration-agent-qemu-1003 (Bullseye based).

Solved!

thcipriani added projects: Release-Engineering-Team (Doing 😎), Essential-Work.Fri, Apr 10, 10:59 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits