Changes are in production

Rollout plan:

• Scale down services:
  • kubectl -n patchdemo scale deployment patchdemo --replicas=0
  • kubectl -n control-plane scale deployment catalyst-api-deployment --replicas=0
• Cordon master: kubectl cordon k3s
• Stop master & workers:
  • sudo systemctl stop k3s
  • sudo systemctl stop k3s-agent
• Move K3s' config in /etc to the data volume (all nodes):
  • sudo mv /etc/rancher /mnt/k3s-data
  • sudo ln -s /mnt/k3s-data/rancher /etc/rancher
• Apply stack to master node only: tofu apply -exclude='module.envDB_node.openstack_compute_instance_v2.node' -exclude='module.envDB_k3s.terraform_data.install_k3s' -exclude='module.worker_nodes["worker01"].openstack_compute_instance_v2.node' -exclude='module.workers_k3s["worker01"].terraform_data.install_k3s' -exclude='module.worker_nodes["worker02"].openstack_compute_instance_v2.node' -exclude='module.workers_k3s["worker02"].terraform_data.install_k3s'
• Stop K3s on master temporarily and disable node lifecycle controller. This will keep the cluster from trying to reschedule workloads while workers are down:
  • sudo systemctl stop k3s
  • In /etc/rancher/k3s/config.yaml add:

kube-controller-manager-arg:
  - "controllers=*,-nodelifecycle"

• sudo rm -f /var/lib/rancher/k3s/server/cred/passwd
• sudo systemctl start k3s

We have been using the autoscaler in production for a couple of days and things looks stable. I've adapted the quota increase request to match the resources mentioned here: https://www.mediawiki.org/wiki/Patch_demo/Code_server#1._KEDA_autoscaling_works

As it turns out an OpenStack VM can only be added to a server group on creation time, it's not possible to assign the VM to a group afterwards. This has a significant impact on the scope of the story as we will need to recreate our existing VMs.

We are also seeing less frequent hits for function 'Z0':

T426874 is a prereq for this task

Changes are in production and behavior looks as expected. I'm going to monitor this for a while but for now I'm calling it done

In T430833#12176845, @Bugreporter wrote:

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/cldr/+/1308044 and related patches (except last one) should be merged by someone with +2. They are already backported to wmf.13

Reading through the comment history it seems wmf.13 was used to run some performance checks on the patches in question, but that the patches themselves are not a blocker for wmf.14.

There's a claim here that a number of patches related to this task should be merged into main: https://phabricator.wikimedia.org/T430833#12176845 for the wmf.14 train. Is there someone here who can provide +2? Are those changes indeed a blocker?

We have put together a wiki page giving context to this request: https://www.mediawiki.org/wiki/Patch_demo/Code_server

I got the autoscaler working for the code server on my local cluster.

Both the dev and prod clusters now support the creation of Cinder-backed persistent volumes.

Pinging @Andrew since the Catalyst team originally discussed this quota increase with him

Unfortunately I don't see any PHP-FPM logs in t432284.catalyst.wmcloud.org.

In T384584#12123161, @santhosh wrote:

Hi @gkyziridis, I am planning to take up this. Hope you dont mind. I will work on top of your commits.

@dduvall Could you please give me the permissions to submit merge request to blubber repo? My gitlab username is santhosh

The environments created by the catalyst-daily-core job use the shared env DB, meaning that the envs are creating their own schema/DB in a common DB server. That is the same server used by many of the other CI jobs and, notably, Patchdemo. None of those other components are showing the same hiccups when trying to connect to their respective schemas, which makes me suspect the problem is an artifact of the tests + the MW code itself, not the DB infra. It's possible the tests are exposing a real issue in the code.

The error seems to be coming from the MW persistence layer. It may be flaky infrastructure, but could also be an actual issue in the code (e.g. a deadlock) — the error is too generic to be able to tell.

Thank you!

In the future please use https://cloudvps-quota.toolforge.org to file quota request -- thank you !

Is the issue that it's being monitored even though it's not the active server?

Plugins updated in the Jenkins releases instances

Files have now been removed and replaced with a README pointing to their new location in the Helm chart.

Done. No deployment was necessary after the changes. Prod & dev can now be assigned different resource quotas

In T418299#12025527, @Dzahn wrote:

@jnuche You think we can close this as resolved again? Did I cover the issue?

@Dzahn at least mount is now a valid unit type thanks to your change. Thanks for fixing that

Jobs are now running with the new image. Thanks to @Jdforrester-WMF for this change! https://gerrit.wikimedia.org/r/c/integration/config/+/1301363

I've deployed the changes and I can see them in the latest image:

$ docker run -it --rm --entrypoint bash docker-registry.wikimedia.org/repos/test-platform/catalyst/catalyst-ci-client:latest -c 'grep SEC deploy_env.py'
if 'SECONDARY_ID' in os.environ:
	ENV_NAME += f"-{os.getenv('SECONDARY_ID')}"

I can also run that code locally and see the SECONDARY_ID and 4-digit timestamp being added to the name & URL of the env:

INFO: Environment is ready and reachable at https://test-ci-1234567-12-7948.catalyst.wmcloud.org. Enjoy!

And additionally I can see that new CI jobs being triggered use the same image: docker-registry.wikimedia.org/repos/test-platform/catalyst/catalyst-ci-client:latest used in https://integration.wikimedia.org/ci/view/All/job/wikilambda-catalyst-end-to-end/3807/consoleFull

👋 sorry to reopen, but I found a small issue on the new trixie machines.

ZUUL_PATCHSET is not currently being passed along to the CI client. But I can add that as well. On it

I have more details now.

scap update-patch leaving a mess is the same problem you solved

scap prep ran successfully last night, that's one problem down.

In T428316#11993846, @taavi wrote:

It seems like the cleanest option for me would be to make the prep job abort in case /srv/patches is dirty (or has stashed changes, as those cause problems?) as both of those usually but not always indicate that the changes should just be committed?

The pretrain is indeed trying to copy patches from the latest version (1.47.0-wmf.5 in this case) to next every morning at 2:00 UTC, and Wednesday last week it started failing due to a permission error:

02:00:02 Copying <patch_path> to /srv/patches/next/<patch_path>
02:00:02 Finished scap prep next (duration: 00m 00s)
02:00:02 Unhandled error:
Traceback (most recent call last):
[...]
PermissionError: [Errno 1] Operation not permitted
02:00:02 prep failed: <PermissionError> [Errno 1] Operation not permitted (scap version: 4.267.0) (duration: 00m 00s)

In T418109#11989242, @Dzahn wrote:

Hi @jnuche and @hashar

I have upgraded the non-active releases hosts to trixie (T418299) and puppet worked without needing any changes.

But it is on Java 21 vs 17.

Do you see a good way we can test if releases-jenkins works before touching anything on the active server?

Cheers,

Daniel

Three wikilambda-catalyst-end-to-end jobs just ran successfully in parallel. Calling it

Both Patchdemo can CAPI are now using the image + deploying successfully