Page MenuHomePhabricator
Create Task
Maniphest T402377

[k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30
Closed, ResolvedPublic
Actions

Description

Upgrade procedure: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes

Refer to the link above for the detailed procedure, and update the checkboxes as you complete them.

If multiple people are working on the upgrade, you can copy the checklist to an Etherpad for easier collaborative editing.

Use this command from a toolforge control node to quickly generate a list of nodes:

for node in $(kubectl get nodes -o json | jq '.items[].metadata.name' -r); do echo "  - [] $node"; done

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
k8s.worker.upgrade_bastions: add cookbook to upgrade bastionscloud/wmcs-cookbooksmain+147 -0
k8s.worker.upgrade_ingresses: add cookbook to upgrade ingressescloud/wmcs-cookbooksmain+207 -0
k8s.worker.upgrade_workers: add a cookbook to upgrade a batchcloud/wmcs-cookbooksmain+235 -8
Customize query in gerrit

Related Objects
Search...

Event Timeline

dcaro created this task.Aug 20 2025, 9:29 AM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptAug 20 2025, 9:29 AM
dcaro updated the task description. (Show Details)Aug 20 2025, 9:29 AM
dcaro triaged this task as High priority.Aug 20 2025, 2:50 PM
dcaro changed the task status from Open to In Progress.Aug 21 2025, 3:29 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 23) board.
dcaro edited projects, added Toolforge (Toolforge iteration 24); removed Toolforge (Toolforge iteration 23).Aug 27 2025, 9:43 AM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 24) board.
taavi updated the task description. (Show Details)Aug 27 2025, 6:03 PM
Stashbot added a comment.Sep 1 2025, 8:06 AM

Mentioned in SAL (#wikimedia-cloud) [2025-09-01T08:06:49Z] <dcaro> starting upgrade to v1.30 (T402377)

dcaro updated the task description. (Show Details)Sep 1 2025, 8:08 AM
dcaro updated the task description. (Show Details)Sep 1 2025, 8:28 AM
Stashbot added a comment.Sep 1 2025, 8:32 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T08:32:10Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 8:32 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T08:32:30Z] <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.29.15 to 1.30.14 (T402377)

dcaro updated the task description. (Show Details)Sep 1 2025, 8:34 AM
Stashbot added a comment.Sep 1 2025, 8:35 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T08:35:56Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 8:39 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T08:39:09Z] <dcaro@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 8:42 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T08:42:57Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 8:43 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T08:43:00Z] <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 (T402377)

dcaro added a comment.Sep 1 2025, 8:51 AM

The upgrade of the first control node failed when uncordoning:

2025-09-01 08:39:01,749 dcaro 750609 [INFO cookbooks.wmcs.toolforge.k8s.worker.upgrade:224 in run] Uncordoning node
2025-09-01 08:39:01,751 dcaro 750609 [DEBUG spicerack.remote:750 in _execute] Executing commands [cumin.transports.Command('sudo -i kubectl get nodes --output=json --selector=\'kubernetes.io/hostname=toolsbeta-test-k8s-control-10\'', ok_codes=[0])] on 1 hosts: toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud
2025-09-01 08:39:01,751 dcaro 750609 [INFO cumin.transports.clustershell.ClusterShellWorker:78 in execute] Executing commands [cumin.transports.Command('sudo -i kubectl get nodes --output=json --selector=\'kubernetes.io/hostname=toolsbeta-test-k8s-control-10\'', ok_codes=[0])] on '1' hosts: toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud
2025-09-01 08:39:01,756 dcaro 750609 [DEBUG cumin.transports.clustershell.SyncEventHandler:590 in ev_pickup] node=toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud, command='sudo -i kubectl get nodes --output=json --selector='kubernetes.io/hostname=toolsbeta-test-k8s-control-10''
2025-09-01 08:39:09,705 dcaro 750609 [DEBUG cumin.transports.clustershell.SyncEventHandler:783 in ev_hup] node=toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud, rc=1, command='sudo -i kubectl get nodes --output=json --selector='kubernetes.io/hostname=toolsbeta-test-k8s-control-10''
2025-09-01 08:39:09,706 dcaro 750609 [INFO cumin.transports.clustershell.SyncEventHandler:853 in ev_timer] Completed command 'sudo -i kubectl get nodes --output=json --selector='kubernetes.io/hostname=toolsbeta-test-k8s-control-10''
2025-09-01 08:39:09,707 dcaro 750609 [DEBUG wmcs_libs.common:828 in _wrapped_run] Cleaning up recorder.
2025-09-01 08:39:09,707 dcaro 750609 [ERROR spicerack._menu:292 in _run] Exception raised while executing cookbook wmcs.toolforge.k8s.worker.upgrade:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 265, in _run
    raw_ret = runner.run()
  File "/srv/deployment/wmcs-cookbooks/wmcs_libs/common.py", line 825, in _wrapped_run
    return object.__getattribute__(self, __name)(*args, **kwargs)
  File "/srv/deployment/wmcs-cookbooks/cookbooks/wmcs/toolforge/k8s/worker/upgrade.py", line 225, in run
    kubectl.uncordon_node(node_hostname=self.hostname)
  File "/srv/deployment/wmcs-cookbooks/wmcs_libs/k8s/kubernetes.py", line 304, in uncordon_node
    current_nodes = self.get_nodes(selector=f"kubernetes.io/hostname={node_hostname}")
  File "/srv/deployment/wmcs-cookbooks/wmcs_libs/k8s/kubernetes.py", line 189, in get_nodes
    output = run_one_as_dict(
  File "/srv/deployment/wmcs-cookbooks/wmcs_libs/common.py", line 324, in run_one_as_dict
    result = run_one_formatted(
  File "/srv/deployment/wmcs-cookbooks/wmcs_libs/common.py", line 353, in run_one_formatted
    raw_result = run_one_raw(
  File "/srv/deployment/wmcs-cookbooks/wmcs_libs/common.py", line 278, in run_one_raw
    return run_one_raw_needed_to_be_able_to_mock(
  File "/srv/deployment/wmcs-cookbooks/wmcs_libs/common.py", line 216, in run_one_raw_needed_to_be_able_to_mock
    result = next(node.run_sync(command, **run_sync_params))
  File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 556, in run_sync
    return self._execute(
  File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 763, in _execute
    raise RemoteExecutionError(ret, "Cumin execution failed", worker.get_results())
spicerack.remote.RemoteExecutionError: Cumin execution failed (exit_code=2)

Not clear why, a manual run of the get nodes on that control nodes worked after, and a subsequent run of the coobkook also worked. So just manually uncordoned the node and checked the logs (what was left from the cookbook).

dcaro added a comment.Sep 1 2025, 8:55 AM

The apiserver was unhealthy for a second there while starting on the new version:

│ Name:             kube-apiserver-toolsbeta-test-k8s-control-10.184ca232bac52001                                                                                                                                                                                                                                                                     │
│ Namespace:        kube-system                                                                                                                                                                                                                                                                                                                       │
│ Labels:           <none>                                                                                                                                                                                                                                                                                                                            │
│ Annotations:      <none>                                                                                                                                                                                                                                                                                                                            │
│ API Version:      v1                                                                                                                                                                                                                                                                                                                                │
│ Count:            9                                                                                                                                                                                                                                                                                                                                 │
│ Event Time:       <nil>                                                                                                                                                                                                                                                                                                                             │
│ First Timestamp:  2025-06-26T15:44:53Z                                                                                                                                                                                                                                                                                                              │
│ Involved Object:                                                                                                                                                                                                                                                                                                                                    │
│   API Version:   v1                                                                                                                                                                                                                                                                                                                                 │
│   Field Path:    spec.containers{kube-apiserver}                                                                                                                                                                                                                                                                                                    │
│   Kind:          Pod                                                                                                                                                                                                                                                                                                                                │
│   Name:          kube-apiserver-toolsbeta-test-k8s-control-10                                                                                                                                                                                                                                                                                       │
│   Namespace:     kube-system                                                                                                                                                                                                                                                                                                                        │
│   UID:           5eb1cdfec67127b130dc7be70fc9c235                                                                                                                                                                                                                                                                                                   │
│ Kind:            Event                                                                                                                                                                                                                                                                                                                              │
│ Last Timestamp:  2025-09-01T08:37:40Z                                                                                                                                                                                                                                                                                                               │
│ Message:         Readiness probe failed: Get "https://172.16.5.70:6443/readyz": dial tcp 172.16.5.70:6443: connect: connection refused                                                                                                                                                                                                              │
│ Metadata:                                                                                                                                                                                                                                                                                                                                           │
│   Creation Timestamp:  2025-09-01T08:37:51Z                                                                                                                                                                                                                                                                                                         │
│   Resource Version:    882298934                                                                                                                                                                                                                                                                                                                    │
│   UID:                 890bacc0-57d0-4604-8f20-c668e8555920                                                                                                                                                                                                                                                                                         │
│ Reason:                Unhealthy                                                                                                                                                                                                                                                                                                                    │
│ Reporting Component:   kubelet                                                                                                                                                                                                                                                                                                                      │
│ Reporting Instance:    toolsbeta-test-k8s-control-10                                                                                                                                                                                                                                                                                                │
│ Source:                                                                                                                                                                                                                                                                                                                                             │
│   Component:  kubelet                                                                                                                                                                                                                                                                                                                               │
│   Host:       toolsbeta-test-k8s-control-10                                                                                                                                                                                                                                                                                                         │
│ Type:         Warning                                                                                                                                                                                                                                                                                                                               │
│ Events:       <none>

Will keep an eye for the rest of the control nodes, we might want to change the cookbook to give the api-server some time to finish coming up before uncordoning.

Stashbot added a comment.Sep 1 2025, 8:56 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T08:56:23Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 9:07 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:07:30Z] <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.29.15 to 1.30.14 (T402377)

dcaro updated the task description. (Show Details)Sep 1 2025, 9:08 AM
Stashbot added a comment.Sep 1 2025, 9:08 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:08:18Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 9:15 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:15:27Z] <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.29.15 to 1.30.14 (T402377)

dcaro updated the task description. (Show Details)Sep 1 2025, 9:20 AM
dcaro updated the task description. (Show Details)Sep 1 2025, 9:26 AM
dcaro mentioned this in T362869: [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30).Sep 1 2025, 9:28 AM
Stashbot added a comment.Sep 1 2025, 9:30 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:30:11Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 9:31 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:31:10Z] <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 9:34 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:34:06Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.29.15 to 1.30.14 (T402377)

dcaro updated the task description. (Show Details)Sep 1 2025, 9:34 AM
Stashbot added a comment.Sep 1 2025, 9:35 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:35:04Z] <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 9:37 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:37:57Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 9:38 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:38:13Z] <dcaro@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 (T402377)

dcaro updated the task description. (Show Details)Sep 1 2025, 9:38 AM

One of the workers failed to upgrade because puppet was already running:

----- OUTPUT of 'sudo -i run-puppet-agent' -----                                                                                                                                                                                                                                                                                                       
Notice: Run of Puppet configuration client already in progress; skipping  (/var/lib/puppet/state/agent_catalog_run.lock exists)
Stashbot added a comment.Sep 1 2025, 9:39 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:39:38Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 9:40 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:40:22Z] <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 9:41 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:41:11Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 9:42 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:42:09Z] <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 9:44 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:44:42Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 9:45 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:45:45Z] <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 9:46 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:46:15Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 9:47 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:47:10Z] <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.29.15 to 1.30.14 (T402377)

Stashbot added a comment.Sep 1 2025, 9:47 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:47:32Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 (T402377)

dcaro updated the task description. (Show Details)Sep 1 2025, 9:48 AM
Stashbot added a comment.Sep 1 2025, 9:48 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T09:48:32Z] <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 (T402377)

gerritbot added a comment.Sep 1 2025, 12:40 PM

Change #1183657 had a related patch set uploaded (by David Caro; author: David Caro):

[cloud/wmcs-cookbooks@main] k8s.worker.upgrade_workers: add a cookbook to upgrade a batch

https://gerrit.wikimedia.org/r/1183657

gerritbot added a project: Patch-For-Review.Sep 1 2025, 12:40 PM
gerritbot added a comment.Sep 1 2025, 1:34 PM

Change #1183682 had a related patch set uploaded (by David Caro; author: David Caro):

[cloud/wmcs-cookbooks@main] k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses

https://gerrit.wikimedia.org/r/1183682

Stashbot added a comment.Sep 1 2025, 1:41 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T13:41:19Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 (T402377)

Stashbot added a comment.Sep 1 2025, 1:41 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T13:41:51Z] <dcaro@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 (T402377)

Stashbot added a comment.Sep 1 2025, 1:43 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T13:43:33Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 (T402377)

Stashbot added a comment.Sep 1 2025, 1:47 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T13:47:54Z] <dcaro@cloudcumin1001> END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=97) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 (T402377)

Stashbot added a comment.Sep 1 2025, 1:48 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T13:48:44Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 (T402377)

Stashbot added a comment.Sep 1 2025, 1:49 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T13:49:35Z] <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 (T402377)

Stashbot added a comment.Sep 1 2025, 1:51 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T13:51:51Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 (T402377)

Stashbot added a comment.Sep 1 2025, 1:52 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T13:52:00Z] <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 (T402377)

dcaro updated the task description. (Show Details)Sep 1 2025, 1:53 PM
gerritbot added a comment.Sep 1 2025, 2:12 PM

Change #1183697 had a related patch set uploaded (by David Caro; author: David Caro):

[cloud/wmcs-cookbooks@main] k8s.worker.upgrade_bastions: add cookbook to upgrade bastions

https://gerrit.wikimedia.org/r/1183697

Stashbot added a comment.Sep 1 2025, 2:16 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T14:16:19Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T402377)

Stashbot added a comment.Sep 1 2025, 2:16 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T14:16:25Z] <dcaro@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=99) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T402377)

Stashbot added a comment.Sep 1 2025, 2:16 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T14:16:44Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T402377)

Stashbot added a comment.Sep 1 2025, 2:16 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T14:16:54Z] <dcaro@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=99) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T402377)

Stashbot added a comment.Sep 1 2025, 2:21 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T14:21:39Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T402377)

Stashbot added a comment.Sep 1 2025, 2:21 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T14:21:43Z] <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T402377)

Stashbot added a comment.Sep 1 2025, 2:22 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T14:22:42Z] <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T402377)

Stashbot added a comment.Sep 1 2025, 2:22 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-01T14:22:49Z] <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T402377)

dcaro updated the task description. (Show Details)Sep 1 2025, 2:23 PM
dcaro updated the task description. (Show Details)

Used the cookbooks to upgrade too, will update the docs once they are reviewed

dcaro closed this task as Resolved.Sep 1 2025, 2:24 PM
dcaro moved this task from In Progress to Done on the Toolforge (Toolforge iteration 24) board.
gerritbot added a comment.Sep 8 2025, 11:20 AM

Change #1183657 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] k8s.worker.upgrade_workers: add a cookbook to upgrade a batch

https://gerrit.wikimedia.org/r/1183657

gerritbot added a comment.Sep 8 2025, 11:20 AM

Change #1183682 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses

https://gerrit.wikimedia.org/r/1183682

gerritbot added a comment.Sep 8 2025, 11:20 AM

Change #1183697 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] k8s.worker.upgrade_bastions: add cookbook to upgrade bastions

https://gerrit.wikimedia.org/r/1183697

jenkins-bot mentioned this in rCCKB6682e38d5d8c: k8s.worker.upgrade_workers: add a cookbook to upgrade a batch.Sep 8 2025, 11:21 AM
jenkins-bot mentioned this in rCCKB65f977c39d33: k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses.
jenkins-bot mentioned this in rCCKBe54440ce8ea3: k8s.worker.upgrade_bastions: add cookbook to upgrade bastions.
CodeReviewBot added a comment.Sep 15 2025, 12:06 PM

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/964

run_functional_tests: use the python3.13 image for venv creation

CodeReviewBot added a comment.Sep 15 2025, 12:21 PM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/964

run_functional_tests: use the python3.13 image for venv creation

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits