Page MenuHomePhabricator

Upgrade paws-dev to Kubernetes 1.22 in codfw1dev
Closed, ResolvedPublic

Description

Test the magnum upgrade in codfw1dev

openstack coe cluster upgrade paws-dev paws-dev-k8s22

Event Timeline

The upgrade command initially fails, giving an UPGRADE_FAILED status. At this point the control node is upgrade to k8s 1.22.11 however all of the worker nodes remain on 1.21.8
Running:

openstack coe cluster resize paws-dev 3

Will resume the upgrade.
The worker nodes will then be cordoned and upgraded one at a time. However they return, and briefly are set to "Ready" as reported by kubectl get nodes but quickly switch to NotReady describe shows:

Normal  NodeNotReady             2m37s (x2 over 6m2s)   node-controller  Node paws-dev-rkc62ygu4fei-node-2 status is now: NodeNotReady

Which, when all the worker nodes reach this state, starves the cluster of resource and nothing will schedule.

Running:

openstack coe cluster resize paws-dev 0 # wait a few minutes
openstack coe cluster resize paws-dev 3

Brings the worker nodes back and upgraded, however the hub container in paws does not seem to reach a ready state. This may be a paws thing.

This does not appear to work with a direct upgrade. We'll use a blue/green deploy as noted in

https://wikitech.wikimedia.org/wiki/PAWS/Admin#Deployment