Page MenuHomePhabricator

Upgrade paws-dev to Kubernetes 1.22 in codfw1dev
Closed, ResolvedPublic


Test the magnum upgrade in codfw1dev

openstack coe cluster upgrade paws-dev paws-dev-k8s22

Event Timeline

The upgrade command initially fails, giving an UPGRADE_FAILED status. At this point the control node is upgrade to k8s 1.22.11 however all of the worker nodes remain on 1.21.8

openstack coe cluster resize paws-dev 3

Will resume the upgrade.
The worker nodes will then be cordoned and upgraded one at a time. However they return, and briefly are set to "Ready" as reported by kubectl get nodes but quickly switch to NotReady describe shows:

Normal  NodeNotReady             2m37s (x2 over 6m2s)   node-controller  Node paws-dev-rkc62ygu4fei-node-2 status is now: NodeNotReady

Which, when all the worker nodes reach this state, starves the cluster of resource and nothing will schedule.


openstack coe cluster resize paws-dev 0 # wait a few minutes
openstack coe cluster resize paws-dev 3

Brings the worker nodes back and upgraded, however the hub container in paws does not seem to reach a ready state. This may be a paws thing.

This does not appear to work with a direct upgrade. We'll use a blue/green deploy as noted in