Test the magnum upgrade in codfw1dev
openstack coe cluster upgrade paws-dev paws-dev-k8s22
Test the magnum upgrade in codfw1dev
openstack coe cluster upgrade paws-dev paws-dev-k8s22
The upgrade command initially fails, giving an UPGRADE_FAILED status. At this point the control node is upgrade to k8s 1.22.11 however all of the worker nodes remain on 1.21.8
Running:
openstack coe cluster resize paws-dev 3
Will resume the upgrade.
The worker nodes will then be cordoned and upgraded one at a time. However they return, and briefly are set to "Ready" as reported by kubectl get nodes but quickly switch to NotReady describe shows:
Normal NodeNotReady 2m37s (x2 over 6m2s) node-controller Node paws-dev-rkc62ygu4fei-node-2 status is now: NodeNotReady
Which, when all the worker nodes reach this state, starves the cluster of resource and nothing will schedule.
Running:
openstack coe cluster resize paws-dev 0 # wait a few minutes openstack coe cluster resize paws-dev 3
Brings the worker nodes back and upgraded, however the hub container in paws does not seem to reach a ready state. This may be a paws thing.
This does not appear to work with a direct upgrade. We'll use a blue/green deploy as noted in