@jnuche discovered this in a Magnum produced cluster:
[root@zuul-k8s-v128-edzervntmdrq-master-0 kubernetes]# kubectl -n kube-system get events LAST SEEN TYPE REASON OBJECT MESSAGE 3m37s Warning NetworkNotReady pod/coredns-54d6b4866b-cfkl2 network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized 3m38s Warning NetworkNotReady pod/coredns-54d6b4866b-wg54k network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized 3m34s Warning NetworkNotReady pod/csi-cinder-controllerplugin-6f7bfcc65d-6vxsc network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized 4m19s Warning FailedScheduling pod/dashboard-metrics-scraper-6f455dfbc-z2lsl 0/4 nodes are available: 4 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.. 5m49s Warning FailedScheduling pod/kube-dns-autoscaler-679fb56494-pkt8s 0/4 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 3 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.. 40m Normal Pulling pod/kube-flannel-ds-b92pw Pulling image "quay.io/coreos/flannel-cni:v0.3.0" 9s Normal BackOff pod/kube-flannel-ds-b92pw Back-off pulling image "quay.io/coreos/flannel-cni:v0.3.0" 3m29s Normal BackOff pod/kube-flannel-ds-l4jh6 Back-off pulling image "quay.io/coreos/flannel-cni:v0.3.0" 4m58s Normal BackOff pod/kube-flannel-ds-rxww6 Back-off pulling image "quay.io/coreos/flannel-cni:v0.3.0" 10s Normal BackOff pod/kube-flannel-ds-zfbfr Back-off pulling image "quay.io/coreos/flannel-cni:v0.3.0" 5m19s Warning FailedScheduling pod/kubernetes-dashboard-5685578847-km7rh 0/4 nodes are available: 4 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling..
The image pull failure is because the upstream "quay.io/coreos/flannel-cni" registry is gone.
We can try switching to Calico as the cluster network driver or we can do more research upstream to find out what the replacement for the https://github.com/coreos/flannel-cni container is. (The latest upstream code is still referencing it, so this might be a long pull.)