We could add a small timeout to retry the version check several times before erroing out.

A retry function could be added. I recommend we shelf this until we decide what is happening with magnum (which will possibly not be terribly applicable to our current toolforge setup). Ultimately the issue is a minor one, the problem it introduces is the need to manually uncordon a node before instructing the script to move on to the next node. Since the script still prompts to continue or not regardless, it isn't much more time spent doing an upgrade.

I'm going to close this. I believe I'm the only remaining person who can see this. Assuming the working theory is the reason behind the error, no one else is very likely to see this, and T326554 significantly reduces my exposure to this, which is a fairly minor thing to overcome (it involved running kubectl uncordon <node>)