@bking my understanding is that the host changes IP address so IPVS/Pybal need to be fixed to list the new value. What is the use case of not using depool before a reimage?

@elukey As I stated above, I always believed the cookbooks depooled the hosts automatically, and I only became aware of the flag to depool yesterday. I admit it's a bad assumption on my part, but based on my discussion in IRC yesterday I know I'm not the only one.

Since the reimage cookbook invites you to use --move-vlan, I would say that the cookbook should explain the risks associated with using that flag as well. I accept full responsibility for not depooling the host, but if there's a way to avoid setting off alerts that other teams have to respond to, I think we should try it.

@elukey I was thinking about this in the shower this morning and I may have finally grasped the meaning behind your comment "What is the use case of not using depool before a reimage?" Honestly, there does not really seem to be a use case for not depooling. Would you accept a patch to the reimage cookbook that defaults to depooling the host?

Okay, I think I figured out what's going on here.

Setting pooled=inactive is the only way to avoid the pybal/LVS mismatch, which otherwise happens deterministically.

As such with respect to this ticket, the fix is the following:

When --move-vlan is supplied for a host with Conftool objects, it must enable --conftool and require --conftool-value=inactive before changing the address

(because there's an edge case where someone could supply --conftool --conftool-value=no - I don't know if anyone has ever actually done that, but we should guard against it all the same)

Here's a table of live observations from T430880, which shows that the issue only manifests if we don't set pooled=inactive (i.e. it *does* manifest on pooled=no):

Alright, on to tracing the actual mechanism:

(1) DNS gets resolved inside initialize()
(2) And the initialize() itself is only called in the branch where a new host is detected
(3) And we only hit that branch when the host was absent from the config pybal assembled:
pooled=inactive is what makes it absent, because it decodes to the same sentinel as a deleted key, and the caller then deletes those keys from the config
pooled=no leaves the host present but disabled, so we take the merge branch instead and the object keeps the address it resolved when it was first built