Page MenuHomePhabricator

Puppet agent takes a long time to finish when adding IPv6 addresses
Open, MediumPublic

Description

When adding IPv6 to a host using interface::add_ip6_mapped, the first time Puppet runs, it takes a long time to finish:

...
Notice: /Stage[main]/Main/Node[__node_regexp__cloudvirt1020-2.eqiad.wmnet]/Interface::Add_ip6_mapped[main]/Augeas[eth0_v6_token]/returns: executed successfully
Notice: /Stage[main]/Main/Node[__node_regexp__cloudvirt1020-2.eqiad.wmnet]/Interface::Add_ip6_mapped[main]/Exec[eth0_v6_token]/returns: executed successfully

If you stop the agent and run it again (sudo puppet agent -tv), it finishes faster:

$ sudo puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for cloudvirt1022.eqiad.wmnet
Notice: /Stage[main]/Base::Environment/Tidy[/var/tmp/core]: Tidying 0 files
Info: Applying configuration version '1537991954'
Notice: Augeas[eth0_2620:0:861:118:10:64:20:41/64](provider=augeas): 
--- /etc/network/interfaces	2018-09-26 19:54:41.453432419 +0000
+++ /etc/network/interfaces.augnew	2018-09-26 19:59:28.461688243 +0000
@@ -19,6 +19,7 @@
 	dns-nameservers 208.80.154.254 208.80.153.254
 	dns-search eqiad.wmnet
    pre-up /sbin/ip token set ::10:64:20:41 dev eth0
+   up ip addr add 2620:0:861:118:10:64:20:41/64 dev eth0
 auto eth1.1105
 iface eth1.1105 inet manual
    up ip link set $IFACE up

Notice: /Stage[main]/Main/Node[__node_regexp__cloudvirt1020-2.eqiad.wmnet]/Interface::Add_ip6_mapped[main]/Interface::Ip[main]/Augeas[eth0_2620:0:861:118:10:64:20:41/64]/returns: executed successfully
Notice: Applied catalog in 12.42 seconds

IPv6 is added successfully.

Event Timeline

Did the host have an existing IPv6 address when the puppet run was started? If so puppet changing the IP mid-run is probably enough to interrupt the run in progress.

herron triaged this task as Medium priority.Sep 28 2018, 5:26 PM

I don't think it's actually puppet hanging. I think what is happening is that in the moment puppet adds the new IP to the interface your existing connecting gets interruped. If you ctrl+c and reconnect it should all run fine, right? edit: Well that's what i always thought it was. i am probably wrong.

I don't think it's actually puppet hanging. I think what is happening is that in the moment puppet adds the new IP to the interface your existing connecting gets interruped. If you ctrl+c and reconnect it should all run fine, right?

Good point. And from "If you stop the agent and run it again (sudo puppet agent -tv), it finishes faster:" it does sound like that's the case.

Naturally after asking I realized I could answer my own question in the puppetmaster logs. Indeed it looks like puppetmaster has seen two IPv6 addresses for cloudvirt1022.eqiad.wmnet. 2620:0:861:118:10:64:20:41 and
2620:0:861:118:d294:66ff:fe07:8f0a

Sadly I think this type of thing comes with the territory of doing "lower level" OS config with puppet. Maybe it's something that could move to OS install and also remove the manual step (puppet commit) to enable mapped IPv6? I don't know

Also see this epic ticket to solve the entire thing "once and for all" :) -> T102099

The host did have an IPv6 address in the same network but using the MAC address instead of the IPv4 address like we wanted.

I didn't have to reconnect SSH. The session was never interrupted.

If there are bigger plans for the IPv6 config code, I think we can close this one.