TL;DR: Our zuul merger servers (contint1001 and contint2001) need firewall rules to allow port 9418 connections from their ipv6 addresses.
Today I noticed a build of our new service-jenkinsjob job taking an extremely long time to fetch refs from contint2001 (seemed like a couple minutes per git fetch comment). Since it was running on contint1001 this seemed odd, as it should have a speedy network route within the same subnet.
After a bunch of digging, it turns out git was resolving two addresses, one ipv6 and one ipv4, trying the ipv6 address first and getting a timeout, then falling back to the ipv4 address. I verified this was the case using strace (thanks @thcipriani !).
This problem can be worked around using the --ipv4 git argument, but that's not a great solution for CI/Jenkins scripts. I think the proper solution is to modify the ferm rules to allow port 9418 connections from the ipv6 addresses as well ipv4.