Page MenuHomePhabricator

contint1001 and contint2001 should allow port 9418 from their ipv6 addresses
Closed, ResolvedPublic

Description

TL;DR: Our zuul merger servers (contint1001 and contint2001) need firewall rules to allow port 9418 connections from their ipv6 addresses.

Today I noticed a build of our new service-jenkinsjob job taking an extremely long time to fetch refs from contint2001 (seemed like a couple minutes per git fetch comment). Since it was running on contint1001 this seemed odd, as it should have a speedy network route within the same subnet.

After a bunch of digging, it turns out git was resolving two addresses, one ipv6 and one ipv4, trying the ipv6 address first and getting a timeout, then falling back to the ipv4 address. I verified this was the case using strace (thanks @thcipriani !).

This problem can be worked around using the --ipv4 git argument, but that's not a great solution for CI/Jenkins scripts. I think the proper solution is to modify the ferm rules to allow port 9418 connections from the ipv6 addresses as well ipv4.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 14 2019, 11:20 PM
dduvall triaged this task as Normal priority.
dduvall added a subscriber: hashar.

Change 490790 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[operations/puppet@production] ci: Permit git traffic between zuul mergers ipv6

https://gerrit.wikimedia.org/r/490790

Change 490791 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] contint: allow port 9418 on IPv6 for zuul-mergers

https://gerrit.wikimedia.org/r/490791

Change 490791 abandoned by Dzahn:
contint: allow port 9418 on IPv6 for zuul-mergers

Reason:
duplicate of https://gerrit.wikimedia.org/r/490790

https://gerrit.wikimedia.org/r/490791

Change 490790 merged by Dzahn:
[operations/puppet@production] ci: Permit git traffic between zuul mergers over ipv6

https://gerrit.wikimedia.org/r/490790

dduvall closed this task as Resolved.Feb 15 2019, 12:24 AM

Deployed and tested.

dduvall@contint1001:~$ telnet -6 contint2001.wikimedia.org 9418
Trying 2620:0:860:1:208:80:153:15...
Connected to contint2001.wikimedia.org.
...
dduvall@contint2001:~$ telnet -6 contint1001.wikimedia.org 9418
Trying 2620:0:861:1:208:80:154:17...
Connected to contint1001.wikimedia.org.
...