Page MenuHomePhabricator

contint1001 and contint2001 should allow port 9418 from their ipv6 addresses
Closed, ResolvedPublic

Description

TL;DR: Our zuul merger servers (contint1001 and contint2001) need firewall rules to allow port 9418 connections from their ipv6 addresses.

Today I noticed a build of our new service-jenkinsjob job taking an extremely long time to fetch refs from contint2001 (seemed like a couple minutes per git fetch comment). Since it was running on contint1001 this seemed odd, as it should have a speedy network route within the same subnet.

After a bunch of digging, it turns out git was resolving two addresses, one ipv6 and one ipv4, trying the ipv6 address first and getting a timeout, then falling back to the ipv4 address. I verified this was the case using strace (thanks @thcipriani !).

This problem can be worked around using the --ipv4 git argument, but that's not a great solution for CI/Jenkins scripts. I think the proper solution is to modify the ferm rules to allow port 9418 connections from the ipv6 addresses as well ipv4.

Event Timeline

dduvall created this task.Feb 14 2019, 11:20 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 14 2019, 11:20 PM
dduvall claimed this task.Feb 14 2019, 11:31 PM
dduvall triaged this task as Normal priority.
dduvall moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.
dduvall added a subscriber: hashar.

Change 490790 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[operations/puppet@production] ci: Permit git traffic between zuul mergers ipv6

https://gerrit.wikimedia.org/r/490790

Change 490791 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] contint: allow port 9418 on IPv6 for zuul-mergers

https://gerrit.wikimedia.org/r/490791

Change 490791 abandoned by Dzahn:
contint: allow port 9418 on IPv6 for zuul-mergers

Reason:
duplicate of https://gerrit.wikimedia.org/r/490790

https://gerrit.wikimedia.org/r/490791

Change 490790 merged by Dzahn:
[operations/puppet@production] ci: Permit git traffic between zuul mergers over ipv6

https://gerrit.wikimedia.org/r/490790

dduvall closed this task as Resolved.Feb 15 2019, 12:24 AM

Deployed and tested.

dduvall@contint1001:~$ telnet -6 contint2001.wikimedia.org 9418
Trying 2620:0:860:1:208:80:153:15...
Connected to contint2001.wikimedia.org.
...
dduvall@contint2001:~$ telnet -6 contint1001.wikimedia.org 9418
Trying 2620:0:861:1:208:80:154:17...
Connected to contint1001.wikimedia.org.
...