Page MenuHomePhabricator

pybal health checks are ipv4 even for ipv6 vips
Closed, ResolvedPublic

Description

Author: lcarr

Description:

so i found this out as i put in an incorrect ipv6 for ssl1005 and ssl1006 - however they health checked fine... though when traffic was sent to them, it was sent ot the incorrect ipv6 address. it appears that the health checks are all ipv4, even when it's an ipv6 vip. some sort of knowledge or ability to specify ipv4 versus ipv6 health checks would be preferred.
please let me know if this is better suited for bugzilla

Details

Reference
rt5500
Related Gerrit Patches:
operations/debs/pybal : 1.13DnsQuery monitor: query over IPv4
operations/debs/pybal : masterDnsQuery monitor: query over IPv4
operations/debs/pybal : 1.13Add IPv6 support to all monitors, improve IdleConnection logging
operations/debs/pybal : masterAdd IPv6 support to all monitors, improve IdleConnection logging
operations/debs/pybal : 1.14New release: 1.14

Event Timeline

rtimport raised the priority of this task from to Medium.Dec 18 2014, 1:39 AM
rtimport added a project: ops-core.
rtimport set Reference to rt5500.
Dzahn changed the visibility from "WMF-NDA (Project)" to "Public (No Login Required)".Feb 25 2015, 12:22 AM
Dzahn changed the edit policy from "WMF-NDA (Project)" to "All Users".
Dzahn set Security to None.
mark closed this task as Resolved.Feb 18 2016, 2:43 PM
mark claimed this task.

And according to @BBlack this is already deployed, so resolving. :)

BBlack reopened this task as Open.Feb 18 2016, 2:46 PM

No, it's not :P

BBlack moved this task from Triage to LoadBalancer on the Traffic board.Sep 30 2016, 2:03 PM

Change 345340 had a related patch set uploaded (by Ema):
[operations/debs/pybal@1.14] New release: 1.14

https://gerrit.wikimedia.org/r/345340

Change 345340 abandoned by Ema:
New release: 1.14

Reason:
Instead of cherry-picking the patch, we'll base 1.14 off master after merging the 1.13 changes into it.

https://gerrit.wikimedia.org/r/345340

ema moved this task from Backlog to In Progress on the Pybal board.Mar 30 2017, 6:08 AM

Change 363188 had a related patch set uploaded (by Ema; owner: Ema):
[operations/debs/pybal@master] Add IPv6 support to all monitors, improve IdleConnection logging

https://gerrit.wikimedia.org/r/363188

Change 363188 merged by Ema:
[operations/debs/pybal@master] Add IPv6 support to all monitors, improve IdleConnection logging

https://gerrit.wikimedia.org/r/363188

Change 363193 had a related patch set uploaded (by Ema; owner: Ema):
[operations/debs/pybal@1.13] Add IPv6 support to all monitors, improve IdleConnection logging

https://gerrit.wikimedia.org/r/363193

Change 363193 merged by Ema:
[operations/debs/pybal@1.13] Add IPv6 support to all monitors, improve IdleConnection logging

https://gerrit.wikimedia.org/r/363193

Mentioned in SAL (#wikimedia-operations) [2017-07-04T14:57:00Z] <ema> pybal 1.13.7 uploaded to apt.w.o, testing it on pybal-test2001 T82747 T154759

Mentioned in SAL (#wikimedia-operations) [2017-07-12T15:07:01Z] <ema> lvs2006: upgrade pybal to 1.13.7 T82747 T154759

Mentioned in SAL (#wikimedia-operations) [2017-07-12T16:06:20Z] <ema> lvs1006, lvs1010: upgrade pybal to 1.13.7 T82747 T154759

Mentioned in SAL (#wikimedia-operations) [2017-07-12T16:14:45Z] <ema> downgrade pybal to 1.13.6 on lvs1010 T82747 T154759 (1.13.7 throwing exceptions)

ema added a subscriber: ema.Jul 12 2017, 4:50 PM

Almost there! I had to downgrade pybal to 1.13.6 on lvs1010 because of the following exception:

File "/usr/lib/python2.7/dist-packages/twisted/names/dns.py", line 2776, in _query
  writeMessage(m)
File "/usr/lib/python2.7/dist-packages/twisted/names/dns.py", line 2896, in writeMessage
  self.writeMessage(m, address)
File "/usr/lib/python2.7/dist-packages/twisted/names/dns.py", line 2824, in writeMessage
  self.transport.write(message.toStr(), address)
File "/usr/lib/python2.7/dist-packages/twisted/internet/udp.py", line 295, in write
  addr[0], "IPv4 port write() called with IPv6 address")

The problem can be reproduced with:

import sys

from twisted.names import client
from twisted.internet import task

def printit(result):
    print result

def main(reactor, hostname):
    #ip = "208.80.154.50"
    ip = "2620:0:861:1:208:80:154:50"

    resolver = client.createResolver([(ip, 53)])
    deferred = resolver.lookupIPV6Address(hostname, timeout=[5])

    deferred.addCallback(printit)
    return deferred

task.react(main, sys.argv[1:])

Using 208.80.154.5 instead of 2620:0:861:1:208:80:154:50 the script works.

That's a twisted bug mentioned here https://matrix.org/jira/browse/SYN-254.

Change 364948 had a related patch set uploaded (by Ema; owner: Ema):
[operations/debs/pybal@master] DnsQuery monitor: use IPv4

https://gerrit.wikimedia.org/r/364948

Change 364948 merged by Ema:
[operations/debs/pybal@master] DnsQuery monitor: query over IPv4

https://gerrit.wikimedia.org/r/364948

Change 364963 had a related patch set uploaded (by Ema; owner: Ema):
[operations/debs/pybal@1.13] DnsQuery monitor: query over IPv4

https://gerrit.wikimedia.org/r/364963

Change 364963 merged by Ema:
[operations/debs/pybal@1.13] DnsQuery monitor: query over IPv4

https://gerrit.wikimedia.org/r/364963

Mentioned in SAL (#wikimedia-operations) [2017-07-13T09:43:34Z] <ema> lvs1010: upgrade pybal to 1.13.9 T82747 T154759

Mentioned in SAL (#wikimedia-operations) [2017-07-13T10:04:56Z] <ema> lvs[12]006: upgrade pybal to 1.13.9 T82747 T154759

Mentioned in SAL (#wikimedia-operations) [2017-07-17T08:06:32Z] <ema> lvs2003: upgrade pybal to 1.13.9 T82747 T154759

Mentioned in SAL (#wikimedia-operations) [2017-07-17T08:17:15Z] <ema> lvs100[39]: upgrade pybal to 1.13.9 T82747 T154759

Mentioned in SAL (#wikimedia-operations) [2017-07-18T09:13:06Z] <ema> lvs300[34] upgrade pybal to 1.13.9 T82747

Mentioned in SAL (#wikimedia-operations) [2017-07-18T09:15:51Z] <ema> lvs300[12] upgrade pybal to 1.13.9 T82747

Mentioned in SAL (#wikimedia-operations) [2017-07-18T10:32:34Z] <ema> lvs100[45] upgrade pybal to 1.13.9 T82747 T154759

Mentioned in SAL (#wikimedia-operations) [2017-07-18T10:43:57Z] <ema> lvs100[12] upgrade pybal to 1.13.9 T82747 T154759

Mentioned in SAL (#wikimedia-operations) [2017-07-18T10:52:23Z] <ema> lvs400[34] upgrade pybal to 1.13.9 T82747 T154759

Mentioned in SAL (#wikimedia-operations) [2017-07-18T10:54:16Z] <ema> lvs400[12] upgrade pybal to 1.13.9 T82747 T154759

Mentioned in SAL (#wikimedia-operations) [2017-07-18T10:59:15Z] <ema> lvs200[45] upgrade pybal to 1.13.9 T82747 T154759

Mentioned in SAL (#wikimedia-operations) [2017-07-18T11:00:36Z] <ema> lvs200[12] upgrade pybal to 1.13.9 T82747 T154759

ema closed this task as Resolved.Jul 18 2017, 11:07 AM

All LVSs upgraded to 1.13.9, which fixes this bug.