Page MenuHomePhabricator

Python 3's eventlet.green getaddrinfo timeout in Bullseye
Open, MediumPublic

Description

Found today while debugging Swift on Bullseye in Cloud VPS. It looks like getaddrinfo from eventlet times out (e.g. on ms-be-01.swift.eqiad1.wikimedia.cloud)

$ python3 -c 'from eventlet.green import socket ; print(socket.getaddrinfo("ms-fe-02.swift.eqiad1.wikimedia.cloud", 11211, socket.AF_UNSPEC, socket.SOCK_STREAM))'                                                                    
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 435, in resolve
    return _proxy.query(name, rdtype, raise_on_no_answer=raises,
  File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 391, in query
    return end()
  File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 370, in end
    raise result[1]
  File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 351, in step
    a = fun(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1089, in query
    return self.resolve(qname, rdtype, rdclass, tcp, source,
  File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve
    timeout = self._compute_timeout(start, lifetime)
  File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout
    raise Timeout(timeout=duration)
dns.exception.Timeout: The DNS operation timed out after 5.107318878173828 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 528, in getaddrinfo
    qname, addrs = _getaddrinfo_lookup(host, family, flags)
  File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 501, in _getaddrinfo_lookup
    raise err
  File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 490, in _getaddrinfo_lookup
    answer = resolve(host, qfamily, False, use_network=use_network)
  File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 443, in resolve
    raise EAI_EAGAIN_ERROR
  File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 490, in _getaddrinfo_lookup
    answer = resolve(host, qfamily, False, use_network=use_network)
  File "/usr/lib/python3/dist-packages/eventlet/support/greendns.py", line 443, in resolve
    raise EAI_EAGAIN_ERROR
socket.gaierror: [Errno -3] Lookup timed out

Whereas non-eventlet works as expected

$ python3 -c 'import socket ; print(socket.getaddrinfo("ms-fe-02.swift.eqiad1.wikimedia.cloud", 11211, socket.AF_UNSPEC, socket.SOCK_STREAM))'                                                                                        
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('172.16.3.119', 11211))]

Event Timeline

Relevant Debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=971530 apparently and incompatibility with dnspython >= 2.0 and eventlet; eventlet upstream issue https://github.com/eventlet/eventlet/issues/619

+SRE for visibility as this will be true in production too

fgiunchedi renamed this task from Python 3's eventlet.green getaddrinfo timeout in Cloud VPS + Bullseye to Python 3's eventlet.green getaddrinfo timeout in Bullseye.Wed, May 26, 2:53 PM
Marostegui triaged this task as Medium priority.Wed, May 26, 3:40 PM