Page MenuHomePhabricator

/usr/sbin/ssh-key-ldap-lookup misconfigured in codfw1dev
Open, Needs TriagePublic

Description

$ ssh cloudvirt2001-dev.codfw.wmnet
$ virsh console --devname serial1 843a9e72-d380-4655-bada-b813eb1847a5
Connected to domain i-00000392
Escape character is ^]

$ /usr/sbin/ssh-key-ldap-lookup bd808-labtest
Traceback (most recent call last):
  File "/usr/sbin/ssh-key-ldap-lookup", line 138, in <module>
    main()
  File "/usr/sbin/ssh-key-ldap-lookup", line 119, in main
    config['password'])
  File "/usr/sbin/ssh-key-ldap-lookup", line 86, in robust_connect
    return connect(servers[position], user, password)
  File "/usr/sbin/ssh-key-ldap-lookup", line 43, in connect
    conn.simple_bind_s(username, password)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 444, in simple_bind_s
    resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all=1,timeout=self.timeout)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 749, in result3
    resp_ctrl_classes=resp_ctrl_classes
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 756, in result4
    ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 329, in _ldap_call
    reraise(exc_type, exc_value, exc_traceback)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 313, in _ldap_call
    result = func(*args,**kwargs)
ldap.INVALID_CREDENTIALS: {'desc': u'Invalid credentials'}

/etc/ldap.yaml lists cloudservices2002-dev.wikimedia.org as the ldap server, but /etc/ldap.conf uses ldap-ro.eqiad.wikimedia.org.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, Oct 31, 9:53 PM

Probably a completely different problem, but apt is failing on this new instance (mwv-bd808.andrewtestproject.codfw1dev.cloud) too:

$ apt update
Err:1 http://mirrors.wikimedia.org/debian buster-backports InRelease
  Could not connect to 208.80.153.75:5001 (208.80.153.75). - connect (111: Connection refused)
Err:2 http://apt.wikimedia.org/wikimedia buster-wikimedia InRelease
  Could not connect to 208.80.153.75:5001 (208.80.153.75). - connect (111: Connection refused)
Err:3 http://deb.debian.org/debian buster InRelease
  Could not connect to 208.80.153.75:5001 (208.80.153.75). - connect (111: Connection refused)
Err:4 http://deb.debian.org/debian buster-updates InRelease
  Unable to connect to 208.80.153.75:5001:
Err:5 http://security.debian.org buster/updates InRelease
  Could not connect to 208.80.153.75:5001 (208.80.153.75). - connect (111: Connection refused)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://deb.debian.org/debian/dists/buster/InRelease  Could not connect to 208.80.153.75:5001 (208.80.153.75). - connect (111: Connection refused)
W: Failed to fetch http://security.debian.org/dists/buster/updates/InRelease  Could not connect to 208.80.153.75:5001 (208.80.153.75). - connect (111: Connection refused)
W: Failed to fetch http://deb.debian.org/debian/dists/buster-updates/InRelease  Unable to connect to 208.80.153.75:5001:
W: Failed to fetch http://mirrors.wikimedia.org/debian/dists/buster-backports/InRelease  Could not connect to 208.80.153.75:5001 (208.80.153.75). - connect (111: Connection refused)
W: Failed to fetch http://apt.wikimedia.org/wikimedia/dists/buster-wikimedia/InRelease  Could not connect to 208.80.153.75:5001 (208.80.153.75). - connect (111: Connection refused)
W: Some index files failed to download. They have been ignored, or old ones used instead.
Andrew added a comment.Fri, Nov 1, 6:18 PM

The apt thing is long-standing and mostly not important for our purposes. I'll look at the ldap thing.

Andrew added a comment.Mon, Nov 4, 4:04 PM

/etc/ldap.conf isn't getting updated by puppet at all. I'll track down the dependency chain that's breaking this... it may well be due to the apt failure.

Andrew added a comment.Mon, Nov 4, 6:51 PM

The root of the issue is this:

# telnet cloudcontrol2001-dev.wikimedia.org 5001
Trying 208.80.153.59...
Trying 2620:0:860:2:208:80:153:59...
telnet: Unable to connect to remote host: Network is unreachable

That's breaking apt and subsequently ldap.conf.

Andrew added a comment.Mon, Nov 4, 8:14 PM

Best I can tell there must've been an unpuppetized proxy running on that host back in the day.