Page MenuHomePhabricator
Create Task
Maniphest T142394

Audit the labs infrastructure scripts that depend on LDAP to make sure they are resilient to failover
Closed, ResolvedPublic
Actions

Description

Since we're restarting them often now, we need to make sure failover + failure handling works properly.

Details

SubjectRepoBranchLines +/-
labs: Set timeout for ldap3 using scriptsoperations/puppetproduction+4 -2
tools: Use LDAP servers in HA manner for maintain-kubeusersoperations/puppetproduction+21 -2
labstore: Configure LDAP failover + timeout for create-dbuseroperations/puppetproduction+18 -3
Customize query in gerrit

Related Objects

Event Timeline

yuvipanda created this task.Aug 8 2016, 3:40 PM
Restricted Application added a project: Cloud-Services. · View Herald TranscriptAug 8 2016, 3:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
gerritbot subscribed.Aug 8 2016, 4:08 PM

Change 303565 had a related patch set uploaded (by Yuvipanda):
labstore: Configure LDAP failover timeout for create-dbuser

https://gerrit.wikimedia.org/r/303565

yuvipanda mentioned this in rOPUP825397be2ed9: labstore: Configure LDAP failover + timeout for create-dbuser.Aug 8 2016, 4:11 PM
yuvipanda mentioned this in rOPUPb2e517de8c5e: labstore: Configure LDAP failover + timeout for create-dbuser.Aug 8 2016, 4:21 PM
yuvipanda mentioned this in rOPUP507c920faf1b: labstore: Configure LDAP failover + timeout for create-dbuser.Aug 8 2016, 4:31 PM
yuvipanda mentioned this in rOPUP9881605b05d6: labstore: Configure LDAP failover + timeout for create-dbuser.Aug 8 2016, 4:36 PM
yuvipanda mentioned this in rOPUP98ea146cbd00: labstore: Configure LDAP failover + timeout for create-dbuser.Aug 8 2016, 4:50 PM
yuvipanda mentioned this in rOPUP17a6aae6f7bd: labstore: Configure LDAP failover + timeout for create-dbuser.Aug 8 2016, 5:19 PM
gerritbot added a comment.Aug 8 2016, 6:46 PM

Change 303565 merged by Yuvipanda:
labstore: Configure LDAP failover timeout for create-dbuser

https://gerrit.wikimedia.org/r/303565

yuvipanda mentioned this in rOPUPce745524d6fb: labstore: Configure LDAP failover + timeout for create-dbuser.Aug 8 2016, 6:49 PM
yuvipanda merged a task: T141203: Add appropriate timeouts to maintain-kubeusers.Aug 8 2016, 7:08 PM
gerritbot added a comment.Aug 8 2016, 7:08 PM

Change 303607 had a related patch set uploaded (by Yuvipanda):
tools: Use LDAP servers in HA manner for maintain-kubeusers

https://gerrit.wikimedia.org/r/303607

yuvipanda mentioned this in rOPUPf5da83fbac28: tools: Use LDAP servers in HA manner for maintain-kubeusers.Aug 8 2016, 7:14 PM
yuvipanda mentioned this in rOPUPa8fcc4150e8f: tools: Use LDAP servers in HA manner for maintain-kubeusers.
gerritbot added a comment.Aug 8 2016, 7:19 PM

Change 303607 merged by Yuvipanda:
tools: Use LDAP servers in HA manner for maintain-kubeusers

https://gerrit.wikimedia.org/r/303607

yuvipanda mentioned this in rOPUPf0165d4fe568: tools: Use LDAP servers in HA manner for maintain-kubeusers.Aug 8 2016, 7:20 PM
yuvipanda mentioned this in rOPUP58f9692bf689: tools: Use LDAP servers in HA manner for maintain-kubeusers.Aug 8 2016, 7:25 PM
yuvipanda added a comment.Aug 18 2016, 8:02 PM

maintain-kubeusers was stuck connecting to seaborgium today for minutes, despite there being a 1s connection timeout :(

maintain- 3414 root    4u  IPv4           10668954      0t0     TCP tools-k8s-master-01.tools.eqiad.wmflabs:33368->seaborgium.wikimedia.org:ldap (CLOSE_WAIT)
maintain- 3414 root    5u  IPv4           10670197      0t0     TCP tools-k8s-master-01.tools.eqiad.wmflabs:33432->seaborgium.wikimedia.org:ldap (CLOSE_WAIT)
maintain- 3414 root    7u  IPv4           10668828      0t0     TCP tools-k8s-master-01.tools.eqiad.wmflabs:33302->seaborgium.wikimedia.org:ldap (CLOSE_WAIT)
bd808 subscribed.Aug 18 2016, 8:26 PM

Adding a time_limit=N to conn.search in get_tools_from_ldap might help. The socket was connected and in CLOSE_WAIT according to what @yuvipanda said on irc, so the 1s connect limit wouldn't apply.

time_limit: number of seconds allowed for the search (defaults to None). > If None the search can take an unlimited amount of time, unless the server has a more restrictive rule. -- http://ldap3.readthedocs.io/searches.html

gerritbot added a comment.Aug 19 2016, 3:45 PM

Change 305616 had a related patch set uploaded (by Yuvipanda):
labs: Set timeout for ldap3 using scripts

https://gerrit.wikimedia.org/r/305616

gerritbot added a project: Patch-For-Review.Aug 19 2016, 3:45 PM
bd808 mentioned this in rOPUP8da8e973b55e: labs: Set timeout for ldap3 using scripts.Aug 19 2016, 3:48 PM
gerritbot added a comment.Aug 19 2016, 6:58 PM

Change 305616 merged by Yuvipanda:
labs: Set timeout for ldap3 using scripts

https://gerrit.wikimedia.org/r/305616

yuvipanda mentioned this in rOPUP9c975c2ac6fa: labs: Set timeout for ldap3 using scripts.Aug 19 2016, 7:04 PM
• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:41 PM
• GTirloni edited projects, added cloud-services-team (Kanban); removed Cloud-VPS.Mar 24 2019, 10:33 AM
• Bstorm closed this task as Resolved.Feb 11 2020, 5:40 PM
• Bstorm claimed this task.
• Bstorm subscribed.

resolved by load balancing

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL