Page MenuHomePhabricator
Create Task
Maniphest T379550

openstack: keystone may be failing to add users to the bastion project in Keystone and/or LDAP
Open, HighPublic
Actions

Description

We have detected one event of keystone not adding users to the bastion project after such user being added to other project.

Update: I double-checked both the keystone code and cookbooks involved in the process and they work correctly when I'm watching. We now have monitoring for this issue, and I wrote a runbook explaining causes and solutions here:

https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Users_not_in_bastion_project

Now we need to wait to see if/when this happens again and investigate.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
wmfkeystonehooks: double-check that we're adding a new user to a projectoperations/puppetproduction+17 -7
wmfkeystonehooks: more narration when ldap is adding a user to a project groupoperations/puppetproduction+6 -0
wmfkeystonehooks: add more explicit logging around ldap group syncoperations/puppetproduction+3 -2
Keystone: increase ldap pool sizeoperations/puppetproduction+4 -0
openstack: wmcs-bastionless: Fix conditionoperations/puppetproduction+9 -6
Add wmcs-bastionless utility scriptoperations/puppetproduction+134 -0
Add alerting for cloud-vps users outside of the Bastion projectoperations/alertsmaster+18 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

• aborrero created this task.Nov 11 2024, 12:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 11 2024, 12:08 PM
• aborrero triaged this task as High priority.Nov 11 2024, 12:08 PM
• aborrero moved this task from Backlog to Radar/observer on the User-aborrero board.
• aborrero added a subtask: T379478: Members of project-a4b03ac235d443159feaeafec24a92dd not seemingly added to project-bastion.
• aborrero closed subtask T379478: Members of project-a4b03ac235d443159feaeafec24a92dd not seemingly added to project-bastion as Resolved.Nov 11 2024, 12:11 PM
• aborrero mentioned this in T379478: Members of project-a4b03ac235d443159feaeafec24a92dd not seemingly added to project-bastion.
• bd808 added a project: Cloud-VPS.Nov 12 2024, 8:42 PM
• bd808 subscribed.Jan 25 2025, 9:52 PM
[18:19]  <    taavi> hmmmm
[18:19]  <    taavi> the yochayco user is not in the bastion project
[18:19]  <    taavi> that'd explain it
[18:21]  <    taavi> @yochayco: try now?
[18:22]  <    taavi> so it seems like this is T379550
[18:22]  <    wm-bb> <yochayco> Works. Amazing 👑👑

Maybe we should have some script that looks for Keystone users who are in this unexpected situation of having membership in some Cloud VPS project and no membership in the bastion project?

Andrew claimed this task.Feb 10 2025, 12:48 PM
gerritbot added a comment.Feb 10 2025, 2:47 PM

Change #1118526 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Add wmcs-bastionless utility script

https://gerrit.wikimedia.org/r/1118526

gerritbot added a project: Patch-For-Review.Feb 10 2025, 2:47 PM
Andrew added a comment.Feb 10 2025, 7:15 PM

One possible explanation for us getting into this state: If a user is added to a project as a /member/ but doesn't have the /reader/ role anywhere, they are not added to the bastion project.

That could be fixed in wmfkeystonehooks, but imo being member w/out reader is a broken state anyway so I'm going to see if the cookbooks prevent it.

Andrew added a comment.Feb 10 2025, 7:19 PM
In T379550#10536531, @Andrew wrote:

That could be fixed in wmfkeystonehooks, but imo being member w/out reader is a broken state anyway so I'm going to see if the cookbooks prevent it.

nope, the cookbook does the right thing, adding to 'reader' before 'member'.

gerritbot added a comment.Feb 11 2025, 12:40 PM

Change #1118526 merged by Andrew Bogott:

[operations/puppet@production] Add wmcs-bastionless utility script

https://gerrit.wikimedia.org/r/1118526

gerritbot added a comment.Feb 11 2025, 1:00 PM

Change #1118798 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/alerts@master] Add alerting for cloud-vps users outside of the Bastion project

https://gerrit.wikimedia.org/r/1118798

gerritbot added a comment.Feb 11 2025, 1:43 PM

Change #1118798 merged by Andrew Bogott:

[operations/alerts@master] Add alerting for cloud-vps users outside of the Bastion project

https://gerrit.wikimedia.org/r/1118798

Andrew lowered the priority of this task from High to Medium.Feb 11 2025, 2:26 PM
Andrew updated the task description. (Show Details)
Maintenance_bot removed a project: Patch-For-Review.Feb 11 2025, 2:32 PM
taavi merged a task: T387226: Members of https://ldap.toolforge.org/group/project-recommendation-api not added to project-bastion.Feb 25 2025, 3:47 PM
taavi added a subscriber: • XiaoXiao-WMF.
Andrew reopened subtask T387226: Members of https://ldap.toolforge.org/group/project-recommendation-api not added to project-bastion as Open.Mar 18 2025, 6:21 PM
Andrew mentioned this in T387226: Members of https://ldap.toolforge.org/group/project-recommendation-api not added to project-bastion.
Andrew closed subtask T387226: Members of https://ldap.toolforge.org/group/project-recommendation-api not added to project-bastion as Resolved.Mar 18 2025, 6:25 PM
Novem_Linguae subscribed.Apr 15 2025, 5:32 PM
taavi raised the priority of this task from Medium to High.May 24 2025, 11:41 AM
taavi mentioned this in T395184: Cannot access VPS instance uwl.wikicommunityhealth.eqiad1.wikimedia.cloud via SSH.
taavi subscribed.

This is still happening and affecting users so raising priority. The alert is also more or less broken.

gerritbot added a comment.May 24 2025, 11:48 AM

Change #1149811 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: wmcs-bastionless: Fix condition

https://gerrit.wikimedia.org/r/1149811

gerritbot added a project: Patch-For-Review.May 24 2025, 11:48 AM
dcaro subscribed.May 24 2025, 2:42 PM
Andrew mentioned this in Unknown Object (Task).May 24 2025, 6:33 PM
gerritbot added a comment.May 26 2025, 10:46 AM

Change #1149811 merged by Majavah:

[operations/puppet@production] openstack: wmcs-bastionless: Fix condition

https://gerrit.wikimedia.org/r/1149811

Maintenance_bot removed a project: Patch-For-Review.May 26 2025, 11:30 AM
tappof subscribed.May 27 2025, 10:21 AM
Andrew added a comment.Jun 2 2025, 12:26 PM
root@cloudcontrol1006:~# openstack role remove --user mortalandrew --project bastion reader
root@cloudcontrol1006:~# openstack role assignment list --user mortalandrew
root@cloudcontrol1006:~#  # it's empty!
root@cloudcontrol1006:~# openstack role add --user mortalandrew --project testlabs reader
root@cloudcontrol1006:~# openstack role assignment list --user mortalandrew
+----------------------------------+--------------+-------+----------+--------+--------+-----------+
| Role                             | User         | Group | Project  | Domain | System | Inherited |
+----------------------------------+--------------+-------+----------+--------+--------+-----------+
| f75a3c410bca4e96a1cf6ac103b0ccaf | mortalandrew |       | bastion  |        |        | False     |
| f75a3c410bca4e96a1cf6ac103b0ccaf | mortalandrew |       | testlabs |        |        | False     |
+----------------------------------+--------------+-------+----------+--------+--------+-----------+

So... that works, at least in a petri dish.

root@cloudcontrol1006:~# openstack role assignment list --user mortalandrew

root@cloudcontrol1006:~# openstack role add --user mortalandrew --project testlabs member
root@cloudcontrol1006:~# openstack role assignment list --user mortalandrew
+----------------------------------+--------------+-------+----------+--------+--------+-----------+
| Role                             | User         | Group | Project  | Domain | System | Inherited |
+----------------------------------+--------------+-------+----------+--------+--------+-----------+
| 38676f30eaeb44518bf7e144a73c8da6 | mortalandrew |       | testlabs |        |        | False     |
+----------------------------------+--------------+-------+----------+--------+--------+-----------+

That doesn't work. In theory none of our workflows support adding a user as a member w/out also being a reader anyway, but is it possible that this bug is a result of a different bug where that is happening?

• bd808 added a comment.Jun 18 2025, 10:51 PM

https://ldap.toolforge.org/user/zuuldevopsbot was not added to the bastion project when it was added to the zuul project. I will add it manually.

• bd808 added a comment.Jun 18 2025, 11:51 PM
In T379550#10930166, @bd808 wrote:

https://ldap.toolforge.org/user/zuuldevopsbot was not added to the bastion project when it was added to the zuul project. I will add it manually.

Manually adding the zuuldevopsbot user with sudo wmcs-openstack role add --user zuuldevopsbot --project bastion reader was really slow. Like really slow. I can't remember if the hook runs in-line or is a background process, but if it is backgrounded somehow I wonder if it might be timing out occasionally? This is just random speculation, so feel free to ignore me if this seems ridiculous.

Andrew added a comment.Jun 23 2025, 4:02 PM

suspicious log:

2025-06-18 11:38:07.604 127265 CRITICAL keystone [None req-c0a32d5d-203a-4817-badc-6856e1f13556 - - - - - -] Unhandled error: AttributeError: StateConnector has no attribute '_l'#0122025-06-18 11:38:07.604 127265 ERROR keystone Traceback (most recent call last):#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 972, in _apply_method_s#0122025-06-18 11:38:07.604 127265 ERROR keystone     return func(self,*args,**kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 625, in search_ext_s#0122025-06-18 11:38:07.604 127265 ERROR keystone     return self.result(msgid,all=1,timeout=timeout)[1]#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 535, in result#0122025-06-18 11:38:07.604 127265 ERROR keystone     resp_type, resp_data, resp_msgid = self.result2(msgid,all,timeout)#0122025-06-18 11:38:07.604 127265 ERROR keystone                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 539, in result2#0122025-06-18 11:38:07.604 127265 ERROR keystone     resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all,timeout)#0122025-06-18 11:38:07.604 127265 ERROR keystone                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 543, in result3#0122025-06-18 11:38:07.604 127265 ERROR keystone     resp_type, resp_data, resp_msgid, decoded_resp_ctrls, retoid, retval = self.result4(#0122025-06-18 11:38:07.604 127265 ERROR keystone                                                                            ^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 553, in result4#0122025-06-18 11:38:07.604 127265 ERROR keystone     ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)#0122025-06-18 11:38:07.604 127265 ERROR keystone                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 128, in _ldap_call#0122025-06-18 11:38:07.604 127265 ERROR keystone     result = func(*args,**kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone              ^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone ldap.SERVER_DOWN: {'result': -1, 'desc': "Can't contact LDAP server", 'ctrls': []}#0122025-06-18 11:38:07.604 127265 ERROR keystone #0122025-06-18 11:38:07.604 127265 ERROR keystone During handling of the above exception, another exception occurred:#0122025-06-18 11:38:07.604 127265 ERROR keystone #0122025-06-18 11:38:07.604 127265 ERROR keystone Traceback (most recent call last):#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldappool/__init__.py", line 394, in connection#0122025-06-18 11:38:07.604 127265 ERROR keystone     yield conn#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 781, in wrapper#0122025-06-18 11:38:07.604 127265 ERROR keystone     return func(self, conn, *args, **kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 939, in search_s#0122025-06-18 11:38:07.604 127265 ERROR keystone     return conn.search_s(base, scope, filterstr, attrlist, attrsonly)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 631, in search_s#0122025-06-18 11:38:07.604 127265 ERROR keystone     return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1033, in search_ext_s#0122025-06-18 11:38:07.604 127265 ERROR keystone     return self._apply_method_s(SimpleLDAPObject.search_ext_s,*args,**kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 975, in _apply_method_s#0122025-06-18 11:38:07.604 127265 ERROR keystone     self.reconnect(self._uri,retry_max=self._retry_max,retry_delay=self._retry_delay,force=True)#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 943, in reconnect#0122025-06-18 11:38:07.604 127265 ERROR keystone     self._apply_last_bind()#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 904, in _apply_last_bind#0122025-06-18 11:38:07.604 127265 ERROR keystone     func(self,*args,**kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 248, in simple_bind_s#0122025-06-18 11:38:07.604 127265 ERROR keystone     msgid = self.simple_bind(who,cred,serverctrls,clientctrls)#0122025-06-18 11:38:07.604 127265 ERROR keystone             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 242, in simple_bind#0122025-06-18 11:38:07.604 127265 ERROR keystone     return self._ldap_call(self._l.simple_bind,who,cred,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls))#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 128, in _ldap_call#0122025-06-18 11:38:07.604 127265 ERROR keystone     result = func(*args,**kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone              ^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone ldap.SERVER_DOWN: {'result': -1, 'desc': "Can't contact LDAP server", 'errno': 107, 'ctrls': [], 'info': 'Transport endpoint is not connected'}#0122025-06-18 11:38:07.604 127265 ERROR keystone #0122025-06-18 11:38:07.604 127265 ERROR keystone During handling of the above exception, another exception occurred:#0122025-06-18 11:38:07.604 127265 ERROR keystone #0122025-06-18 11:38:07.604 127265 ERROR keystone Traceback (most recent call last):#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask/app.py", line 1536, in __call__#0122025-06-18 11:38:07.604 127265 ERROR keystone     return self.wsgi_app(environ, start_response)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/werkzeug/middleware/proxy_fix.py", line 183, in __call__#0122025-06-18 11:38:07.604 127265 ERROR keystone     return self.app(environ, start_response)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__#0122025-06-18 11:38:07.604 127265 ERROR keystone     resp = self.call_func(req, *args, **kw)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func#0122025-06-18 11:38:07.604 127265 ERROR keystone     return self.func(req, *args, **kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/oslo_middleware/base.py", line 124, in __call__#0122025-06-18 11:38:07.604 127265 ERROR keystone     response = req.get_response(self.application)#0122025-06-18 11:38:07.604 127265 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send#0122025-06-18 11:38:07.604 127265 ERROR keystone     status, headers, app_iter = self.call_application(#0122025-06-18 11:38:07.604 127265 ERROR keystone                                 ^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application#0122025-06-18 11:38:07.604 127265 ERROR keystone     app_iter = application(self.environ, start_response)#0122025-06-18 11:38:07.604 127265 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 143, in __call__#0122025-06-18 11:38:07.604 127265 ERROR keystone     return resp(environ, start_response)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__#0122025-06-18 11:38:07.604 127265 ERROR keystone     resp = self.call_func(req, *args, **kw)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func#0122025-06-18 11:38:07.604 127265 ERROR keystone     return self.func(req, *args, **kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/oslo_middleware/base.py", line 124, in __call__#0122025-06-18 11:38:07.604 127265 ERROR keystone     response = req.get_response(self.application)#0122025-06-18 11:38:07.604 127265 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send#0122025-06-18 11:38:07.604 127265 ERROR keystone     status, headers, app_iter = self.call_application(#0122025-06-18 11:38:07.604 127265 ERROR keystone                                 ^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application#0122025-06-18 11:38:07.604 127265 ERROR keystone     app_iter = application(self.environ, start_response)#0122025-06-18 11:38:07.604 127265 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__#0122025-06-18 11:38:07.604 127265 ERROR keystone     resp = self.call_func(req, *args, **kw)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func#0122025-06-18 11:38:07.604 127265 ERROR keystone     return self.func(req, *args, **kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/osprofiler/web.py", line 111, in __call__#0122025-06-18 11:38:07.604 127265 ERROR keystone     return request.get_response(self.application)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send#0122025-06-18 11:38:07.604 127265 ERROR keystone     status, headers, app_iter = self.call_application(#0122025-06-18 11:38:07.604 127265 ERROR keystone                                 ^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application#0122025-06-18 11:38:07.604 127265 ERROR keystone     app_iter = application(self.environ, start_response)#0122025-06-18 11:38:07.604 127265 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__#0122025-06-18 11:38:07.604 127265 ERROR keystone     resp = self.call_func(req, *args, **kw)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func#0122025-06-18 11:38:07.604 127265 ERROR keystone     return self.func(req, *args, **kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/oslo_middleware/request_id.py", line 58, in __call__#0122025-06-18 11:38:07.604 127265 ERROR keystone     response = req.get_response(self.application)#0122025-06-18 11:38:07.604 127265 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send#0122025-06-18 11:38:07.604 127265 ERROR keystone     status, headers, app_iter = self.call_application(#0122025-06-18 11:38:07.604 127265 ERROR keystone                                 ^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application#0122025-06-18 11:38:07.604 127265 ERROR keystone     app_iter = application(self.environ, start_response)#0122025-06-18 11:38:07.604 127265 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__#0122025-06-18 11:38:07.604 127265 ERROR keystone     return self.app(environ, start_response)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__#0122025-06-18 11:38:07.604 127265 ERROR keystone     resp = self.call_func(req, *args, **kw)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func#0122025-06-18 11:38:07.604 127265 ERROR keystone     return self.func(req, *args, **kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 340, in __call__#0122025-06-18 11:38:07.604 127265 ERROR keystone     response = req.get_response(self._app)#0122025-06-18 11:38:07.604 127265 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send#0122025-06-18 11:38:07.604 127265 ERROR keystone     status, headers, app_iter = self.call_application(#0122025-06-18 11:38:07.604 127265 ERROR keystone                                 ^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application#0122025-06-18 11:38:07.604 127265 ERROR keystone     app_iter = application(self.environ, start_response)#0122025-06-18 11:38:07.604 127265 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/werkzeug/middleware/dispatcher.py", line 81, in __call__#0122025-06-18 11:38:07.604 127265 ERROR keystone     return app(environ, start_response)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask/app.py", line 1514, in wsgi_app#0122025-06-18 11:38:07.604 127265 ERROR keystone     response = self.handle_exception(e)#0122025-06-18 11:38:07.604 127265 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 271, in error_router#0122025-06-18 11:38:07.604 127265 ERROR keystone     return original_handler(e)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 271, in error_router#0122025-06-18 11:38:07.604 127265 ERROR keystone     return original_handler(e)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 271, in error_router#0122025-06-18 11:38:07.604 127265 ERROR keystone     return original_handler(e)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone   [Previous line repeated 28 more times]#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask/app.py", line 1511, in wsgi_app#0122025-06-18 11:38:07.604 127265 ERROR keystone     response = self.full_dispatch_request()#0122025-06-18 11:38:07.604 127265 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask/app.py", line 919, in full_dispatch_request#0122025-06-18 11:38:07.604 127265 ERROR keystone     rv = self.handle_user_exception(e)#0122025-06-18 11:38:07.604 127265 ERROR keystone          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 271, in error_router#0122025-06-18 11:38:07.604 127265 ERROR keystone     return original_handler(e)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 271, in error_router#0122025-06-18 11:38:07.604 127265 ERROR keystone     return original_handler(e)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 271, in error_router#0122025-06-18 11:38:07.604 127265 ERROR keystone     return original_handler(e)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone   [Previous line repeated 28 more times]#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask/app.py", line 917, in full_dispatch_request#0122025-06-18 11:38:07.604 127265 ERROR keystone     rv = self.dispatch_request()#0122025-06-18 11:38:07.604 127265 ERROR keystone          ^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask/app.py", line 902, in dispatch_request#0122025-06-18 11:38:07.604 127265 ERROR keystone     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 467, in wrapper#0122025-06-18 11:38:07.604 127265 ERROR keystone     resp = resource(*args, **kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask/views.py", line 110, in view#0122025-06-18 11:38:07.604 127265 ERROR keystone     return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request#0122025-06-18 11:38:07.604 127265 ERROR keystone     resp = meth(*args, **kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/server/flask/common.py", line 1159, in wrapper#0122025-06-18 11:38:07.604 127265 ERROR keystone     return f(*args, **kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/api/auth.py", line 327, in post#0122025-06-18 11:38:07.604 127265 ERROR keystone     token = authentication.authenticate_for_token(auth_data)#0122025-06-18 11:38:07.604 127265 ERROR keystone             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/api/_shared/authentication.py", line 201, in authenticate_for_token#0122025-06-18 11:38:07.604 127265 ERROR keystone     authenticate(auth_info, auth_context)#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/api/_shared/authentication.py", line 169, in authenticate#0122025-06-18 11:38:07.604 127265 ERROR keystone     resp = method.authenticate(auth_info.get_method_data(method_name))#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/wmfkeystoneauth/password_safelist.py", line 87, in authenticate#0122025-06-18 11:38:07.604 127265 ERROR keystone     return super(PasswordSafelist, self).authenticate(auth_payload)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/auth/plugins/password.py", line 32, in authenticate#0122025-06-18 11:38:07.604 127265 ERROR keystone     PROVIDERS.identity_api.authenticate(#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/common/manager.py", line 114, in wrapped#0122025-06-18 11:38:07.604 127265 ERROR keystone     __ret_val = __f(*args, **kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone                 ^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/notifications.py", line 710, in wrapper#0122025-06-18 11:38:07.604 127265 ERROR keystone     result = f(wrapped_self, user_id, *args, **kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 530, in wrapper#0122025-06-18 11:38:07.604 127265 ERROR keystone     return f(self, *args, **kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 542, in wrapper#0122025-06-18 11:38:07.604 127265 ERROR keystone     return f(self, *args, **kwargs)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 1061, in authenticate#0122025-06-18 11:38:07.604 127265 ERROR keystone     ref = driver.authenticate(entity_id, password)#0122025-06-18 11:38:07.604 127265 ERROR keystone           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 61, in authenticate#0122025-06-18 11:38:07.604 127265 ERROR keystone     user_ref = self._get_user(user_id)#0122025-06-18 11:38:07.604 127265 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 81, in _get_user#0122025-06-18 11:38:07.604 127265 ERROR keystone     return self.user.get(user_id)#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 317, in get#0122025-06-18 11:38:07.604 127265 ERROR keystone     obj = super().get(user_id, ldap_filter=ldap_filter)#0122025-06-18 11:38:07.604 127265 ERROR keystone           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 2197, in get#0122025-06-18 11:38:07.604 127265 ERROR keystone     ref = super().get(object_id, ldap_filter)#0122025-06-18 11:38:07.604 127265 ERROR keystone           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1874, in get#0122025-06-18 11:38:07.604 127265 ERROR keystone     res = self._ldap_get(object_id, ldap_filter)#0122025-06-18 11:38:07.604 127265 ERROR keystone           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1787, in _ldap_get#0122025-06-18 11:38:07.604 127265 ERROR keystone     res = conn.search_s(#0122025-06-18 11:38:07.604 127265 ERROR keystone           ^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1168, in search_s#0122025-06-18 11:38:07.604 127265 ERROR keystone     ldap_result = self.conn.search_s(#0122025-06-18 11:38:07.604 127265 ERROR keystone                   ^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 779, in wrapper#0122025-06-18 11:38:07.604 127265 ERROR keystone     with self._get_pool_connection() as conn:#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__#0122025-06-18 11:38:07.604 127265 ERROR keystone     self.gen.throw(typ, value, traceback)#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldappool/__init__.py", line 396, in connection#0122025-06-18 11:38:07.604 127265 ERROR keystone     self._release_connection(conn)#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldappool/__init__.py", line 354, in _release_connection#0122025-06-18 11:38:07.604 127265 ERROR keystone     connection.unbind_ext_s()#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldappool/__init__.py", line 113, in unbind_ext_s#0122025-06-18 11:38:07.604 127265 ERROR keystone     return ReconnectLDAPObject.unbind_ext_s(self, serverctrls,#0122025-06-18 11:38:07.604 127265 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 668, in unbind_ext_s#0122025-06-18 11:38:07.604 127265 ERROR keystone     msgid = self.unbind_ext(serverctrls,clientctrls)#0122025-06-18 11:38:07.604 127265 ERROR keystone             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 660, in unbind_ext#0122025-06-18 11:38:07.604 127265 ERROR keystone     res = self._ldap_call(self._l.unbind_ext,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls))#0122025-06-18 11:38:07.604 127265 ERROR keystone                           ^^^^^^^#0122025-06-18 11:38:07.604 127265 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 162, in __getattr__#0122025-06-18 11:38:07.604 127265 ERROR keystone     raise AttributeError('{} has no attribute {}'.format(#0122025-06-18 11:38:07.604 127265 ERROR keystone AttributeError: StateConnector has no attribute '_l'#0122025-06-18 11:38:07.604 127265 ERROR keystone
Andrew added a comment.Jun 23 2025, 4:03 PM
2025-06-18 15:27:43.714 3950140 CRITICAL keystone [None req-3f3de4ae-4796-4ba0-9363-f2ff944332f4 novaobserver teyora - - default default] Unhandled error: ldappool.MaxConnectionReachedError: ldap://ldap-rw.eqiad.wikimedia.org,ldap://ldap-rw.codfw.wikimedia.org,#0122025-06-18 15:27:43.714 3950140 ERROR keystone Traceback (most recent call last):#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask/app.py", line 1536, in __call__#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return self.wsgi_app(environ, start_response)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/werkzeug/middleware/proxy_fix.py", line 183, in __call__#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return self.app(environ, start_response)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__#0122025-06-18 15:27:43.714 3950140 ERROR keystone     resp = self.call_func(req, *args, **kw)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return self.func(req, *args, **kwargs)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/oslo_middleware/base.py", line 124, in __call__#0122025-06-18 15:27:43.714 3950140 ERROR keystone     response = req.get_response(self.application)#0122025-06-18 15:27:43.714 3950140 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send#0122025-06-18 15:27:43.714 3950140 ERROR keystone     status, headers, app_iter = self.call_application(#0122025-06-18 15:27:43.714 3950140 ERROR keystone                                 ^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application#0122025-06-18 15:27:43.714 3950140 ERROR keystone     app_iter = application(self.environ, start_response)#0122025-06-18 15:27:43.714 3950140 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 143, in __call__#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return resp(environ, start_response)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__#0122025-06-18 15:27:43.714 3950140 ERROR keystone     resp = self.call_func(req, *args, **kw)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return self.func(req, *args, **kwargs)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/oslo_middleware/base.py", line 124, in __call__#0122025-06-18 15:27:43.714 3950140 ERROR keystone     response = req.get_response(self.application)#0122025-06-18 15:27:43.714 3950140 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send#0122025-06-18 15:27:43.714 3950140 ERROR keystone     status, headers, app_iter = self.call_application(#0122025-06-18 15:27:43.714 3950140 ERROR keystone                                 ^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application#0122025-06-18 15:27:43.714 3950140 ERROR keystone     app_iter = application(self.environ, start_response)#0122025-06-18 15:27:43.714 3950140 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__#0122025-06-18 15:27:43.714 3950140 ERROR keystone     resp = self.call_func(req, *args, **kw)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return self.func(req, *args, **kwargs)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/osprofiler/web.py", line 111, in __call__#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return request.get_response(self.application)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send#0122025-06-18 15:27:43.714 3950140 ERROR keystone     status, headers, app_iter = self.call_application(#0122025-06-18 15:27:43.714 3950140 ERROR keystone                                 ^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application#0122025-06-18 15:27:43.714 3950140 ERROR keystone     app_iter = application(self.environ, start_response)#0122025-06-18 15:27:43.714 3950140 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__#0122025-06-18 15:27:43.714 3950140 ERROR keystone     resp = self.call_func(req, *args, **kw)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return self.func(req, *args, **kwargs)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/oslo_middleware/request_id.py", line 58, in __call__#0122025-06-18 15:27:43.714 3950140 ERROR keystone     response = req.get_response(self.application)#0122025-06-18 15:27:43.714 3950140 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send#0122025-06-18 15:27:43.714 3950140 ERROR keystone     status, headers, app_iter = self.call_application(#0122025-06-18 15:27:43.714 3950140 ERROR keystone                                 ^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application#0122025-06-18 15:27:43.714 3950140 ERROR keystone     app_iter = application(self.environ, start_response)#0122025-06-18 15:27:43.714 3950140 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return self.app(environ, start_response)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__#0122025-06-18 15:27:43.714 3950140 ERROR keystone     resp = self.call_func(req, *args, **kw)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return self.func(req, *args, **kwargs)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 340, in __call__#0122025-06-18 15:27:43.714 3950140 ERROR keystone     response = req.get_response(self._app)#0122025-06-18 15:27:43.714 3950140 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send#0122025-06-18 15:27:43.714 3950140 ERROR keystone     status, headers, app_iter = self.call_application(#0122025-06-18 15:27:43.714 3950140 ERROR keystone                                 ^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application#0122025-06-18 15:27:43.714 3950140 ERROR keystone     app_iter = application(self.environ, start_response)#0122025-06-18 15:27:43.714 3950140 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/werkzeug/middleware/dispatcher.py", line 81, in __call__#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return app(environ, start_response)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask/app.py", line 1514, in wsgi_app#0122025-06-18 15:27:43.714 3950140 ERROR keystone     response = self.handle_exception(e)#0122025-06-18 15:27:43.714 3950140 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 271, in error_router#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return original_handler(e)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 271, in error_router#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return original_handler(e)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 271, in error_router#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return original_handler(e)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone   [Previous line repeated 28 more times]#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask/app.py", line 1511, in wsgi_app#0122025-06-18 15:27:43.714 3950140 ERROR keystone     response = self.full_dispatch_request()#0122025-06-18 15:27:43.714 3950140 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask/app.py", line 919, in full_dispatch_request#0122025-06-18 15:27:43.714 3950140 ERROR keystone     rv = self.handle_user_exception(e)#0122025-06-18 15:27:43.714 3950140 ERROR keystone          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 271, in error_router#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return original_handler(e)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 271, in error_router#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return original_handler(e)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 271, in error_router#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return original_handler(e)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone   [Previous line repeated 28 more times]#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask/app.py", line 917, in full_dispatch_request#0122025-06-18 15:27:43.714 3950140 ERROR keystone     rv = self.dispatch_request()#0122025-06-18 15:27:43.714 3950140 ERROR keystone          ^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask/app.py", line 902, in dispatch_request#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 467, in wrapper#0122025-06-18 15:27:43.714 3950140 ERROR keystone     resp = resource(*args, **kwargs)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask/views.py", line 110, in view#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request#0122025-06-18 15:27:43.714 3950140 ERROR keystone     resp = meth(*args, **kwargs)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/server/flask/common.py", line 1159, in wrapper#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return f(*args, **kwargs)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/api/auth.py", line 327, in post#0122025-06-18 15:27:43.714 3950140 ERROR keystone     token = authentication.authenticate_for_token(auth_data)#0122025-06-18 15:27:43.714 3950140 ERROR keystone             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/api/_shared/authentication.py", line 201, in authenticate_for_token#0122025-06-18 15:27:43.714 3950140 ERROR keystone     authenticate(auth_info, auth_context)#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/api/_shared/authentication.py", line 169, in authenticate#0122025-06-18 15:27:43.714 3950140 ERROR keystone     resp = method.authenticate(auth_info.get_method_data(method_name))#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/wmfkeystoneauth/password_safelist.py", line 87, in authenticate#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return super(PasswordSafelist, self).authenticate(auth_payload)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/auth/plugins/password.py", line 32, in authenticate#0122025-06-18 15:27:43.714 3950140 ERROR keystone     PROVIDERS.identity_api.authenticate(#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/common/manager.py", line 114, in wrapped#0122025-06-18 15:27:43.714 3950140 ERROR keystone     __ret_val = __f(*args, **kwargs)#0122025-06-18 15:27:43.714 3950140 ERROR keystone                 ^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/notifications.py", line 710, in wrapper#0122025-06-18 15:27:43.714 3950140 ERROR keystone     result = f(wrapped_self, user_id, *args, **kwargs)#0122025-06-18 15:27:43.714 3950140 ERROR keystone              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 530, in wrapper#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return f(self, *args, **kwargs)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 542, in wrapper#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return f(self, *args, **kwargs)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 1061, in authenticate#0122025-06-18 15:27:43.714 3950140 ERROR keystone     ref = driver.authenticate(entity_id, password)#0122025-06-18 15:27:43.714 3950140 ERROR keystone           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 61, in authenticate#0122025-06-18 15:27:43.714 3950140 ERROR keystone     user_ref = self._get_user(user_id)#0122025-06-18 15:27:43.714 3950140 ERROR keystone                ^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 81, in _get_user#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return self.user.get(user_id)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 317, in get#0122025-06-18 15:27:43.714 3950140 ERROR keystone     obj = super().get(user_id, ldap_filter=ldap_filter)#0122025-06-18 15:27:43.714 3950140 ERROR keystone           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 2196, in get#0122025-06-18 15:27:43.714 3950140 ERROR keystone     with self.get_connection() as conn:#0122025-06-18 15:27:43.714 3950140 ERROR keystone          ^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1534, in get_connection#0122025-06-18 15:27:43.714 3950140 ERROR keystone     conn.simple_bind_s(user, password)#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1115, in simple_bind_s#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return self.conn.simple_bind_s(#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 916, in simple_bind_s#0122025-06-18 15:27:43.714 3950140 ERROR keystone     with self._get_pool_connection() as conn:#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__#0122025-06-18 15:27:43.714 3950140 ERROR keystone     return next(self.gen)#0122025-06-18 15:27:43.714 3950140 ERROR keystone            ^^^^^^^^^^^^^^#0122025-06-18 15:27:43.714 3950140 ERROR keystone
   File "/usr/lib/python3/dist-packages/ldappool/__init__.py", line 391, in connection#0122025-06-18 15:27:43.714 3950140 ERROR keystone     raise MaxConnectionReachedError(self.uri)#0122025-06-18 15:27:43.714 3950140 ERROR keystone ldappool.MaxConnectionReachedError: ldap://ldap-rw.eqiad.wikimedia.org,ldap://ldap-rw.codfw.wikimedia.org,#0122025-06-18 15:27:43.714 3950140 ERROR keystone
Andrew added a comment.Jun 23 2025, 4:03 PM
2025-06-18 19:12:06.914 275182 ERROR ldappool [None req-50ef4a30-984b-4be5-9faa-f21fabe9ec15 - - - - - -] Invalid credentials. Bind is uid=novaadmin,ou=people,dc=wikimedia,dc=org: ldap.INVALID_CREDENTIALS: {'msgtype': 97, 'msgid': 1, 'result': 49, 'desc': 'Invalid credentials', 'ctrls': []}#0122025-06-18 19:12:06.914 275182 ERROR ldappool Traceback (most recent call last):#0122025-06-18 19:12:06.914 275182 ERROR ldappool   File "/usr/lib/python3/dist-packages/ldappool/__init__.py", line 266, in _create_connector#0122025-06-18 19:12:06.914 275182 ERROR ldappool     self._bind(conn, bind, passwd)#0122025-06-18 19:12:06.914 275182 ERROR ldappool   File "/usr/lib/python3/dist-packages/ldappool/__init__.py", line 230, in _bind#0122025-06-18 19:12:06.914 275182 ERROR ldappool     conn.simple_bind_s(bind, passwd)#0122025-06-18 19:12:06.914 275182 ERROR ldappool   File "/usr/lib/python3/dist-packages/ldappool/__init__.py", line 102, in simple_bind_s#0122025-06-18 19:12:06.914 275182 ERROR ldappool     res = ReconnectLDAPObject.simple_bind_s(self, who, cred, serverctrls,#0122025-06-18 19:12:06.914 275182 ERROR ldappool           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 19:12:06.914 275182 ERROR ldappool   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 989, in simple_bind_s#0122025-06-18 19:12:06.914 275182 ERROR ldappool     res = self._apply_method_s(SimpleLDAPObject.simple_bind_s,*args,**kwargs)#0122025-06-18 19:12:06.914 275182 ERROR ldappool           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 19:12:06.914 275182 ERROR ldappool   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 972, in _apply_method_s#0122025-06-18 19:12:06.914 275182 ERROR ldappool     return func(self,*args,**kwargs)#0122025-06-18 19:12:06.914 275182 ERROR ldappool            ^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 19:12:06.914 275182 ERROR ldappool   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 249, in simple_bind_s#0122025-06-18 19:12:06.914 275182 ERROR ldappool     resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all=1,timeout=self.timeout)#0122025-06-18 19:12:06.914 275182 ERROR ldappool                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 19:12:06.914 275182 ERROR ldappool   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 543, in result3#0122025-06-18 19:12:06.914 275182 ERROR ldappool     resp_type, resp_data, resp_msgid, decoded_resp_ctrls, retoid, retval = self.result4(#0122025-06-18 19:12:06.914 275182 ERROR ldappool                                                                            ^^^^^^^^^^^^^#0122025-06-18 19:12:06.914 275182 ERROR ldappool   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 553, in result4#0122025-06-18 19:12:06.914 275182 ERROR ldappool     ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)#0122025-06-18 19:12:06.914 275182 ERROR ldappool                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#0122025-06-18 19:12:06.914 275182 ERROR ldappool   File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 128, in _ldap_call#0122025-06-18 19:12:06.914 275182 ERROR ldappool     result = func(*args,**kwargs)#0122025-06-18 19:12:06.914 275182 ERROR ldappool              ^^^^^^^^^^^^^^^^^^^^#0122025-06-18 19:12:06.914 275182 ERROR ldappool ldap.INVALID_CREDENTIALS: {'msgtype': 97, 'msgid': 1, 'result': 49, 'desc': 'Invalid credentials', 'ctrls': []}#0122025-06-18 19:12:06.914 275182 ERROR ldappool
gerritbot added a comment.Jun 23 2025, 5:21 PM

Change #1162979 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Keystone: increase ldap pool size

https://gerrit.wikimedia.org/r/1162979

gerritbot added a project: Patch-For-Review.Jun 23 2025, 5:21 PM

Change #1162979 merged by Andrew Bogott:

[operations/puppet@production] Keystone: increase ldap pool size

https://gerrit.wikimedia.org/r/1162979

Maintenance_bot removed a project: Patch-For-Review.Jun 23 2025, 5:30 PM
tappof unsubscribed.Jun 30 2025, 8:12 AM
Urbanecm_WMF mentioned this in T403052: Newly-added member of deployment-prep is not in bastion project.Aug 27 2025, 12:49 PM
Urbanecm_WMF added a subtask: T403052: Newly-added member of deployment-prep is not in bastion project.
RhinosF1 closed subtask T403052: Newly-added member of deployment-prep is not in bastion project as Resolved.Aug 27 2025, 3:34 PM
Restricted Application added a subscriber: RhinosF1. · View Herald TranscriptAug 27 2025, 3:34 PM
Urbanecm_WMF mentioned this in T404382: Newly-added member of wikitextexp is not in project-bastion LDAP group, but is in bastion project.Sep 11 2025, 5:49 PM
fnegri added a subtask: T404382: Newly-added member of wikitextexp is not in project-bastion LDAP group, but is in bastion project.Sep 12 2025, 12:58 PM
fnegri changed the status of subtask T404382: Newly-added member of wikitextexp is not in project-bastion LDAP group, but is in bastion project from Open to In Progress.Sep 12 2025, 1:04 PM
fnegri closed subtask T404382: Newly-added member of wikitextexp is not in project-bastion LDAP group, but is in bastion project as Resolved.Sep 12 2025, 1:13 PM
Andrew added a comment.Oct 22 2025, 1:31 PM

I just encountered a variation on this: new user dpogorzelski was in the bastion project with the proper role, but 'project-bastion' didn't appear in 'groups dpogorzelski'. It's not unusual for that to take a few minutes but this time it had been hours.

• bd808 renamed this task from openstack: keystone may be failing to add users to the bastion project to openstack: keystone may be failing to add users to the bastion project in Keystone and/or LDAP.Jan 26 2026, 4:48 PM
• bd808 added a subtask: T416689: Jacob Thwaites - unable to log into Cloud VPS Bastion.Feb 6 2026, 6:29 PM
Jacob_WMDE closed subtask T416689: Jacob Thwaites - unable to log into Cloud VPS Bastion as Resolved.Feb 9 2026, 11:08 AM
Andrew added a comment.Mar 31 2026, 2:39 PM
In T379550#11298228, @Andrew wrote:

I just encountered a variation on this: new user dpogorzelski was in the bastion project with the proper role, but 'project-bastion' didn't appear in 'groups dpogorzelski'. It's not unusual for that to take a few minutes but this time it had been hours.

This happened again today with user atsuko -- openstack says they're in 'bastion' but they have no entry in the actual ldap project.

I removed and re-added their openstack role and then they appeared in ldap.

As always, there's no sign of an ldap timeout or other error in the keystone logs. I think the next step is to noisy up those logs.

Andrew created subtask T421911: Keystone logs no longer appearing in logstash.Mar 31 2026, 2:40 PM
gerritbot added a comment.May 1 2026, 2:25 PM

Change #1281512 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] wmfkeystonehooks: add more explicit logging around ldap group sync

https://gerrit.wikimedia.org/r/1281512

gerritbot added a project: Patch-For-Review.May 1 2026, 2:25 PM

Change #1281512 merged by Andrew Bogott:

[operations/puppet@production] wmfkeystonehooks: add more explicit logging around ldap group sync

https://gerrit.wikimedia.org/r/1281512

Maintenance_bot removed a project: Patch-For-Review.May 1 2026, 2:31 PM
gerritbot added a comment.May 1 2026, 2:54 PM

Change #1281519 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] wmfkeystonehooks: more narration when ldap is adding a user to a project group

https://gerrit.wikimedia.org/r/1281519

gerritbot added a project: Patch-For-Review.May 1 2026, 2:54 PM
gerritbot added a comment.May 1 2026, 2:57 PM

Change #1281519 merged by Andrew Bogott:

[operations/puppet@production] wmfkeystonehooks: more narration when ldap is adding a user to a project group

https://gerrit.wikimedia.org/r/1281519

Maintenance_bot removed a project: Patch-For-Review.May 1 2026, 3:31 PM
JJMC89 added a subtask: T425815: Bastion immediately dropping SSH connection (LDAP sync issue?).May 9 2026, 4:33 PM
JJMC89 mentioned this in T425815: Bastion immediately dropping SSH connection (LDAP sync issue?).
Andrew closed subtask T425815: Bastion immediately dropping SSH connection (LDAP sync issue?) as Resolved.May 11 2026, 1:23 PM
gerritbot added a comment.May 11 2026, 4:00 PM

Change #1285854 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] wmfkeystonehooks: double-check that we're adding a new user to a project

https://gerrit.wikimedia.org/r/1285854

gerritbot added a project: Patch-For-Review.May 11 2026, 4:00 PM
Jony reopened subtask T425815: Bastion immediately dropping SSH connection (LDAP sync issue?) as Open.May 13 2026, 12:58 AM
Andrew closed subtask T425815: Bastion immediately dropping SSH connection (LDAP sync issue?) as Resolved.May 13 2026, 2:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits