Page MenuHomePhabricator
Create Task
Maniphest T226647

nova-fullstack crashed from a keystone timeout
Closed, ResolvedPublic
Actions

Description

A seemingly-transient keystone issue caused the nova-fullstack service to die off entirely. Probably this should be caught as an exception unless it happens repeatedly.

Jun 26 15:08:37 cloudcontrol1003 puppet-agent[36948]: (/Stage[main]/Openstack::Nova::Fullstack::Service/Systemd::Service[nova-fullstack]/Service[nova-fullstack]/ensure) ensure changed 'stopped' to 'running'
Jun 26 15:08:37 cloudcontrol1003 puppet-agent[36948]: (/Stage[main]/Openstack::Nova::Fullstack::Service/Systemd::Service[nova-fullstack]/Service[nova-fullstack]) Unscheduling refresh on Service[nova-fullstack]
Jun 26 15:08:37 cloudcontrol1003 neutron-server[30160]: 2019-06-26 15:08:37.076 30574 INFO neutron.wsgi [req-a501e719-50c5-475e-b7b4-e79fe5e1fb48 novaadmin admin - - -] 10.64.20.28 - - [26/Jun/2019 15:08:37] "GET /v2.0/subnets.json?id=a69bdfad-d7d2-4cfa-8231-3d6d3e0074c9 HTTP/1.1" 200 774 0.264693
Jun 26 15:08:37 cloudcontrol1003 neutron-server[30160]: 2019-06-26 15:08:37.107 30587 INFO neutron.db.agents_db [req-53653daf-9b3e-4509-a2e6-4eb887e1202b - - - - -] Heartbeat received from L3 agent agent on host cloudnet1004, uuid 059d28ed-f858-486b-9fdb-215baa32d3c0 after 0:00:06.107591
Jun 26 15:08:37 cloudcontrol1003 neutron-server[30160]: 2019-06-26 15:08:37.122 30574 INFO neutron.wsgi [req-0d89f06a-bb4a-4ea3-8510-6b9a2ab04e25 novaadmin admin - - -] 10.64.20.28 - - [26/Jun/2019 15:08:37] "GET /v2.0/ports.json?network_id=7425e328-560c-4f00-8e99-706f3fb90bb4&device_owner=network%3Adhcp HTTP/1.1" 200 2682 0.035319
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]: 2019-06-26 15:08:37,691 INFO Starting new HTTP connection (1): cloudcontrol1003.wikimedia.org
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]: 2019-06-26 15:08:37,901 INFO Starting new HTTP connection (1): cloudcontrol1003.wikimedia.org
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]: Traceback (most recent call last):
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:   File "/usr/local/sbin/nova-fullstack", line 606, in <module>
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:     main()
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:   File "/usr/local/sbin/nova-fullstack", line 495, in main
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:     exist = nova_conn.servers.list()
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 749, in list
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:     "servers")
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 242, in _list
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:     resp, body = self.api.client.get(url)
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 173, in get
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:     return self.request(url, 'GET', **kwargs)
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 89, in request
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:     **kwargs)
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 331, in request
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:     return self.session.request(url, method, **kwargs)
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:   File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:     return func(*args, **kwargs)
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:   File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 405, in request
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:     resp = send(**kwargs)
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:   File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 449, in _send_request
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]:     raise exceptions.ConnectionRefused(msg)
Jun 26 15:08:37 cloudcontrol1003 nova-fullstack[38999]: keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://cloudcontrol1003.wikimedia.org:8774/v2.1/servers/detail
Jun 26 15:08:37 cloudcontrol1003 systemd[1]: nova-fullstack.service: Main process exited, code=exited, status=1/FAILURE
Jun 26 15:08:37 cloudcontrol1003 systemd[1]: nova-fullstack.service: Unit entered failed state.
Jun 26 15:08:37 cloudcontrol1003 systemd[1]: nova-fullstack.service: Failed with result 'exit-code'.

Details

SubjectRepoBranchLines +/-
fullstack monitoring: adjust behavior of the leak counteroperations/puppetproduction+10 -14
nova-fullstack: fix name of nrpe checkoperations/puppetproduction+1 -1
nova-fullstack monitoring: fix a misnamed fileoperations/puppetproduction+1 -1
nova-fullstack: add a watch on the number of leaked VMs.operations/puppetproduction+59 -0
Customize query in gerrit

Related Objects

Event Timeline

Andrew created this task.Jun 26 2019, 3:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 26 2019, 3:25 PM
Paladox subscribed.Jun 26 2019, 3:25 PM
• JHedden subscribed.Jun 26 2019, 3:45 PM

This is related to the task T226632. Nova-api was stopped and started for the wmf_scheduler_hosts_pool config update.

Techguru.pc subscribed.EditedJun 26 2019, 9:04 PM

Looks like the transient error auto-resolved after the deployment completed. Seems like the appropriate action may be to suppress the watchdog during service restart/service upgrade. thoughts? Should this issue be closed in favor of a task addressing the watchdog? also to look at -- somewhere between 60 and 120 seconds seems like an eternity for a soft service restart of a critical service.

17:10 <#wikimedia-operations> icinga-wmPROBLEM - nova instance creation test on cloudcontrol1003 is CRITICAL: PROCS CRITICAL: 0 processes with command name python, args nova-fullstack https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting
17:11 <#wikimedia-operations> icinga-wmPROBLEM - Check systemd state on cloudcontrol1003 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
17:12 <#wikimedia-operations> icinga-wmRECOVERY - nova instance creation test on cloudcontrol1003 is OK: PROCS OK: 1 process with command name python, args nova-fullstack https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting
17:12 <#wikimedia-operations> icinga-wmRECOVERY - Check systemd state on cloudcontrol1003 is OK: OK - running: The system is fully operational
gerritbot added a comment.Jun 27 2019, 6:21 PM

Change 519478 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova-fullstack: add a watch on the number of leaked VMs.

https://gerrit.wikimedia.org/r/519478

gerritbot added a project: Patch-For-Review.Jun 27 2019, 6:21 PM
gerritbot added a comment.Jun 27 2019, 8:17 PM

Change 519478 merged by Andrew Bogott:
[operations/puppet@production] nova-fullstack: add a watch on the number of leaked VMs.

https://gerrit.wikimedia.org/r/519478

gerritbot added a comment.Jun 27 2019, 8:24 PM

Change 519513 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova-fullstack monitoring: fix a misnamed file

https://gerrit.wikimedia.org/r/519513

gerritbot added a comment.Jun 27 2019, 8:25 PM

Change 519513 merged by Andrew Bogott:
[operations/puppet@production] nova-fullstack monitoring: fix a misnamed file

https://gerrit.wikimedia.org/r/519513

Maintenance_bot removed a project: Patch-For-Review.Jun 27 2019, 9:10 PM
gerritbot added a comment.Jun 27 2019, 9:49 PM

Change 519531 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova-fullstack: fix name of nrpe check

https://gerrit.wikimedia.org/r/519531

gerritbot added a project: Patch-For-Review.Jun 27 2019, 9:49 PM
gerritbot added a comment.Jun 27 2019, 9:52 PM

Change 519531 merged by Andrew Bogott:
[operations/puppet@production] nova-fullstack: fix name of nrpe check

https://gerrit.wikimedia.org/r/519531

Maintenance_bot removed a project: Patch-For-Review.Jun 27 2019, 10:10 PM
gerritbot added a comment.Jun 28 2019, 3:29 PM

Change 519653 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] fullstack monitoring: adjust behavior of the leak counter

https://gerrit.wikimedia.org/r/519653

gerritbot added a project: Patch-For-Review.Jun 28 2019, 3:29 PM

Change 519653 merged by Andrew Bogott:
[operations/puppet@production] fullstack monitoring: adjust behavior of the leak counter

https://gerrit.wikimedia.org/r/519653

Maintenance_bot removed a project: Patch-For-Review.Jun 28 2019, 4:10 PM
• Bstorm triaged this task as Medium priority.Feb 11 2020, 4:30 PM
Andrew closed this task as Resolved.Apr 24 2021, 8:19 PM
Andrew claimed this task.

I don't think this is happening anymore; closing in favor of T280514

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL