I detected today a couple of http/500 errors when working with horizon. The error seems transient.
I don't have more information right now, but opening task so I don't forget to check logs and investigate more.
Description
Description
Details
Details
Related Changes in Gerrit:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| openstack: Configure WSGI to use horizon virtualenv | operations/puppet | production | +1 -1 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Invalid | None | T239352 CloudVPS: horizon improvements | |||
| Resolved | • JHedden | T240852 CloudVPS: horizon giving http/500 intermitently |
Event Timeline
Comment Actions
possibly related, in the horizon error log:
2019-12-16 12:38:00.187475 Traceback (most recent call last):
2019-12-16 12:38:00.187527 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/openstack_dashboard/wsgi/django.wsgi", line 14, in <module>
2019-12-16 12:38:00.187536 application = get_wsgi_application()
2019-12-16 12:38:00.187548 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/django/core/wsgi.py", line 14, in get_wsgi_application
2019-12-16 12:38:00.187554 django.setup()
2019-12-16 12:38:00.187565 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/django/__init__.py", line 18, in setup
2019-12-16 12:38:00.187571 apps.populate(settings.INSTALLED_APPS)
2019-12-16 12:38:00.187581 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/django/apps/registry.py", line 78, in populate
2019-12-16 12:38:00.187587 raise RuntimeError("populate() isn't reentrant")
2019-12-16 12:38:00.187611 RuntimeError: populate() isn't reentrantI reloaded Apache on both labweb boxes; we'll see if that helps.
Comment Actions
I've been working with horizon for a couple of days and never see this again. Closing task now.
Comment Actions
Happened to me now while trying to create a VM in the analytics project. I see intermittent 500s with the generic error msg "The server encountered an internal error or misconfiguration and was unable to complete your request", and errors in populating dropbox lists due to various errors when creating a VM.
Comment Actions
I see many errors like this today in both labweb servers:
2020-01-21 10:21:58.000296 mod_wsgi (pid=15346): Target WSGI script '/srv/deployment/horizon/venv/lib/python3.5/site-packages/openstack_dashboard/wsgi.py' cannot be loaded as Python module.
2020-01-21 10:21:58.000484 mod_wsgi (pid=15346): Exception occurred processing WSGI script '/srv/deployment/horizon/venv/lib/python3.5/site-packages/openstack_dashboard/wsgi.py'.
2020-01-21 10:21:58.000677 Traceback (most recent call last):
2020-01-21 10:21:58.000741 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/openstack_dashboard/wsgi.py", line 29, in <module>
2020-01-21 10:21:58.000754 application = get_wsgi_application()
2020-01-21 10:21:58.000768 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
2020-01-21 10:21:58.000777 django.setup(set_prefix=False)
2020-01-21 10:21:58.000790 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/django/__init__.py", line 24, in setup
2020-01-21 10:21:58.000798 apps.populate(settings.INSTALLED_APPS)
2020-01-21 10:21:58.000811 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/django/apps/registry.py", line 81, in populate
2020-01-21 10:21:58.000819 raise RuntimeError("populate() isn't reentrant")
2020-01-21 10:21:58.000845 RuntimeError: populate() isn't reentrantComment Actions
Mentioned in SAL (#wikimedia-cloud) [2020-01-21T10:24:02Z] <arturo> running sudo systemctl restart apache2.service in both labweb servers to try mitigating T240852
Comment Actions
On apache restart, I see many of:
aborrero@labweb1001:~ 12s $ sudo tail -f /var/log/apache2/horizon_error.log File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/horizon/utils/memoized.py", line 90, in remove NameError: name 'KeyError' is not defined Exception ignored in: <function memoized.<locals>.decorate.<locals>.wrapped.<locals>.remove at 0x7f1b0423a620> Traceback (most recent call last): File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/horizon/utils/memoized.py", line 90, in remove NameError: name 'KeyError' is not defined Exception ignored in: <function memoized.<locals>.decorate.<locals>.wrapped.<locals>.remove at 0x7f1b0423aae8> Traceback (most recent call last): File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/horizon/utils/memoized.py", line 90, in remove NameError: name 'KeyError' is not defined
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2020-03-23T13:28:03Z] <jeh> disable puppet on labweb100[1,2] to enable horizon event traces T240852
Comment Actions
Change 582907 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: Configure WSGI to use horizon virtualenv
Comment Actions
The above patch ensures we're using the correct virtual environment and directly addresses this exception:
2020-03-23 19:49:53.704141 mod_wsgi (pid=15112): Target WSGI script '/srv/deployment/horizon/venv/lib/python3.5/site-packages/openstack_dashboard/wsgi.py' cannot be loaded as Python module.
2020-03-23 19:49:53.704299 mod_wsgi (pid=15112): Exception occurred processing WSGI script '/srv/deployment/horizon/venv/lib/python3.5/site-packages/openstack_dashboard/wsgi.py'.
2020-03-23 19:49:53.706486 Traceback (most recent call last):
2020-03-23 19:49:53.706582 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/openstack_dashboard/wsgi.py", line 29, in <module>
2020-03-23 19:49:53.706593 application = get_wsgi_application()
2020-03-23 19:49:53.706600 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
2020-03-23 19:49:53.706606 django.setup(set_prefix=False)
2020-03-23 19:49:53.706612 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/django/__init__.py", line 24, in setup
2020-03-23 19:49:53.706617 apps.populate(settings.INSTALLED_APPS)
2020-03-23 19:49:53.706623 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/django/apps/registry.py", line 114, in populate
2020-03-23 19:49:53.706627 app_config.import_models()
2020-03-23 19:49:53.706637 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/django/apps/config.py", line 198, in import_models
2020-03-23 19:49:53.706642 self.models_module = import_module(models_module_name)
2020-03-23 19:49:53.706649 File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
2020-03-23 19:49:53.706653 return _bootstrap._gcd_import(name[level:], package, level)
2020-03-23 19:49:53.706659 File "<frozen importlib._bootstrap>", line 986, in _gcd_import
2020-03-23 19:49:53.706667 File "<frozen importlib._bootstrap>", line 969, in _find_and_load
2020-03-23 19:49:53.706674 File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
2020-03-23 19:49:53.706682 File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
2020-03-23 19:49:53.706689 File "<frozen importlib._bootstrap_external>", line 673, in exec_module
2020-03-23 19:49:53.706696 File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
2020-03-23 19:49:53.706704 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/openstack_auth/models.py", line 15, in <module>
2020-03-23 19:49:53.706711 from openstack_auth.user import User
2020-03-23 19:49:53.706718 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/openstack_auth/user.py", line 22, in <module>
2020-03-23 19:49:53.706723 from openstack_auth import utils
2020-03-23 19:49:53.706729 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/openstack_auth/utils.py", line 25, in <module>
2020-03-23 19:49:53.706748 from keystoneclient.v3 import client as client_v3
2020-03-23 19:49:53.706755 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/keystoneclient/v3/__init__.py", line 2, in <module>
2020-03-23 19:49:53.706760 from keystoneclient.v3.client import Client # noqa
2020-03-23 19:49:53.706765 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/keystoneclient/v3/client.py", line 23, in <module>
2020-03-23 19:49:53.706770 from keystoneclient import httpclient
2020-03-23 19:49:53.706775 File "<frozen importlib._bootstrap>", line 1013, in _handle_fromlist
2020-03-23 19:49:53.706782 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/keystoneclient/__init__.py", line 72, in __getattr__
2020-03-23 19:49:53.706787 return importlib.import_module('keystoneclient.%s' % name)
2020-03-23 19:49:53.706793 File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
2020-03-23 19:49:53.706797 return _bootstrap._gcd_import(name[level:], package, level)
2020-03-23 19:49:53.706802 File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/keystoneclient/httpclient.py", line 43, in <module>
2020-03-23 19:49:53.706807 import keyring
2020-03-23 19:49:53.706812 File "/usr/lib/python3/dist-packages/keyring/__init__.py", line 6, in <module>
2020-03-23 19:49:53.706817 from .core import (set_keyring, get_keyring, set_password, get_password,
2020-03-23 19:49:53.706823 File "/usr/lib/python3/dist-packages/keyring/core.py", line 149, in <module>
2020-03-23 19:49:53.706828 init_backend()
2020-03-23 19:49:53.706834 File "/usr/lib/python3/dist-packages/keyring/core.py", line 65, in init_backend
2020-03-23 19:49:53.706838 keyrings = filter(limit, backend.get_all_keyring())
2020-03-23 19:49:53.706843 File "/usr/lib/python3/dist-packages/keyring/util/__init__.py", line 20, in wrapper
2020-03-23 19:49:53.706847 func.always_returns = func(*args, **kwargs)
2020-03-23 19:49:53.706853 File "/usr/lib/python3/dist-packages/keyring/backend.py", line 191, in get_all_keyring
2020-03-23 19:49:53.706857 exceptions=TypeError))
2020-03-23 19:49:53.706862 File "/usr/lib/python3/dist-packages/keyring/util/__init__.py", line 29, in suppress_exceptions
2020-03-23 19:49:53.706866 for callable in callables:
2020-03-23 19:49:53.706873 File "/usr/lib/python3/dist-packages/keyring/backend.py", line 183, in is_class_viable
2020-03-23 19:49:53.706876 keyring_cls.priority
2020-03-23 19:49:53.706882 File "/usr/lib/python3/dist-packages/keyring/util/properties.py", line 22, in __get__
2020-03-23 19:49:53.706886 return self.fget.__get__(None, owner)()
2020-03-23 19:49:53.706892 File "/usr/lib/python3/dist-packages/keyring/backends/kwallet.py", line 127, in priority
2020-03-23 19:49:53.706896 return super(DBusKeyringKWallet4, cls).priority - 1
2020-03-23 19:49:53.706901 File "/usr/lib/python3/dist-packages/keyring/util/properties.py", line 22, in __get__
2020-03-23 19:49:53.706905 return self.fget.__get__(None, owner)()
2020-03-23 19:49:53.706911 File "/usr/lib/python3/dist-packages/keyring/backends/kwallet.py", line 30, in priority
2020-03-23 19:49:53.706915 bus = dbus.SessionBus()
2020-03-23 19:49:53.706922 File "/usr/lib/python3/dist-packages/dbus/_dbus.py", line 211, in __new__
2020-03-23 19:49:53.706926 mainloop=mainloop)
2020-03-23 19:49:53.706931 File "/usr/lib/python3/dist-packages/dbus/_dbus.py", line 100, in __new__
2020-03-23 19:49:53.706935 bus = BusConnection.__new__(subclass, bus_type, mainloop=mainloop)
2020-03-23 19:49:53.706941 File "/usr/lib/python3/dist-packages/dbus/bus.py", line 122, in __new__
2020-03-23 19:49:53.706945 bus = cls._new_for_bus(address_or_type, mainloop=mainloop)
2020-03-23 19:49:53.706962 dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NotSupported: Unable to autolaunch a dbus-daemon without a $DISPLAY for X11Comment Actions
Change 582907 merged by Jhedden:
[operations/puppet@production] openstack: Configure WSGI to use horizon virtualenv
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2020-03-25T17:09:20Z] <jeh> depool labweb1002 for horizon testing T240852
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2020-03-25T17:56:17Z] <jeh> add labweb1002 back into the pool - completed horizon testing T240852