It needs to run Buster, to avoid T257587
Description
Description
Details
Details
Customize query in gerrit
Related Objects
Related Objects
- Mentioned In
- T258799: Deprecate Hue (if possible)
T260929: hue.wikimedia.org throws an exception when trying to log in with a non-ASCII username
T233073: Test if Hue can run with Python3 - Mentioned Here
- T233073: Test if Hue can run with Python3
T257587: mod_auth_cas segfaulting on Stretch Apache setups using OpenSSL 1.0.2 and 1.1 (netmon/yarn)
Event Timeline
Comment Actions
Change 616011 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/dns@master] Add DNS record for an-tool1009
Comment Actions
Change 616011 merged by Muehlenhoff:
[operations/dns@master] Add DNS record for an-tool1009
Comment Actions
Change 616057 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add an-tool1009 to site.pp/DHCP
Comment Actions
Change 616057 merged by Muehlenhoff:
[operations/puppet@production] Add an-tool1009 to site.pp/DHCP
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2020-07-27T11:28:55Z] <moritzm> installing an-tool1009 T258768
Comment Actions
Change 616507 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Assign the Hue role to an-tool1009
Comment Actions
Change 616507 merged by Muehlenhoff:
[operations/puppet@production] Assign the Hue role to an-tool1009
Comment Actions
Change 616712 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Create component/cloudera for buster-wikimedia
Comment Actions
Change 616712 merged by Muehlenhoff:
[operations/puppet@production] Create component/cloudera for buster-wikimedia
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2020-07-28T09:35:07Z] <moritzm> imported libmysqlclient18 to component/cloudera T258768
Comment Actions
Change 616715 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Install libmysqlclient18 from component/cloudera
Comment Actions
Change 616715 merged by Muehlenhoff:
[operations/puppet@production] Install libmysqlclient18 from component/cloudera
Comment Actions
Hue fails to start due to some conflicts between the system Python and the modules bundled by Hue:
Jul 28 10:17:38 an-tool1009 systemd[1]: Failed to start LSB: Hue.
Jul 28 10:17:58 an-tool1009 systemd[1]: Starting LSB: Hue...
Jul 28 10:17:58 an-tool1009 hue[17322]: Starting Hue for Hadoop : hueTraceback (most recent call last):
Jul 28 10:17:58 an-tool1009 hue[17322]: File "/usr/lib/hue/build/env/bin/supervisor", line 8, in <module>
Jul 28 10:17:58 an-tool1009 hue[17322]: from pkg_resources import load_entry_point
Jul 28 10:17:58 an-tool1009 hue[17322]: File "/usr/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init.py", line 3018, in <module>
Jul 28 10:17:58 an-tool1009 hue[17322]: working_set = WorkingSet._build_master()
Jul 28 10:17:58 an-tool1009 hue[17322]: File "/usr/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/init.py", line 612, in _build_master
Jul 28 10:17:58 an-tool1009 hue[17322]: ws.require(requires)
Jul 28 10:17:58 an-tool1009 hue[17322]: File "/usr/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/init.py", line 918, in require
Jul 28 10:17:58 an-tool1009 hue[17322]: needed = self.resolve(parse_requirements(requirements))
Jul 28 10:17:58 an-tool1009 hue[17322]: File "/usr/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/init.py", line 813, in resolve
Jul 28 10:17:58 an-tool1009 hue[17322]: new_requirements = dist.requires(req.extras)[::-1]
Jul 28 10:17:58 an-tool1009 hue[17322]: File "/usr/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/init.py", line 2550, in requires
Jul 28 10:17:58 an-tool1009 hue[17322]: dm = self._dep_map
Jul 28 10:17:58 an-tool1009 hue[17322]: File "/usr/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/init.py", line 2751, in _dep_map
Jul 28 10:17:58 an-tool1009 hue[17322]: self.dep_map = self._compute_dependencies()
Jul 28 10:17:58 an-tool1009 hue[17322]: File "/usr/lib/hue/build/env/lib/python2.7/site-packages/pkg_resources/__init.py", line 2768, in _compute_dependencies
Jul 28 10:17:58 an-tool1009 hue[17322]: from _markerlib import compile as compile_marker
Jul 28 10:17:58 an-tool1009 hue[17322]: File "/usr/lib/hue/build/env/lib/python2.7/site-packages/_markerlib/init__.py", line 3, in <module>
Jul 28 10:17:58 an-tool1009 hue[17322]: from _markerlib.markers import default_environment, compile, interpret
Jul 28 10:17:58 an-tool1009 hue[17322]: File "/usr/lib/hue/build/env/lib/python2.7/site-packages/_markerlib/markers.py", line 24, in <module>
Jul 28 10:17:58 an-tool1009 hue[17322]: import weakref
Jul 28 10:17:58 an-tool1009 hue[17322]: File "/usr/lib/python2.7/weakref.py", line 14, in <module>
Jul 28 10:17:58 an-tool1009 hue[17322]: from _weakref import (
Jul 28 10:17:58 an-tool1009 hue[17322]: ImportError: cannot import name _remove_dead_weakref
Jul 28 10:17:58 an-tool1009 hue[17322]: failed!
Jul 28 10:17:58 an-tool1009 systemd[1]: hue.service: Control process exited, code=exited, status=1/FAILURE
Jul 28 10:17:58 an-tool1009 systemd[1]: hue.service: Failed with result 'exit-code'.
Jul 28 10:17:58 an-tool1009 systemd[1]: Failed to start LSB: Hue.
Comment Actions
elukey@an-tool1009:~$ /usr/lib/hue/build/env/bin/python2.7 --version Python 2.7.9 elukey@an-tool1009:~$ /usr/lib/hue/build/env/bin/python2.7 Python 2.7.9 (default, Mar 1 2015, 12:57:24) [GCC 4.9.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from _weakref import _remove_dead_weakref Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: cannot import name _remove_dead_weakref elukey@an-tool1009:~$ python2 --version Python 2.7.16 elukey@an-tool1009:~$ python2 Python 2.7.16 (default, Oct 10 2019, 22:02:15) [GCC 8.3.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from _weakref import _remove_dead_weakref >>>
Comment Actions
This commit landed in v2.7.14rc1 (and Stretch has 2.7.13): https://github.com/python/cpython/commit/3e37f4a11547a226c3c2f8bd612510465db397b9
Comment Actions
Change 617091 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add hue overrides to an-tool1009
Comment Actions
Change 617091 merged by Elukey:
[operations/puppet@production] Add hue overrides to an-tool1009
Comment Actions
In T233073 I am experimenting with building Hue 4.7.1, that is the latest upstream and it should support python3.7. The build procedure seems good (https://gerrit.wikimedia.org/r/618728) but it still suffers from a problem, namely inside the files generated there are references to absolute paths: for example, I see references to the directory used to build inside pbuilder rather than something like /usr/lib/hue/etc.. I tried to check the Makefile to spot for any variable to use to fix it, but didn't find one.
If I manually fix the paths with find and sed, then I see the issues indicated in https://github.com/cloudera/hue/issues/1239 (more runtime issues related to python3).
Comment Actions
This is pending https://github.com/cloudera/hue/issues/1239, there are some py2->py3 issues to fix before using 4.7.1
In https://gerrit.wikimedia.org/r/c/operations/debs/hue/+/618728 I added a basic debian packaging set up, that should be good for our purposes.
In the CAS context, https://github.com/cloudera/hue/blob/master/desktop/conf.dist/hue.ini#L365-L373 is very interesting, since it would allow us to have the same setting as Superset (httpd handles the login and sets an header that Hue reads). It would simplify a lot the Hue management (currently we sync manually every user from LDAP) and it would allow to use CAS without an extra login step.
Comment Actions
My pull requests were merged, but I opened https://github.com/cloudera/hue/issues/1262 and https://github.com/cloudera/hue/issues/1264.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2020-09-14T05:54:31Z] <elukey> execute "gnt-instance modify -B vcpus=4 an-tool1009.eqiad.wmnet" on ganeti1011 - T258768
Comment Actions
Change 627830 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] hue: add specific settings for version 4
Comment Actions
Change 627830 merged by Elukey:
[operations/puppet@production] hue: add specific settings for version 4
Comment Actions
Change 627838 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] hue: use gunicorn with Hue 4
Comment Actions
Change 627838 merged by Elukey:
[operations/puppet@production] hue: use gunicorn with Hue 4
Comment Actions
Change 627856 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add hue-next.wikimedia.org settings for ATS
Comment Actions
Change 627856 merged by Elukey:
[operations/puppet@production] Add hue-next.wikimedia.org settings for ATS
Comment Actions
Currently two outstanding UI issues:
https://github.com/cloudera/hue/issues/1273
https://github.com/cloudera/hue/issues/1272
In theory those are not blocking the migration of Hue, so this task could probably be closed and Analytics can keep working on these issues separately.
@MoritzMuehlenhoff thoughts?
Comment Actions
What do you mean with "migration to Hue" in this context? Do you mean that the migration to Buster per se is done with the setup of 1009/hue-next.wikimedia.org (agreed on that) or that we switch over hue.wikimedia.org to the 1009/v4? (Up to analytics depending on impact the remaining v4 bugs are to people's workflows)
Comment Actions
Yep I mean when we should pull the trigger on hue-next vs hue.wikimedia.org. I agree that my team should decide, so probably this task can be closed.