Page MenuHomePhabricator

Move internal sites hosted on thorium to ganeti instance(s)
Closed, ResolvedPublic13 Estimated Story Points

Description

thorium.eqiad.wmnet hosts the following sites:

Internal only

Public

Of these, all of the internal sites can be hosted from anywhere, and are prime candidates to move to Ganeti instances. The public sites (with the exception of the 2 redirects) require large amount of disk space (around 1.5TB total).

We'd like to reimage thorium to Debian Stretch, but before we do, it might be easier to move the internal sites off of thorium onto one or multiple Ganeti instances. This will also allow us to upgrade turnilo and superset (T201430) to newer versions for Debian Stretch before we take down the existing instances.

So! Please allocate one or multiple Ganeti instances that we can use for this purpose. With the exception of yarn.wikimedia.org, the sites do a fair amount of work to query for and build graphs and dashboards, so a decently speced node would be good. Perhaps 3 instances would be best, one for turnilo, one for superset, and another for both hue and yarn. 4G RAM each, 4 core each?

Details

SubjectRepoBranchLines +/-
operations/puppetproduction+1 -323
operations/puppetproduction+2 -2
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+9 -0
operations/puppetproduction+13 -1
operations/puppetproduction+86 -15
operations/puppetproduction+2 -0
operations/puppetproduction+5 -5
operations/puppetproduction+122 -5
operations/puppetproduction+6 -0
operations/puppetproduction+25 -1
operations/puppetproduction+101 -0
analytics/turnilo/deploymaster+979 K -5
operations/puppetproduction+29 -1
analytics/turnilo/deploymaster+1 K -2 M
Show related patches Customize query in gerrit

Event Timeline

Ottomata triaged this task as Medium priority.Aug 15 2018, 1:48 PM
Ottomata created this task.

There are now:

analytics-tool1001 - for superset
analytics-tool1002 - for turnilo
analytics-tool1003 - for hue

more details on T202013#4516863

Currently the use role(spare::system). Next step would be to apply individual puppet roles. Note we are on stretch.

Change 454552 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Include turnilo on analytics-tool1002

https://gerrit.wikimedia.org/r/454552

Change 454575 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/turnilo/deploy@master] Update to version 1.7.2 and prepare for migration to analytics-tool1002

https://gerrit.wikimedia.org/r/454575

Change 454575 merged by Ottomata:
[analytics/turnilo/deploy@master] Update to version 1.7.2 and prepare for migration to analytics-tool1002

https://gerrit.wikimedia.org/r/454575

Change 454552 merged by Ottomata:
[operations/puppet@production] Include turnilo on analytics-tool1002

https://gerrit.wikimedia.org/r/454552

Change 454827 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/turnilo/deploy@master] Fix dependencies for turnilo 1.7.2, some weren't properly added to git last time

https://gerrit.wikimedia.org/r/454827

Change 454827 merged by Ottomata:
[analytics/turnilo/deploy@master] Fix dependencies for turnilo 1.7.2, some weren't properly added to git last time

https://gerrit.wikimedia.org/r/454827

Change 454831 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Create auth ldap proxy for turnilo on analytics-tool1002

https://gerrit.wikimedia.org/r/454831

Change 454831 merged by Ottomata:
[operations/puppet@production] Create auth ldap proxy for turnilo on analytics-tool1002

https://gerrit.wikimedia.org/r/454831

Change 454863 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Include superset on analytics-tool1003

https://gerrit.wikimedia.org/r/454863

Change 454863 merged by Ottomata:
[operations/puppet@production] Include superset on analytics-tool1003

https://gerrit.wikimedia.org/r/454863

Change 454869 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Install libmariadbclient18 for mysql databse for superset in stretch

https://gerrit.wikimedia.org/r/454869

Change 454869 merged by Ottomata:
[operations/puppet@production] Install libmariadbclient18 for mysql databse for superset in stretch

https://gerrit.wikimedia.org/r/454869

Change 454892 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Include yarn.wikimedia.org proxy vhost on analytics-tool1001

https://gerrit.wikimedia.org/r/454892

Change 454892 merged by Ottomata:
[operations/puppet@production] Include yarn.wikimedia.org proxy vhost on analytics-tool1001

https://gerrit.wikimedia.org/r/454892

Change 454905 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Rename profile::hadoop::sites::yarn to profile::hadoop::yarn_proxy

https://gerrit.wikimedia.org/r/454905

Change 454905 merged by Ottomata:
[operations/puppet@production] Rename profile::hadoop::sites::yarn to profile::hadoop::yarn_proxy

https://gerrit.wikimedia.org/r/454905

Change 454909 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Include hue on analytics-tool1001

https://gerrit.wikimedia.org/r/454909

Change 454910 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Don't include turnilo proxy on thorium

https://gerrit.wikimedia.org/r/454910

Change 454910 merged by Ottomata:
[operations/puppet@production] Don't include turnilo proxy on thorium

https://gerrit.wikimedia.org/r/454910

Change 454909 merged by Ottomata:
[operations/puppet@production] Include hue on analytics-tool1001

https://gerrit.wikimedia.org/r/454909

Oo, just ran into a problem with Hue that I need a little help with from some debian packaging pros. Let's try @MoritzMuehlenhoff :) (Also @elukey).

analytics-tool1001 is Stretch. thorium is Jessie. Hue (Hadoop-User-Experience) site runs on Jessie. We install Hue from packages created by Cloudera. Cloudera has not made and stretch package releases yet, so everywhere else that we run Stretch, we include the Jessie packages for Stretch and install them anyway. This has worked for everything so far, because most things Cloudera are JVM based. However, Hue is a Python Django app that uses a MySQL database. As such, it depends on some python MySQL library package, which itself depends on libmysqlclient18.

The following packages have unmet dependencies:
 hue-common : Depends: libmysqlclient18 (>= 5.5.24+dfsg-1) but it is not installable

libmysqlclient18 is not available in Stretch. AFAICT, it has been replaced in Stretch by libmariadbclient18.

What is the best way to resolve this? I could be wrong, but I think that if I install libmariadbclient18, actually .so dependency will be satisfied (will it?). If I then force install the hue packages, they should work (maybe?).

Is there a way to tell apt/dpkg that the dependency has been satisfied, and to install hue-common anyway?

@Dzahn suggests:

i think.. the proper way would be to unpack the hue-common packags, find the "Depends" line in the control file, change it from libmysqlclient18 to libmariadbclient18 and pack it again and then import it on our repo (with a -wmf version suffix?)

Oo, just ran into a problem with Hue that I need a little help with from some debian packaging pros. Let's try @MoritzMuehlenhoff :) (Also @elukey).

analytics-tool1001 is Stretch. thorium is Jessie. Hue (Hadoop-User-Experience) site runs on Jessie. We install Hue from packages created by Cloudera. Cloudera has not made and stretch package releases yet, so everywhere else that we run Stretch, we include the Jessie packages for Stretch and install them anyway. This has worked for everything so far, because most things Cloudera are JVM based. However, Hue is a Python Django app that uses a MySQL database. As such, it depends on some python MySQL library package, which itself depends on libmysqlclient18.

The following packages have unmet dependencies:
 hue-common : Depends: libmysqlclient18 (>= 5.5.24+dfsg-1) but it is not installable

libmysqlclient18 is not available in Stretch. AFAICT, it has been replaced in Stretch by libmariadbclient18.

What is the best way to resolve this? I could be wrong, but I think that if I install libmariadbclient18, actually .so dependency will be satisfied (will it?). If I then force install the hue packages, they should work (maybe?).

Is there a way to tell apt/dpkg that the dependency has been satisfied, and to install hue-common anyway?

You can generate a dummy package using the "equivs" package, but it's unlikely to work in practice here. The hue-common package embed a site copy of Python and that fail will when it attempts to open the library. I've had a look at exported symbols and they are quite different (you can check these with "nm -D LIBRARY_SO". In the past Mariadb re-used the mysql qlient API to ensure binary compatibility with MySQL, but that now went different paths.

That wouldn't be an issue if we had the source of the Hue deb ofc (then we could simply recompile against mariadb). I'm not sure if it's feasible to keep the rest of Cloudera and only install Hue from the upstream sources?

As a (terribly unclean!) hack/workaround it should be possible to install libmysqlclient18 from jessie on stretch, all the binary dependencies (libstdc++6, libc6, zlib, libgcc1) have compatible interfaces via versioned symbols or long-standing APIs.

I'm not sure if it's feasible to keep the rest of Cloudera and only install Hue from the upstream sources?

This might be possible; we'd have to build our own deb package for it though.

As a (terribly unclean!) hack/workaround it should be possible to install libmysqlclient18 from jessie on stretch, all the binary dependencies (libstdc++6, libc6, zlib, libgcc1) have compatible interfaces via versioned symbols or long-standing APIs.

Ah! This would definitely be easier! I'm not opposed to doing this because we only install Hue in this one place on this one VM. If this were a Hadoop worker I'd be less inclined, but as a standalone UI tool, I don't mind if you don't! :)

Change 455618 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add comment about Jessie deps in Stretch for Hue

https://gerrit.wikimedia.org/r/455618

Change 455618 merged by Ottomata:
[operations/puppet@production] Add comment about Jessie deps in Stretch for Hue

https://gerrit.wikimedia.org/r/455618

Change 455743 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add cache app directors for analytics_ui, superset and thorium

https://gerrit.wikimedia.org/r/455743

Change 455743 merged by Ottomata:
[operations/puppet@production] Add cache app directors for analytics_ui, superset and thorium

https://gerrit.wikimedia.org/r/455743

Change 455837 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Route yarn.wikimedia.org to hadoop_ui director on analytics-tool1001

https://gerrit.wikimedia.org/r/455837

Change 455837 merged by Ottomata:
[operations/puppet@production] Route yarn.wikimedia.org to hadoop_ui director on analytics-tool1001

https://gerrit.wikimedia.org/r/455837

Change 455839 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Route huewq.wikimedia.org to hadoop_ui director on analytics-tool1001

https://gerrit.wikimedia.org/r/455839

Change 455839 merged by Ottomata:
[operations/puppet@production] Route huewq.wikimedia.org to hadoop_ui director on analytics-tool1001

https://gerrit.wikimedia.org/r/455839

Change 455840 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Route turnilo.wikimedia.org to turnilo director on analytics-tool1002

https://gerrit.wikimedia.org/r/455840

Change 455840 merged by Ottomata:
[operations/puppet@production] Route turnilo.wikimedia.org to turnilo director on analytics-tool1002

https://gerrit.wikimedia.org/r/455840

Change 455842 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Route turnilo.wikimedia.org to turnilo director on analytics-tool1003

https://gerrit.wikimedia.org/r/455842

Change 455842 merged by Ottomata:
[operations/puppet@production] Route superset.wikimedia.org to superset director on analytics-tool1003

https://gerrit.wikimedia.org/r/455842

Change 455864 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove now unused turnilo, superset, hue, yarn puppetization from thorium

https://gerrit.wikimedia.org/r/455864

Ottomata set the point value for this task to 13.Aug 28 2018, 9:27 PM

Change 455864 merged by Ottomata:
[operations/puppet@production] Remove now unused turnilo, superset, hue, yarn puppetization from thorium

https://gerrit.wikimedia.org/r/455864