Page MenuHomePhabricator

Deprecate Hue and stop the services
Closed, ResolvedPublic

Description

We have been guiding our users away from Hue and towards Superset instead, for some time.

See these links for more information on Hue.

https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hue
https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hue/Administration

Our version is also rather old and is affected by a number of known vulnerabilites.
Given all of this, we would now like to formulate a plan to deprecate and disable the service.

Tagging Data-Engineering to verify that the service is indeed deprecated and to help determine a schedule for implementing it.

Event Timeline

BTullis triaged this task as High priority.Jul 18 2023, 5:05 PM
BTullis changed the task status from Open to Stalled.Sep 5 2023, 3:57 PM
BTullis lowered the priority of this task from High to Medium.
BTullis moved this task from Misc to Blocked / Waiting on the Data-Platform-SRE board.

There is ongoing discussion about the possible deprecation of Hue happening on Slack.
Marking this as stalled and unsetting the priority, whilst we work out how to proceed.

This can be unblocked when we have a newer version of Superset (as part of T347710) that allows better browsing of data.

The new version of Superset allows nested data types to be visualized and we have spoken with users about using Hive cli to update their datasets instead of using Hue. Conversation in slack

I think we can turn off Hue now when SRE has availability.

Gehel changed the task status from Stalled to Open.Feb 15 2024, 3:08 PM
Gehel removed lbowmaker as the assignee of this task.
Gehel added a subscriber: lbowmaker.

@lbowmaker I'm ready to kill the hue server. I've heard from the grapevine that some users were relying on Hue's R/W querying capabilities. Aren't they going to be impacted by the service removal?

The new version of Superset allows nested data types to be visualized and we have spoken with users about using Hive cli to update their datasets instead of using Hue. Conversation in slack

I think we can turn off Hue now when SRE has availability.

@brouberol thanks for checking, please see comment and Slack links above. We discussed this with users and we found an alternate approach so we are good to kill it! Thanks!

Change #1014549 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/dns@master] hue: drop CNAME DNS record

https://gerrit.wikimedia.org/r/1014549

Change #1014550 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] ats: drop mapping rule redirecting to hue.wikimedia.org

https://gerrit.wikimedia.org/r/1014550

Change #1014553 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] cache: remove caching config for hue.wikimedia.org

https://gerrit.wikimedia.org/r/1014553

Change #1014554 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] cumin: remove hue alias

https://gerrit.wikimedia.org/r/1014554

Change #1014555 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] site: change an-tool1009 role back to insetup

https://gerrit.wikimedia.org/r/1014555

Change #1014556 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] idp: drop hue client configuration

https://gerrit.wikimedia.org/r/1014556

Change #1014557 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] aqs: remove manifests and configuration

https://gerrit.wikimedia.org/r/1014557

Change #1014550 merged by Brouberol:

[operations/puppet@production] ats: drop mapping rule redirecting to hue.wikimedia.org

https://gerrit.wikimedia.org/r/1014550

Change #1014553 merged by Brouberol:

[operations/puppet@production] cache: remove caching config for hue.wikimedia.org

https://gerrit.wikimedia.org/r/1014553

Change #1014554 merged by Brouberol:

[operations/puppet@production] cumin: remove hue alias

https://gerrit.wikimedia.org/r/1014554

Change #1014555 merged by Brouberol:

[operations/puppet@production] site: change an-tool1009 role back to insetup

https://gerrit.wikimedia.org/r/1014555

Change #1014556 merged by Brouberol:

[operations/puppet@production] idp: drop hue client configuration

https://gerrit.wikimedia.org/r/1014556

Change #1014549 merged by Brouberol:

[operations/dns@master] hue: drop CNAME DNS record

https://gerrit.wikimedia.org/r/1014549

Mentioned in SAL (#wikimedia-analytics) [2024-03-27T15:02:15Z] <brouberol> dropping the hue.wikimedia.org CNAME - T341895

Mentioned in SAL (#wikimedia-analytics) [2024-03-27T15:14:49Z] <brouberol> decommissioning an-tool1009 now that hue is fully offline - T341895

cookbooks.sre.hosts.decommission executed by brouberol@cumin2002 for hosts: an-tool1009.eqiad.wmnet

  • an-tool1009.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox

Change #1014557 merged by Brouberol:

[operations/puppet@production] hue: remove manifests and configuration

https://gerrit.wikimedia.org/r/1014557

I have archived these Wikitech pages:

https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hue
https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hue/Administration

image.png (306×929 px, 40 KB)

I'll check for stray reference to Hue elsewhere in Wikitech.

Also, I'll send a message to releng to ask them to disable our gerrit repo that we used for building Hue packages: https://gerrit.wikimedia.org/r/admin/repos/operations/debs/hue,general

I've taken care of archiving the gerrit repo

Change #1016293 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove now obsolete site.pp entry

https://gerrit.wikimedia.org/r/1016293

Change #1016293 merged by Muehlenhoff:

[operations/puppet@production] Remove now obsolete site.pp entry

https://gerrit.wikimedia.org/r/1016293