Page MenuHomePhabricator

Move away Hue and Camus (and other crons) from analytics1027
Closed, ResolvedPublic5 Story Points

Description

We decided to not refresh analytics1027 (5 yrs old hw) and move its services to other hosts:

  • Camus and other crons will run on analytics1003
  • Hue will run on thorium

This task is meant to track the work to drain analytics1027 as pre-step before decommissioning.

Overall steps to do:

  • Deploy the refinery to analytics1003
  • Move camus and other crons to analytics1003
  • move Hue to thorium (python backend)

Details

Related Gerrit Patches:
analytics/refinery/scap : masterRemove analytics1027 from scap targets, it will be decomissioned
operations/puppet : productionFix Hue apache config
operations/puppet : productionMove hue.w.o's backend to thorium
operations/puppet : productionAdd simple proxy for hue.w.o's backend on thorium
operations/puppet : productionInstall and run hue on throrium
operations/puppet : productionMove camus jobs from analytics1027 to analytlics1003
operations/puppet : productionMove hdfs balancer from an27 to an03
operations/puppet : productionMove data_drop and hive::site_hdfs roles from analytics1027 to analytics1003

Event Timeline

elukey created this task.Mar 3 2017, 1:18 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 3 2017, 1:18 PM
elukey updated the task description. (Show Details)Mar 3 2017, 1:21 PM
Nuria moved this task from Incoming to Wikistats Production on the Analytics board.Mar 6 2017, 4:30 PM

Change 344482 had a related patch set uploaded (by Ottomata):
[operations/puppet@production] Move data_drop and hive::site_hdfs roles from analytics1027 to analytics1003

https://gerrit.wikimedia.org/r/344482

Change 344482 merged by Ottomata:
[operations/puppet@production] Move data_drop and hive::site_hdfs roles from analytics1027 to analytics1003

https://gerrit.wikimedia.org/r/344482

Change 344487 had a related patch set uploaded (by Ottomata):
[operations/puppet@production] Move hdfs balancer from an27 to an03

https://gerrit.wikimedia.org/r/344487

Change 344487 merged by Ottomata:
[operations/puppet@production] Move hdfs balancer from an27 to an03

https://gerrit.wikimedia.org/r/344487

Change 344489 had a related patch set uploaded (by Ottomata):
[operations/puppet@production] Move camus jobs from analytics1027 to analytlics1003

https://gerrit.wikimedia.org/r/344489

Change 344489 merged by Ottomata:
[operations/puppet@production] Move camus jobs from analytics1027 to analytlics1003

https://gerrit.wikimedia.org/r/344489

Change 344491 had a related patch set uploaded (by Ottomata):
[analytics/refinery/scap@master] Remove analytics1027 from scap targets, it will be decomissioned

https://gerrit.wikimedia.org/r/344491

Mentioned in SAL (#wikimedia-analytics) [2017-03-23T20:17:07Z] <ottomata> moved all analytics cluster cron jobs (camus and other) from analytics1027 to analytics1003: T159527

Ottomata renamed this task from Move away Hue and Camus from analytics1027 to Move away Hue and Camus (and other crons) from analytics1027.Mar 23 2017, 8:18 PM
Ottomata updated the task description. (Show Details)

Crons and camus done. Hue still to do.

Ottomata claimed this task.Mar 23 2017, 8:20 PM
Ottomata edited projects, added Analytics-Kanban; removed Analytics.
Ottomata set the point value for this task to 3.
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.
Ottomata updated the task description. (Show Details)Mar 23 2017, 8:28 PM
Ottomata updated the task description. (Show Details)

Task description originally had ' and create the Apache proxy configuration (atm nginx is used on an1027)'. But we dont' need to! Nginx was originally puppetized on an1027 for SSL purposes, but we moved it to our usual https terminators + varnish a while ago. We'll need to spawn up Hue on thorium, and then make the puppet change to route hue.wikimedia.org requests to thorium instead of an27. That should be it!

Change 344496 had a related patch set uploaded (by Ottomata):
[operations/puppet@production] Install and run hue on throrium

https://gerrit.wikimedia.org/r/344496

Change 344496 merged by Ottomata:
[operations/puppet@production] Install and run hue on throrium

https://gerrit.wikimedia.org/r/344496

Hue is up and running on thorium! Cool!

ssh -N stat1002.eqiad.wmnet -L 8888:thorium.eqiad.wmnet:8888

http://localhost:8888

Now we just need to change puppet to route requests there instead of an27. Let's do that next week.

Change 344916 had a related patch set uploaded (by Elukey):
[operations/puppet@production] Move hue.w.o's backend to thorium

https://gerrit.wikimedia.org/r/344916

Change 344952 had a related patch set uploaded (by Elukey):
[operations/puppet@production] Add simple proxy for hue.w.o's backend on thorium

https://gerrit.wikimedia.org/r/344952

Change 344952 merged by Elukey:
[operations/puppet@production] Add simple proxy for hue.w.o's backend on thorium

https://gerrit.wikimedia.org/r/344952

Change 344916 merged by Elukey:
[operations/puppet@production] Move hue.w.o's backend to thorium

https://gerrit.wikimedia.org/r/344916

Mentioned in SAL (#wikimedia-operations) [2017-03-28T10:14:59Z] <elukey> Switching hue.w.o's backend (cache misc) from anaytics1027 to thorium - T159527

Created https://phabricator.wikimedia.org/T161597 to decommission analytics1027.

@Ottomata anything left to do?

Change 345129 had a related patch set uploaded (by Elukey):
[operations/puppet@production] Fix Hue apache config

https://gerrit.wikimedia.org/r/345129

Change 345129 merged by Elukey:
[operations/puppet@production] Fix Hue apache config

https://gerrit.wikimedia.org/r/345129

Nope! Looks good to me!

Ottomata updated the task description. (Show Details)Mar 28 2017, 1:29 PM
Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.
Ottomata changed the point value for this task from 3 to 5.Mar 28 2017, 3:21 PM
Nuria closed this task as Resolved.Mar 29 2017, 7:39 PM

Change 344491 abandoned by Ottomata:
Remove analytics1027 from scap targets, it will be decomissioned

Reason:
Done elsewhere

https://gerrit.wikimedia.org/r/344491