Page MenuHomePhabricator

Deprecate Spark 1.6 in favor of Spark 2.x only
Closed, ResolvedPublic2 Estimated Story Points


As discussed it would be great to get rid of Spark 1.6 puppet configuration on the Analytics Hadoop cluster to concentrate our efforts only on Spark 2.x. In theory there shouldn't be anybody still using it for regular jobs, but to be sure a more careful approach is needed.

Event Timeline

elukey triaged this task as Medium priority.Dec 17 2018, 3:34 PM
elukey created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 17 2018, 3:34 PM
fdans assigned this task to elukey.Dec 17 2018, 5:39 PM
fdans moved this task from Incoming to Operational Excellence on the Analytics board.
Milimetric reassigned this task from elukey to Ottomata.Jan 21 2019, 4:38 PM
Milimetric added a subscriber: Milimetric.

switching to Andrew per Luca

Sounds good ya I can do this! I'll send an email today and we can do it after All-Hands.

@Ottomata should we proceed?

Yes let's! Ok on todo list for this week!

Change 491767 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove usages of ::cdh::spark, we use ::spark2 now only

Change 491767 merged by Ottomata:
[operations/puppet@production] Remove usages of ::cdh::spark, we use ::spark2 now only

Mentioned in SAL (#wikimedia-analytics) [2019-02-20T16:03:40Z] <ottomata> removing spark 1 from Analytics cluster - T212134

Mentioned in SAL (#wikimedia-operations) [2019-02-20T16:03:44Z] <ottomata> removing spark 1 from Analytics cluster - T212134

sudo cumin '(R:Class = profile::analytics::cluster::client or R:Class = profile::hadoop::worker) and not an-coord1001.eqiad.wmnet' 'apt-get -y remove spark-core spark-python && mv -f /etc/spark /tmp/etc-spark1-to-delete'

Also removed /etc/spark on an-coord1001 and analytics-tool1001, and analytics1030 (hue).

I should have also excluded analytics1030 from my cumin command, it is the coordinator on the test cluster. Puppet fixed.

Ottomata moved this task from Next Up to Done on the Analytics-Kanban board.Feb 20 2019, 4:10 PM
Ottomata set the point value for this task to 2.
Nuria closed this task as Resolved.Feb 25 2019, 10:40 PM