Page MenuHomePhabricator

Upgrade analytics cluster to Cloudera CDH 5.16.1
Closed, ResolvedPublic13 Story Points

Description

We should first evaluate wether this distro provides enough value for us to upgrade, also work is a bit contingent on other testing being done for kerberos, we want to test upgrades on testing cluster once cluster is stable.

Current version: 5.15.0

Issue fixed up to 5.16.1:

What's new up to 5.16.1:

Overall nothing really juicy from what I can see, but there are a couple of security fixes that are surely good to have.

Last upgrade: T204759
Old etherpad: https://etherpad.wikimedia.org/p/analytics-cdh5.15
New etherpad: https://etherpad.wikimedia.org/p/analytics-cdh5.16.1

Related Objects

Event Timeline

Nuria created this task.Mar 14 2019, 7:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 14 2019, 7:05 PM
Milimetric triaged this task as High priority.Mar 18 2019, 3:20 PM
Milimetric moved this task from Incoming to Operational Excellence on the Analytics board.
elukey renamed this task from Upgrade analytics cluster to cloudera distro CDH 5.16 to Upgrade analytics cluster to Cloudera CDH 5.16.1.Mar 27 2019, 2:39 PM
elukey claimed this task.
elukey lowered the priority of this task from High to Normal.
elukey updated the task description. (Show Details)
elukey added a project: User-Elukey.
elukey added subscribers: JAllemandou, Ottomata.
elukey moved this task from Backlog to In Progress on the User-Elukey board.Apr 1 2019, 11:28 AM
elukey updated the task description. (Show Details)Apr 1 2019, 1:43 PM

Change 500453 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] aptrepo: update cloudera-jessie to 5.16.1

https://gerrit.wikimedia.org/r/500453

Change 500465 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] cumin: add aliases for Hadoop HDFS journalnodes

https://gerrit.wikimedia.org/r/500465

Change 500465 merged by Elukey:
[operations/puppet@production] cumin: add aliases for Hadoop HDFS journalnodes

https://gerrit.wikimedia.org/r/500465

Change 500967 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] cumin: add more hadoop-related aliases

https://gerrit.wikimedia.org/r/500967

Change 500967 merged by Elukey:
[operations/puppet@production] cumin: add more hadoop-related aliases

https://gerrit.wikimedia.org/r/500967

Change 501162 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] cumin: add hadoop-hdfs-backup aliases

https://gerrit.wikimedia.org/r/501162

Change 501162 merged by Elukey:
[operations/puppet@production] cumin: add hadoop-hdfs-backup aliases

https://gerrit.wikimedia.org/r/501162

elukey updated the task description. (Show Details)Apr 10 2019, 7:59 AM

Change 500453 merged by Elukey:
[operations/puppet@production] aptrepo: update cloudera-jessie to 5.16.1

https://gerrit.wikimedia.org/r/500453

Mentioned in SAL (#wikimedia-operations) [2019-04-10T08:36:50Z] <elukey> update thirdparty/cloudera packages to cdh 5.16.1 for jessie/stretch-wikimedia - T218343

Change 503266 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet/cdh@master] oozie: override the oozie-setup script

https://gerrit.wikimedia.org/r/503266

I have tested https://etherpad.wikimedia.org/p/analytics-cdh5.16.1 upgrading the Hadoop test cluster, all good!

https://gerrit.wikimedia.org/r/503266 needs to be merged/deployed before upgrading but it is a minor nit. I have also removed all the spark 1 packages and rechecked that the packages that we are installing are already present on the hosts of the Production cluster.

I can confirm that the refine webrequest job in the testing cluster works fine, we only need to test Spark 2 but I don't foresee anything weird on that front. After this, ready to upgrade production!

elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.

Change 503266 merged by Elukey:
[operations/puppet/cdh@master] oozie: override the oozie-setup script

https://gerrit.wikimedia.org/r/503266

Change 504268 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::hadoop::spark2: auto upload of spark2-assembly.zip

https://gerrit.wikimedia.org/r/504268

Change 504268 merged by Elukey:
[operations/puppet@production] profile::hadoop::spark2: auto upload of spark2-assembly.zip

https://gerrit.wikimedia.org/r/504268

Mentioned in SAL (#wikimedia-operations) [2019-04-17T13:52:43Z] <elukey> upgrading hadoop cdh distrubition to 5.16.1 on all the Hadoop-related nodes - T218343

Change 504585 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] cumin: update hadoop alias

https://gerrit.wikimedia.org/r/504585

Change 504585 merged by Elukey:
[operations/puppet@production] cumin: update hadoop alias

https://gerrit.wikimedia.org/r/504585

elukey moved this task from In Progress to Done on the User-Elukey board.Apr 18 2019, 8:32 AM
elukey set the point value for this task to 13.

Upgrade done, no errors reported!

elukey moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Apr 18 2019, 8:32 AM
Nuria closed this task as Resolved.May 14 2019, 9:14 PM