Page MenuHomePhabricator
Create Task
Maniphest T119646

Upgrade to CDH 5.5 {hawk}
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

Most excited about;

  • Apache Spark 1.5 (including Spark SQL, DataFrames API, and MLlib per above)

Details

SubjectRepoBranchLines +/-
Increase analytics mysql meta innodb_buffer_pool_size to 1G, query_cache_size to 16Moperations/puppetproduction+4 -1
Fix camus checker classpath in CDH 5.5analytics/refinerymaster+1 -1
Analytics MySQL Meta instance: 4G innodb_buffer_pool_size 4 instancesoperations/puppetproduction+8 -1
Add analytics1015.yaml for analytics-adminsoperations/puppetproduction+5 -14
Add accidentally uncommitted hiera cdh users.yamloperations/puppetproduction+2 -0
Use proper flume package naems in reprepro updatesoperations/puppetproduction+1 -1
spark-core now depends on flume, mirror it from CDH to our aptoperations/puppetproduction+1 -1
Update Cloudera package reprepro updates to CDH 5.5.2operations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Ottomata created this task.Nov 25 2015, 6:46 PM
Ottomata raised the priority of this task from to Low.
Ottomata updated the task description. (Show Details)
Ottomata added projects: Analytics-Clusters, Analytics-Backlog.
Ottomata subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 25 2015, 6:46 PM
Ottomata merged a task: T121159: Upgrade Spark to 1.5.Dec 11 2015, 2:34 PM
Ottomata added subscribers: • ellery, StudiesWorld.
• ellery added a comment.Dec 14 2015, 6:02 PM

I just did another experiment with using caching while running Spark 1.5 from a jar in my homedir, and it worked just fine. I was probably making a coding error or was too impatient when I tested this earlier and was unsuccessful. Consequently, there is no rush on my end.

Milimetric raised the priority of this task from Low to High.Dec 14 2015, 6:47 PM
Milimetric moved this task from Incoming to Backlog on the Analytics-Backlog board.
Milimetric set Security to None.
• ggellerman edited projects, added Analytics; removed Analytics-Backlog.Jan 12 2016, 7:41 PM
Milimetric moved this task from Incoming to Event Platform on the Analytics board.Jan 12 2016, 7:41 PM
• Nuria closed subtask T125141: Uninstall Impala [3 pts] as Resolved.Feb 2 2016, 4:38 PM
Ottomata added a comment.Feb 16 2016, 3:25 PM

For reference:

http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_upgrade_5_to_latest.html

Ottomata edited projects, added Analytics-Kanban; removed Analytics.Feb 16 2016, 9:14 PM
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.
Ottomata moved this task from In Progress to Next Up on the Analytics-Kanban board.Feb 16 2016, 9:16 PM
Ottomata added a parent task: T116123: Make beeline easier to use as a Hive client {hawk}.Feb 17 2016, 5:34 PM
Milimetric renamed this task from Upgrade to CDH 5.5 to Upgrade to CDH 5.5 {hawk} [8 pts].Feb 17 2016, 5:41 PM
gerritbot subscribed.Feb 17 2016, 9:32 PM

Change 271355 had a related patch set uploaded (by Ottomata):
Update Cloudera package reprepro updates to CDH 5.5.2

https://gerrit.wikimedia.org/r/271355

gerritbot added a project: Patch-For-Review.Feb 17 2016, 9:32 PM
gerritbot added a comment.Feb 17 2016, 9:36 PM

Change 271355 merged by Ottomata:
Update Cloudera package reprepro updates to CDH 5.5.2

https://gerrit.wikimedia.org/r/271355

Milimetric renamed this task from Upgrade to CDH 5.5 {hawk} [8 pts] to Upgrade to CDH 5.5 {hawk}.Feb 22 2016, 9:07 PM
Milimetric set the point value for this task to 8.
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.Feb 23 2016, 2:39 PM
gerritbot added a comment.Feb 23 2016, 4:31 PM

Change 272753 had a related patch set uploaded (by Ottomata):
spark-core now depends on flume, mirror it from CDH to our apt

https://gerrit.wikimedia.org/r/272753

gerritbot added a comment.Feb 23 2016, 4:32 PM

Change 272753 merged by Ottomata:
spark-core now depends on flume, mirror it from CDH to our apt

https://gerrit.wikimedia.org/r/272753

gerritbot added a comment.Feb 23 2016, 4:35 PM

Change 272754 had a related patch set uploaded (by Ottomata):
Use proper flume package naems in reprepro updates

https://gerrit.wikimedia.org/r/272754

gerritbot added a comment.Feb 23 2016, 4:36 PM

Change 272754 merged by Ottomata:
Use proper flume package naems in reprepro updates

https://gerrit.wikimedia.org/r/272754

gerritbot added a comment.Feb 23 2016, 6:00 PM

Change 272779 had a related patch set uploaded (by Ottomata):
Add accidentally uncommitted hiera cdh users.yaml

https://gerrit.wikimedia.org/r/272779

gerritbot added a comment.Feb 23 2016, 6:00 PM

Change 272779 merged by Ottomata:
Add accidentally uncommitted hiera cdh users.yaml

https://gerrit.wikimedia.org/r/272779

gerritbot added a comment.Feb 23 2016, 6:33 PM

Change 272788 had a related patch set uploaded (by Ottomata):
Add analytics1015.yaml for analytics-admins

https://gerrit.wikimedia.org/r/272788

gerritbot added a comment.Feb 23 2016, 6:34 PM

Change 272788 merged by Ottomata:
Add analytics1015.yaml for analytics-admins

https://gerrit.wikimedia.org/r/272788

gerritbot added a comment.Feb 23 2016, 8:04 PM

Change 272808 had a related patch set uploaded (by Ottomata):
Increase analytics mysql meta innodb_buffer_pool_size to 1G, query_cache_size to 16M

https://gerrit.wikimedia.org/r/272808

gerritbot added a comment.Feb 23 2016, 8:05 PM

Change 272808 merged by Ottomata:
Increase analytics mysql meta innodb_buffer_pool_size to 1G, query_cache_size to 16M

https://gerrit.wikimedia.org/r/272808

gerritbot added a comment.Feb 23 2016, 8:30 PM

Change 272813 had a related patch set uploaded (by Ottomata):
Analytics MySQL Meta instance: 4G innodb_buffer_pool_size 4 instances

https://gerrit.wikimedia.org/r/272813

gerritbot added a comment.Feb 23 2016, 8:32 PM

Change 272813 merged by Ottomata:
Analytics MySQL Meta instance: 4G innodb_buffer_pool_size 4 instances

https://gerrit.wikimedia.org/r/272813

Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Feb 23 2016, 9:45 PM
gerritbot added a comment.Feb 23 2016, 10:44 PM

Change 272901 had a related patch set uploaded (by Ottomata):
Fix camus checker classpath in CDH 5.5

https://gerrit.wikimedia.org/r/272901

gerritbot added a comment.Feb 23 2016, 10:47 PM

Change 272901 merged by Joal:
Fix camus checker classpath in CDH 5.5

https://gerrit.wikimedia.org/r/272901

Ottomata claimed this task.Feb 24 2016, 4:31 PM
• Nuria closed this task as Resolved.Feb 26 2016, 5:23 PM
• Nuria closed subtask T127115: Upgrade analytics and beta project Analytics Clusters to CDH 5.5 [8 pts] as Resolved.Feb 26 2016, 5:26 PM
Ottomata mentioned this in T152714: CDH 5.10 upgrade.Feb 2 2017, 3:07 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL