Page MenuHomePhabricator
Create Task
Maniphest T192636

Upgrade Druid nodes (1001->1006) to Debian Stretch
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

Two things would be needed to allow this:

  1. zookeeper upgraded to 3.4.9 (not strictly necessary but safer since the jessie version is 3.4.5).
  2. druid upgraded to 0.10+ (in order to have openjdk-8 only) - T164008

The procedure then should be really simple:

  1. depool (if needed) one host at the time
  2. reimage it
  3. check that everything is fine and proceed with another one

Druid segments that are cached would need to be restored, so leaving enough time between one host and the other would probably be better.

  • druid1001
  • druid1002
  • druid1003
  • druid1004
  • druid1005
  • druid1006

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
role::druid::public: set zookeeper version to 3.4.9-3operations/puppetproduction+1 -3
Override druid1005's zookeeper settingsoperations/puppetproduction+1 -0
Override zookeeper version for druid1004operations/puppetproduction+1 -0
role::druid::analytics::worker: set zookeeper version to 3.4.9-3operations/puppetproduction+1 -3
role::druid::analytics|public::worker: remove hadoop-cdh client depoperations/puppetproduction+0 -2
Set zookeeper verion for druid1002operations/puppetproduction+1 -0
Override druid1003's zookeeper version after reimageoperations/puppetproduction+1 -0
Set druid* PXE boot to Debian Stretchoperations/puppetproduction+0 -12
profile::druid::*: move prometheus jmx exp's config to /etc/prometheusoperations/puppetproduction+15 -10
Customize query in gerrit

Related Objects
Search...

Event Timeline

elukey created this task.Apr 20 2018, 10:08 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 20 2018, 10:08 AM
elukey mentioned this in T192642: Upgrade Analytics infrastructure to Debian Stretch.Apr 20 2018, 11:23 AM
elukey added a parent task: T192642: Upgrade Analytics infrastructure to Debian Stretch.
• fdans triaged this task as Medium priority.Apr 23 2018, 3:54 PM
• fdans raised the priority of this task from Medium to High.
• fdans lowered the priority of this task from High to Medium.
• fdans moved this task from Incoming to Operational Excellence on the Analytics board.
elukey moved this task from Backlog to Analytics Backlog on the User-Elukey board.Apr 24 2018, 7:28 AM
elukey added a comment.May 2 2018, 2:29 PM

All druid nodes are running Druid 0.10 and zookeeper 3.4.9, we can do the work anytime.

gerritbot subscribed.May 23 2018, 7:13 AM

Change 434634 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::druid::*: move prometheus jmx exp's config to /etc/prometheus

https://gerrit.wikimedia.org/r/434634

gerritbot added a project: Patch-For-Review.May 23 2018, 7:13 AM
gerritbot added a comment.May 23 2018, 7:21 AM

Change 434634 merged by Elukey:
[operations/puppet@production] profile::druid::*: move prometheus jmx exp's config to /etc/prometheus

https://gerrit.wikimedia.org/r/434634

elukey added a comment.May 23 2018, 7:40 AM

I am upgrading the Druid labs cluster to Stretch as test, and two things came up:

  1. the druid debs were missing from stretch-wikimedia, uploaded them.
  2. as it happened with Hadoop, the dependency between the prometheus agent saving config files under /etc/druid and the fact that druid packages are the ones creating the directory created some race conditions during the first puppet run. I moved the config files to /etc/prometheus and explicitly required the prometheus jmx agent to be installed before druid, in order to avoid the race condition completely. Prod will pick up the change during the next restart/reimage/upgrade/etc..
elukey added a project: Analytics-Kanban.May 23 2018, 7:49 AM
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.
elukey moved this task from Analytics Backlog to In Progress on the User-Elukey board.May 24 2018, 6:30 AM
gerritbot added a comment.May 24 2018, 6:34 AM

Change 434870 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Set druid* PXE boot to Debian Stretch

https://gerrit.wikimedia.org/r/434870

gerritbot added a comment.May 24 2018, 6:35 AM

Change 434870 merged by Elukey:
[operations/puppet@production] Set druid* PXE boot to Debian Stretch

https://gerrit.wikimedia.org/r/434870

ops-monitoring-bot subscribed.May 28 2018, 6:57 AM

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['druid1003.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805280657_elukey_19222.log.

ops-monitoring-bot added a comment.May 28 2018, 7:12 AM

Completed auto-reimage of hosts:

['druid1003.eqiad.wmnet']

Of which those FAILED:

['druid1003.eqiad.wmnet']
gerritbot added a comment.May 28 2018, 7:19 AM

Change 435727 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Override druid1003's zookeeper version after reimage

https://gerrit.wikimedia.org/r/435727

gerritbot added a comment.May 28 2018, 7:20 AM

Change 435727 merged by Elukey:
[operations/puppet@production] Override druid1003's zookeeper version after reimage

https://gerrit.wikimedia.org/r/435727

elukey updated the task description. (Show Details)May 28 2018, 8:01 AM
ops-monitoring-bot added a comment.May 29 2018, 7:49 AM

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['druid1002.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805290749_elukey_1188.log.

gerritbot added a comment.May 29 2018, 7:52 AM

Change 435970 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Set zookeeper verion for druid1002

https://gerrit.wikimedia.org/r/435970

gerritbot added a comment.May 29 2018, 7:52 AM

Change 435970 merged by Elukey:
[operations/puppet@production] Set zookeeper verion for druid1002

https://gerrit.wikimedia.org/r/435970

ops-monitoring-bot added a comment.May 29 2018, 8:15 AM

Completed auto-reimage of hosts:

['druid1002.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.May 29 2018, 10:25 AM

Change 435983 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::druid::analytics|public::worker: remove hadoop-cdh client dep

https://gerrit.wikimedia.org/r/435983

gerritbot added a comment.May 29 2018, 10:26 AM

Change 435983 merged by Elukey:
[operations/puppet@production] role::druid::analytics|public::worker: remove hadoop-cdh client dep

https://gerrit.wikimedia.org/r/435983

elukey updated the task description. (Show Details)May 29 2018, 1:43 PM
ops-monitoring-bot added a comment.May 30 2018, 6:17 AM

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['druid1001.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805300617_elukey_16317.log.

gerritbot added a comment.May 30 2018, 6:20 AM

Change 436220 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::druid::analytics::worker: set zookeeper version to 3.4.9-3

https://gerrit.wikimedia.org/r/436220

gerritbot added a comment.May 30 2018, 6:20 AM

Change 436220 merged by Elukey:
[operations/puppet@production] role::druid::analytics::worker: set zookeeper version to 3.4.9-3

https://gerrit.wikimedia.org/r/436220

ops-monitoring-bot added a comment.May 30 2018, 6:49 AM

Completed auto-reimage of hosts:

['druid1001.eqiad.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.May 30 2018, 8:46 AM

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['druid1004.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805300846_elukey_15355.log.

gerritbot added a comment.May 30 2018, 8:47 AM

Change 436236 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Override zookeeper version for druid1004

https://gerrit.wikimedia.org/r/436236

gerritbot added a comment.May 30 2018, 8:48 AM

Change 436236 merged by Elukey:
[operations/puppet@production] Override zookeeper version for druid1004

https://gerrit.wikimedia.org/r/436236

ops-monitoring-bot added a comment.May 30 2018, 9:10 AM

Completed auto-reimage of hosts:

['druid1004.eqiad.wmnet']

and were ALL successful.

elukey moved this task from In Progress to Ready to Deploy on the Analytics-Kanban board.May 30 2018, 1:27 PM
ops-monitoring-bot added a comment.May 31 2018, 5:59 AM

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['druid1005.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805310559_elukey_21502.log.

ops-monitoring-bot added a comment.May 31 2018, 6:13 AM

Completed auto-reimage of hosts:

['druid1005.eqiad.wmnet']

Of which those FAILED:

['druid1005.eqiad.wmnet']
gerritbot added a comment.May 31 2018, 6:17 AM

Change 436450 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Override druid1005's zookeeper settings

https://gerrit.wikimedia.org/r/436450

gerritbot added a comment.May 31 2018, 6:18 AM

Change 436450 merged by Elukey:
[operations/puppet@production] Override druid1005's zookeeper settings

https://gerrit.wikimedia.org/r/436450

gerritbot added a comment.May 31 2018, 9:31 AM

Change 436487 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::druid::public: set zookeeper version to 3.4.9-3

https://gerrit.wikimedia.org/r/436487

gerritbot added a comment.May 31 2018, 9:32 AM

Change 436487 merged by Elukey:
[operations/puppet@production] role::druid::public: set zookeeper version to 3.4.9-3

https://gerrit.wikimedia.org/r/436487

ops-monitoring-bot added a comment.May 31 2018, 9:34 AM

Script wmf-auto-reimage was launched by elukey on neodymium.eqiad.wmnet for hosts:

['druid1006.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201805310934_elukey_9600.log.

ops-monitoring-bot added a comment.May 31 2018, 10:01 AM

Completed auto-reimage of hosts:

['druid1006.eqiad.wmnet']

and were ALL successful.

elukey set the point value for this task to 8.May 31 2018, 10:04 AM
elukey moved this task from Ready to Deploy to Done on the Analytics-Kanban board.
elukey moved this task from In Progress to Done on the User-Elukey board.Jun 4 2018, 8:13 AM
• Nuria closed this task as Resolved.Jun 11 2018, 11:05 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits