Page MenuHomePhabricator
Create Task
Maniphest T229003

Roll restart all openjdk-8 jvms in Analytics
Closed, ResolvedPublic13 Estimated Story Points
Actions

Description

A new version of the openjdk-8 package is out, so we need another round of roll restarts of all the Analytics jvm-based daemons. This time we should create a spicerack cookbook for each cluster that can be automated.

  • Hadoop worker nodes
  • Hadoop master nodes
  • Hadoop coordinator
  • Hadoop worker nodes - test cluster
  • Hadoop master nodes - test cluster
  • Hadoop coordinator - test cluster
  • AQS nodes (aqs1004-1009)
  • Druid Private nodes (druid1001-3)
  • Druid Public nodes (druid1004-6)
  • Kafka Jumbo
  • Zookeeper conf100[4-6] and druid100[1-6]
  • /mnt/hdfs fuse mountpoints

Details

Show related patches Customize query in gerrit

Related Objects

Event Timeline

elukey triaged this task as Medium priority.Jul 25 2019, 2:29 PM
elukey created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 25 2019, 2:29 PM
gerritbot added a comment.Jul 25 2019, 2:31 PM

Change 525520 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] Add sre.hadoop.rolling-restart-workers cookbook

https://gerrit.wikimedia.org/r/525520

gerritbot added a project: Patch-For-Review.Jul 25 2019, 2:31 PM
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.Jul 25 2019, 3:06 PM
gerritbot added a comment.Jul 25 2019, 3:29 PM

Change 525520 merged by Elukey:
[operations/cookbooks@master] Add sre.hadoop.rolling-restart-workers cookbook

https://gerrit.wikimedia.org/r/525520

gerritbot added a comment.Jul 25 2019, 3:44 PM

Change 525573 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] sre.hadoop.rolling-restart-workers: fix bugs

https://gerrit.wikimedia.org/r/525573

gerritbot added a comment.Jul 25 2019, 3:48 PM

Change 525573 merged by Elukey:
[operations/cookbooks@master] sre.hadoop.rolling-restart-workers: fix bugs

https://gerrit.wikimedia.org/r/525573

gerritbot added a comment.Jul 25 2019, 3:58 PM

Change 525575 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] sre.hadoop.rolling-restart-workers.py: fix argparse defaults

https://gerrit.wikimedia.org/r/525575

gerritbot added a comment.Jul 25 2019, 4:03 PM

Change 525575 merged by Elukey:
[operations/cookbooks@master] sre.hadoop.rolling-restart-workers.py: fix argparse defaults

https://gerrit.wikimedia.org/r/525575

Maintenance_bot removed a project: Patch-For-Review.Jul 25 2019, 4:10 PM
elukey added a subscriber: Ottomata.Jul 25 2019, 4:18 PM

Dry run for the hadoop test cluster:

elukey@cumin1001:~$ sudo cookbook -d sre.hadoop.rolling-restart-workers.py test --yarn-nm-batch-size 1 --hdfs-dn-batch-size 1
DRY-RUN: Executing cookbook sre.hadoop.rolling-restart-workers with args: ['test', '--yarn-nm-batch-size', '1', '--hdfs-dn-batch-size', '1']
DRY-RUN: START - Cookbook sre.hadoop.rolling-restart-workers
DRY-RUN: Resolved CNAME record for icinga.wikimedia.org: icinga.wikimedia.org. 300 IN CNAME icinga1001.wikimedia.org.
DRY-RUN: Scheduling downtime on Icinga server icinga1001.wikimedia.org for hosts: analytics[1031-1038,1040].eqiad.wmnet
DRY-RUN: Executing commands ['icinga-downtime -h "analytics1031" -d 7200 -r "Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001"', 'icinga-downtime -h "analytics1032" -d 7200 -r "Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001"', 'icinga-downtime -h "analytics1033" -d 7200 -r "Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001"', 'icinga-downtime -h "analytics1034" -d 7200 -r "Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001"', 'icinga-downtime -h "analytics1035" -d 7200 -r "Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001"', 'icinga-downtime -h "analytics1036" -d 7200 -r "Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001"', 'icinga-downtime -h "analytics1037" -d 7200 -r "Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001"', 'icinga-downtime -h "analytics1038" -d 7200 -r "Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001"', 'icinga-downtime -h "analytics1040" -d 7200 -r "Roll restart of jvm daemons for openjdk upgrade. - elukey@cumin1001"'] on 1 hosts: icinga1001.wikimedia.org
DRY-RUN: Restarting Yarn Nodemanagers with batch size 1 and sleep 30.0..
DRY-RUN: Executing commands ['systemctl restart hadoop-yarn-nodemanager'] on 9 hosts: analytics[1031-1038,1040].eqiad.wmnet
DRY-RUN: Restarting HDFS Datanodes with batch size 1 and sleep 30.0..
DRY-RUN: Executing commands ['systemctl restart hadoop-hdfs-datanode'] on 9 hosts: analytics[1031-1038,1040].eqiad.wmnet
DRY-RUN: Restarting HDFS Journalnodes with batch size 1 and sleep 30.0..
DRY-RUN: Executing commands ['systemctl restart hadoop-hdfs-journalnode'] on 3 hosts: analytics[1031,1034,1038].eqiad.wmnet
DRY-RUN: Executing commands ['awk \'/^\\s*command_file=/{split($0, a, "="); print a[2] }\' /etc/icinga/icinga.cfg'] on 1 hosts: icinga1001.wikimedia.org
DRY-RUN: Executing commands ['echo -n "[1564071441] DEL_DOWNTIME_BY_HOST_NAME;analytics1031" > /var/lib/icinga/rw/icinga.cmd', 'echo -n "[1564071441] DEL_DOWNTIME_BY_HOST_NAME;analytics1032" > /var/lib/icinga/rw/icinga.cmd', 'echo -n "[1564071441] DEL_DOWNTIME_BY_HOST_NAME;analytics1033" > /var/lib/icinga/rw/icinga.cmd', 'echo -n "[1564071441] DEL_DOWNTIME_BY_HOST_NAME;analytics1034" > /var/lib/icinga/rw/icinga.cmd', 'echo -n "[1564071441] DEL_DOWNTIME_BY_HOST_NAME;analytics1035" > /var/lib/icinga/rw/icinga.cmd', 'echo -n "[1564071441] DEL_DOWNTIME_BY_HOST_NAME;analytics1036" > /var/lib/icinga/rw/icinga.cmd', 'echo -n "[1564071441] DEL_DOWNTIME_BY_HOST_NAME;analytics1037" > /var/lib/icinga/rw/icinga.cmd', 'echo -n "[1564071441] DEL_DOWNTIME_BY_HOST_NAME;analytics1038" > /var/lib/icinga/rw/icinga.cmd', 'echo -n "[1564071441] DEL_DOWNTIME_BY_HOST_NAME;analytics1040" > /var/lib/icinga/rw/icinga.cmd'] on 1 hosts: icinga1001.wikimedia.org
DRY-RUN: All jvm restarts completed!
DRY-RUN: END (PASS) - Cookbook sre.hadoop.rolling-restart-workers (exit_code=0)

@Ottomata --^

• fdans moved this task from Incoming to Operational Excellence on the Analytics board.Jul 25 2019, 4:47 PM
gerritbot added a comment.Jul 26 2019, 8:29 AM

Change 525775 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add more cumin aliases for Druid clusters

https://gerrit.wikimedia.org/r/525775

gerritbot added a project: Patch-For-Review.Jul 26 2019, 8:29 AM
gerritbot added a comment.Jul 26 2019, 8:43 AM

Change 525775 merged by Elukey:
[operations/puppet@production] Add more cumin aliases for Druid clusters

https://gerrit.wikimedia.org/r/525775

gerritbot added a comment.Jul 26 2019, 8:45 AM

Change 525776 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] Add sre.druid.roll-restart-workers.py

https://gerrit.wikimedia.org/r/525776

gerritbot added a comment.Jul 26 2019, 1:33 PM

Change 525804 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] Add sre.kafka.roll-restart-brokers.py

https://gerrit.wikimedia.org/r/525804

gerritbot added a comment.Jul 29 2019, 6:24 AM

Change 526014 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] sre.hadoop.roll-restart-workers.py: increase sleep time for HDFS

https://gerrit.wikimedia.org/r/526014

gerritbot added a comment.Jul 29 2019, 6:28 AM

Change 526014 merged by Elukey:
[operations/cookbooks@master] sre.hadoop.roll-restart-workers.py: increase sleep time for HDFS

https://gerrit.wikimedia.org/r/526014

elukey updated the task description. (Show Details)Jul 29 2019, 7:25 AM

All 54 Hadoop worker nodes roll restarted with the sre.hadoop.roll-restart-workers.py cookbook!

gerritbot added a comment.Jul 29 2019, 8:23 AM

Change 525776 merged by Elukey:
[operations/cookbooks@master] Add sre.druid.roll-restart-workers.py

https://gerrit.wikimedia.org/r/525776

gerritbot added a comment.Jul 29 2019, 9:06 AM

Change 526111 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] sre.hadoop.roll-restart-workers.py: ensure durability of the shell

https://gerrit.wikimedia.org/r/526111

gerritbot added a comment.Jul 29 2019, 9:14 AM

Change 526111 merged by Elukey:
[operations/cookbooks@master] sre.hadoop.roll-restart-workers.py: ensure durability of the shell

https://gerrit.wikimedia.org/r/526111

gerritbot added a comment.Jul 29 2019, 10:01 AM

Change 526122 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::druid::public::worker: add conftool config/scripts

https://gerrit.wikimedia.org/r/526122

gerritbot added a comment.Jul 29 2019, 10:03 AM

Change 526122 merged by Elukey:
[operations/puppet@production] role::druid::public::worker: add conftool config/scripts

https://gerrit.wikimedia.org/r/526122

elukey updated the task description. (Show Details)Jul 29 2019, 1:35 PM
gerritbot added a comment.Jul 29 2019, 3:19 PM

Change 525804 merged by Elukey:
[operations/cookbooks@master] Add sre.kafka.roll-restart-brokers.py

https://gerrit.wikimedia.org/r/525804

Maintenance_bot removed a project: Patch-For-Review.Jul 29 2019, 4:10 PM
gerritbot added a comment.Jul 30 2019, 7:16 AM

Change 526376 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] sre.kafka.roll-restart-brokers.py: improvements to the procedure

https://gerrit.wikimedia.org/r/526376

gerritbot added a project: Patch-For-Review.Jul 30 2019, 7:16 AM
gerritbot added a comment.Jul 30 2019, 3:16 PM

Change 526376 merged by Elukey:
[operations/cookbooks@master] sre.kafka.roll-restart-brokers.py: improvements to the procedure

https://gerrit.wikimedia.org/r/526376

Maintenance_bot removed a project: Patch-For-Review.Jul 30 2019, 4:11 PM
gerritbot added a comment.Jul 30 2019, 4:24 PM

Change 526472 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] sre.kafka.roll-restart-brokers: source /etc/profile.d/kafka.sh

https://gerrit.wikimedia.org/r/526472

gerritbot added a project: Patch-For-Review.Jul 30 2019, 4:24 PM
gerritbot added a comment.Jul 30 2019, 4:31 PM

Change 526472 merged by Elukey:
[operations/cookbooks@master] sre.kafka.roll-restart-brokers: source /etc/profile.d/kafka.sh

https://gerrit.wikimedia.org/r/526472

Maintenance_bot removed a project: Patch-For-Review.Jul 30 2019, 5:10 PM
elukey updated the task description. (Show Details)Jul 31 2019, 8:46 AM
gerritbot added a comment.Jul 31 2019, 8:54 AM

Change 526620 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add cumin aliases for Kafka Mirror Maker

https://gerrit.wikimedia.org/r/526620

gerritbot added a project: Patch-For-Review.Jul 31 2019, 8:54 AM
gerritbot added a comment.Jul 31 2019, 9:17 AM

Change 526624 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] Add sre.kafka.roll-restart-mirror-maker.py

https://gerrit.wikimedia.org/r/526624

gerritbot added a comment.Jul 31 2019, 9:48 AM

Change 526628 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] Add sre.zookeeper.roll-restart-zookeeper.py

https://gerrit.wikimedia.org/r/526628

gerritbot added a comment.Jul 31 2019, 9:49 AM

Change 526620 merged by Elukey:
[operations/puppet@production] Add cumin aliases for Kafka Mirror Maker and zookeeper

https://gerrit.wikimedia.org/r/526620

gerritbot added a comment.Jul 31 2019, 9:51 AM

Change 526624 merged by Elukey:
[operations/cookbooks@master] Add sre.kafka.roll-restart-mirror-maker.py

https://gerrit.wikimedia.org/r/526624

elukey updated the task description. (Show Details)Jul 31 2019, 10:10 AM
gerritbot added a comment.Jul 31 2019, 12:50 PM

Change 526628 merged by Elukey:
[operations/cookbooks@master] Add sre.zookeeper.roll-restart-zookeeper.py

https://gerrit.wikimedia.org/r/526628

Maintenance_bot removed a project: Patch-For-Review.Jul 31 2019, 1:10 PM
gerritbot added a comment.Jul 31 2019, 1:11 PM

Change 526665 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] Improvements to kafka and zookeeper cookbooks

https://gerrit.wikimedia.org/r/526665

gerritbot added a project: Patch-For-Review.Jul 31 2019, 1:11 PM
gerritbot added a comment.Jul 31 2019, 1:28 PM

Change 526665 merged by Elukey:
[operations/cookbooks@master] Improvements to kafka and zookeeper cookbooks

https://gerrit.wikimedia.org/r/526665

Maintenance_bot removed a project: Patch-For-Review.Jul 31 2019, 2:11 PM
elukey updated the task description. (Show Details)Jul 31 2019, 2:51 PM
elukey updated the task description. (Show Details)Jul 31 2019, 2:55 PM
elukey updated the task description. (Show Details)
gerritbot added a comment.Aug 5 2019, 1:09 PM

Change 528127 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add PartOf configuration in the Kafka mirror systemd units

https://gerrit.wikimedia.org/r/528127

gerritbot added a project: Patch-For-Review.Aug 5 2019, 1:09 PM

Change 528127 merged by Elukey:
[operations/puppet@production] Add PartOf configuration in the Kafka mirror systemd units

https://gerrit.wikimedia.org/r/528127

elukey updated the task description. (Show Details)Aug 6 2019, 12:53 PM
elukey changed the point value for this task from 0 to 13.
elukey moved this task from In Progress to Done on the Analytics-Kanban board.
Maintenance_bot removed a project: Patch-For-Review.Aug 6 2019, 1:10 PM
• Nuria closed this task as Resolved.Aug 21 2019, 5:21 PM
elukey mentioned this in rCCKBa3d23950fbd2: Add sre.hadoop.rolling-restart-workers cookbook.Dec 14 2022, 3:24 PM
elukey mentioned this in rCCKB946590e87806: sre.hadoop.rolling-restart-workers: fix bugs.
elukey mentioned this in rCCKB5accd131e6a2: sre.hadoop.rolling-restart-workers.py: fix argparse defaults.
elukey mentioned this in rCCKB2100173d749f: sre.hadoop.roll-restart-workers.py: increase sleep time for HDFS.
elukey mentioned this in rCCKB2f2299732c64: Add sre.druid.roll-restart-workers.py.
elukey mentioned this in rCCKB9df7ac19bc56: sre.hadoop.roll-restart-workers.py: ensure durability of the shell.
elukey mentioned this in rCCKB5ef097351fbe: Add sre.kafka.roll-restart-brokers.py.
elukey mentioned this in rCCKB57e5fc916013: sre.kafka.roll-restart-brokers.py: improvements to the procedure.
elukey mentioned this in rCCKBb139a500479e: sre.kafka.roll-restart-brokers: source /etc/profile.d/kafka.sh.
elukey mentioned this in rCCKB35aea4ca8e84: Add sre.kafka.roll-restart-mirror-maker.py.
elukey mentioned this in rCCKB80169bf840d0: Add sre.zookeeper.roll-restart-zookeeper.py.
elukey mentioned this in rCCKB2f3fe6e142d8: Improvements to kafka and zookeeper cookbooks.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits