Page MenuHomePhabricator
Create Task
Maniphest T269925

Update Spicerack cookbooks to follow the new class API conventions
Closed, ResolvedPublic
Actions

Description

Very recently a new Class API was added to Spicerack: https://doc.wikimedia.org/spicerack/master/introduction.html#class-interface

This means that all our cookbook would need to be updated to follow the new style, and I think this could be a good occasion to code and get familiar with spicerack/cookbooks for @razzi!

Cookbooks to update:

  • sre.aqs.roll-restart
  • sre.cassandra.roll-restart
  • sre.druid.roll-restart-workers
  • sre.hadoop.change-distro-from-cdh
  • sre.hadoop.init-hadoop-workers
  • sre.hadoop.reboot-workers
  • sre.hadoop.roll-restart-masters
  • sre.hadoop.roll-restart-workers
  • sre.hadoop.stop-cluster
  • sre.kafka.roll-restart-brokers
  • sre.kafka.roll-restart-mirror-maker
  • sre.presto.roll-restart-workers
  • sre.zookeeper.roll-restart-zookeeper

This work is not urgent but it could be a good alternative activity in between more ops-focused tasks.

Some examples:

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

elukey created this task.Dec 11 2020, 10:33 AM
Volans awarded a token.Dec 11 2020, 10:34 AM
elukey updated the task description. (Show Details)Dec 11 2020, 10:34 AM
elukey updated the task description. (Show Details)
gerritbot added a comment.Dec 11 2020, 10:54 AM

Change 648172 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] sre.hadoop.stop-cluster.py: move to class API

https://gerrit.wikimedia.org/r/648172

gerritbot added a project: Patch-For-Review.Dec 11 2020, 10:54 AM
elukey updated the task description. (Show Details)Dec 11 2020, 11:20 AM
gerritbot added a comment.Dec 11 2020, 1:53 PM

Change 648172 merged by Elukey:
[operations/cookbooks@master] sre.hadoop.stop-cluster.py: move to class API

https://gerrit.wikimedia.org/r/648172

Maintenance_bot removed a project: Patch-For-Review.Dec 11 2020, 2:10 PM
gerritbot added a comment.Dec 15 2020, 2:40 PM

Change 649653 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] sre.hadoop.change-distro-from-cdh: move to class API

https://gerrit.wikimedia.org/r/649653

gerritbot added a project: Patch-For-Review.Dec 15 2020, 2:40 PM
gerritbot added a comment.Dec 16 2020, 3:28 PM

Change 649653 merged by Elukey:
[operations/cookbooks@master] sre.hadoop.change-distro-from-cdh: move to class API

https://gerrit.wikimedia.org/r/649653

Maintenance_bot removed a project: Patch-For-Review.Dec 16 2020, 4:10 PM
elukey updated the task description. (Show Details)Dec 16 2020, 6:21 PM
gerritbot added a comment.Jan 14 2021, 4:52 PM

Change 656212 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] sre.hadoop.reboot-workers: move to the new class API

https://gerrit.wikimedia.org/r/656212

gerritbot added a project: Patch-For-Review.Jan 14 2021, 4:52 PM
• razzi assigned this task to elukey.Jan 14 2021, 5:57 PM
• razzi added a project: Analytics-Kanban.
• razzi moved this task from Incoming to Operational Excellence on the Analytics board.Jan 14 2021, 6:07 PM
• razzi edited projects, added Analytics-Clusters; removed Analytics.
gerritbot added a comment.Jan 15 2021, 5:22 PM

Change 656212 merged by Elukey:
[operations/cookbooks@master] sre.hadoop.reboot-workers: move to the new class API

https://gerrit.wikimedia.org/r/656212

gerritbot added a comment.Jan 18 2021, 4:48 PM

Change 656952 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] sre.hadoop.init-hadoop-workers: move to Class API

https://gerrit.wikimedia.org/r/656952

gerritbot added a comment.Jan 18 2021, 5:22 PM

Change 656952 merged by Elukey:
[operations/cookbooks@master] sre.hadoop.init-hadoop-workers: move to Class API

https://gerrit.wikimedia.org/r/656952

gerritbot added a comment.Jan 19 2021, 8:31 AM

Change 657049 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] sre.druid.roll-restart-workers: move to class API

https://gerrit.wikimedia.org/r/657049

elukey updated the task description. (Show Details)Jan 19 2021, 8:33 AM
gerritbot added a comment.Jan 19 2021, 9:36 AM

Change 657049 merged by jenkins-bot:
[operations/cookbooks@master] sre.druid.roll-restart-workers: move to class API

https://gerrit.wikimedia.org/r/657049

Ottomata moved this task from Backlog to Q3 2020/2021 on the Analytics-Clusters board.Feb 8 2021, 4:41 PM
gerritbot added a comment.Feb 9 2021, 7:30 PM

Change 663033 had a related patch set uploaded (by Razzi; owner: Razzi):
[operations/cookbooks@master] sre.druid.roll-restart-workers: properly pass commands list

https://gerrit.wikimedia.org/r/663033

gerritbot added a comment.Feb 9 2021, 7:33 PM

Change 663033 merged by Razzi:
[operations/cookbooks@master] sre.druid.roll-restart-workers: properly pass commands list

https://gerrit.wikimedia.org/r/663033

gerritbot added a comment.Feb 12 2021, 5:08 PM

Change 663863 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/cookbooks@master] sre.presto.roll-restart-workers: move to class api

https://gerrit.wikimedia.org/r/663863

gerritbot added a comment.Feb 17 2021, 5:39 PM

Change 663863 merged by jenkins-bot:
[operations/cookbooks@master] sre.presto.roll-restart-workers: move to class api

https://gerrit.wikimedia.org/r/663863

elukey updated the task description. (Show Details)Feb 17 2021, 5:43 PM
Ottomata moved this task from Q3 2020/2021 to Q4 2020/2021 on the Analytics-Clusters board.Mar 15 2021, 3:44 PM
elukey removed elukey as the assignee of this task.Jun 21 2021, 3:48 PM
Ottomata assigned this task to BTullis.Jul 12 2021, 3:41 PM
gerritbot added a comment.Jul 13 2021, 9:38 AM

Change 704294 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/cookbooks@master] Update AQS Roll Restart cookbook to use new style

https://gerrit.wikimedia.org/r/704294

BTullis added a comment.Jul 13 2021, 9:44 AM

@elukey Can I ask, what would you consider to be the best way to test this change locally, before pushing to gerrit?

gerritbot added a comment.Jul 13 2021, 12:07 PM

Change 704294 merged by jenkins-bot:

[operations/cookbooks@master] Update AQS Roll Restart cookbook to use new style

https://gerrit.wikimedia.org/r/704294

BTullis added a comment.Jul 13 2021, 2:07 PM

My dry-run fails. The error seems to be here:

File "/srv/deployment/spicerack/cookbooks/sre/aqs/roll-restart.py", line 73, in run
  name=r'(?!' + self.aqs_canary.hosts[0] + ').*')

Also I notice that the help text and the actual command needed are different.

The help text says:

Usage example:
        cookbook.sre.aqs.roll_restart --cluster aqs

But when it comes to running it, the cluster argument is positional, therefore the command required is:

cookbook -d sre.aqs.roll-restart aqs
BTullis added a comment.Jul 13 2021, 2:18 PM

Here is the full traceback.

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 234, in run
    raw_ret = runner.run()
  File "/srv/deployment/spicerack/cookbooks/sre/aqs/roll-restart.py", line 73, in run
    name=r'(?!' + self.aqs_canary.hosts[0] + ').*')
  File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 318, in query_confctl
    hosts_conftool = [obj.name for obj in conftool.get(**tags)]
  File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 318, in <listcomp>
    hosts_conftool = [obj.name for obj in conftool.get(**tags)]
  File "/usr/lib/python3/dist-packages/spicerack/confctl.py", line 118, in get
    for obj in self._select(tags):
  File "/usr/lib/python3/dist-packages/spicerack/confctl.py", line 82, in _select
    selectors[tag] = re.compile("^{}$".format(expr))
  File "/usr/lib/python3.7/re.py", line 234, in compile
    return _compile(pattern, flags)
  File "/usr/lib/python3.7/re.py", line 286, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/usr/lib/python3.7/sre_compile.py", line 764, in compile
    p = sre_parse.parse(p, flags)
  File "/usr/lib/python3.7/sre_parse.py", line 930, in parse
    p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, 0)
  File "/usr/lib/python3.7/sre_parse.py", line 426, in _parse_sub
    not nested and not items))
  File "/usr/lib/python3.7/sre_parse.py", line 580, in _parse
    raise source.error(msg, len(this) + 1 + len(that))
re.error: bad character range 4-1 at position 8
DRY-RUN: END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) for AQS aqs[1004-1015].eqiad.wmnet cluster: Roll restart of all AQS's nodejs daemons. - btullis@cumin1001
elukey added a comment.Jul 13 2021, 2:34 PM

@BTullis good finding! Can you send a follow up patch to update the argparse settings?

BTullis moved this task from Next Up to In Progress on the Analytics-Kanban board.Jul 13 2021, 2:40 PM
gerritbot added a comment.Jul 13 2021, 2:44 PM

Change 704347 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/cookbooks@master] Fix the sre.aqs.roll-restart cookbook

https://gerrit.wikimedia.org/r/704347

gerritbot added a comment.Jul 13 2021, 4:05 PM

Change 704347 merged by jenkins-bot:

[operations/cookbooks@master] Fix the sre.aqs.roll-restart cookbook

https://gerrit.wikimedia.org/r/704347

gerritbot added a comment.Jul 14 2021, 8:49 AM

Change 704500 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/cookbooks@master] Update sre.hadoop.roll-restart-masters cookbook

https://gerrit.wikimedia.org/r/704500

gerritbot added a comment.Jul 14 2021, 8:49 AM

Change 704501 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/cookbooks@master] Update sre.hadoop.roll-restart-workers cookbook

https://gerrit.wikimedia.org/r/704501

BTullis added a comment.Jul 14 2021, 9:03 AM

Dry run of sre.aqs.roll-restart was successful this time.

DRY-RUN: END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - btullis@cumin1001

Marking that cookbook as complete.

BTullis triaged this task as Medium priority.Jul 14 2021, 9:03 AM
BTullis updated the task description. (Show Details)
gerritbot added a comment.Jul 14 2021, 2:02 PM

Change 704500 merged by jenkins-bot:

[operations/cookbooks@master] Update sre.hadoop.roll-restart-masters cookbook

https://gerrit.wikimedia.org/r/704500

gerritbot added a comment.Jul 14 2021, 2:02 PM

Change 704501 merged by jenkins-bot:

[operations/cookbooks@master] Update sre.hadoop.roll-restart-workers cookbook

https://gerrit.wikimedia.org/r/704501

BTullis updated the task description. (Show Details)Jul 15 2021, 12:00 PM
BTullis added a comment.Jul 15 2021, 12:08 PM

Dry run succeeded for hadoop masters and workers cookbooks.

gerritbot added a comment.Jul 16 2021, 9:59 AM

Change 704932 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/cookbooks@master] Update sre.kafka.roll-restart cookbooks to new API

https://gerrit.wikimedia.org/r/704932

gerritbot added a comment.Jul 16 2021, 10:35 AM

Change 704937 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/cookbooks@master] Update sre.zookeeper.roll-restart to use new spicerack API

https://gerrit.wikimedia.org/r/704937

gerritbot added a comment.Jul 16 2021, 11:23 AM

Change 704937 merged by jenkins-bot:

[operations/cookbooks@master] Update sre.zookeeper.roll-restart to use new spicerack API

https://gerrit.wikimedia.org/r/704937

BTullis added a comment.Jul 16 2021, 11:46 AM

Dry-run for sre.zookeeper.roll-restart-zookeeper succeeded.

BTullis updated the task description. (Show Details)Jul 16 2021, 11:46 AM
BTullis moved this task from In Progress to In Code Review on the Analytics-Kanban board.Jul 19 2021, 9:43 AM
BTullis updated the task description. (Show Details)Jul 21 2021, 12:31 PM
gerritbot added a comment.Jul 21 2021, 12:32 PM

Change 705869 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/cookbooks@master] Update sre.cassandra.roll-restart cookbook to use new spicerack API

https://gerrit.wikimedia.org/r/705869

elukey moved this task from Q4 2020/2021 to Q1 2021/2022 on the Analytics-Clusters board.Jul 21 2021, 4:59 PM
gerritbot added a comment.Jul 27 2021, 5:19 PM

Change 705869 merged by jenkins-bot:

[operations/cookbooks@master] Update sre.cassandra.roll-restart cookbook to use new spicerack API

https://gerrit.wikimedia.org/r/705869

gerritbot added a comment.Aug 2 2021, 4:42 PM

Change 704932 merged by Btullis:

[operations/cookbooks@master] Update sre.kafka.roll-restart cookbooks to new API

https://gerrit.wikimedia.org/r/704932

BTullis updated the task description. (Show Details)Aug 2 2021, 5:01 PM

Dry run succeeded on the kafka cookbooks, which were the only outstanding change to be tested.

BTullis closed this task as Resolved.Aug 2 2021, 5:02 PM
elukey mentioned this in rCCKB258b853099da: sre.hadoop.stop-cluster.py: move to class API.Dec 14 2022, 3:27 PM
elukey mentioned this in rCCKBe77a439195dc: sre.hadoop.change-distro-from-cdh: move to class API.
elukey mentioned this in rCCKB8dd019f3f4be: sre.hadoop.reboot-workers: move to the new class API.
elukey mentioned this in rCCKBcfe8befa1a79: sre.hadoop.init-hadoop-workers: move to Class API.
elukey mentioned this in rCCKBafc647bccbea: sre.druid.roll-restart-workers: move to class API.
• razzi mentioned this in rCCKB838744efaa42: sre.druid.roll-restart-workers: properly pass commands list.Dec 14 2022, 3:27 PM
elukey mentioned this in rCCKBd8cce389aaf7: sre.presto.roll-restart-workers: move to class api.Dec 14 2022, 3:27 PM
BTullis mentioned this in rCCKBc476a6ebe36d: Update AQS Roll Restart cookbook to use new style.Dec 14 2022, 3:28 PM
BTullis mentioned this in rCCKB63260753df7a: Fix the sre.aqs.roll-restart cookbook.
BTullis mentioned this in rCCKB171c4d0f7aa3: Update sre.hadoop.roll-restart-masters cookbook.
BTullis mentioned this in rCCKBcd59859ca611: Update sre.hadoop.roll-restart-workers cookbook.
BTullis mentioned this in rCCKB45c2634d3bf2: Update sre.zookeeper.roll-restart to use new spicerack API.
BTullis mentioned this in rCCKBc253e8a0da01: Update sre.cassandra.roll-restart cookbook to use new spicerack API.
BTullis mentioned this in rCCKB0ea6bff14b4a: Update sre.kafka.roll-restart cookbooks to new API.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits