Page MenuHomePhabricator
Create Task
Maniphest T429114

sre.mysql.depool failing with downtime for parsercache
Closed, ResolvedPublic
Actions

Description

When running sre.mysql.depool with --downtime against a parsercache instance (e.g. pc2021), the cookbook fails with:

TypeError: 'int' object is not subscriptable
[05:49:11] marostegui@cumin1003:~$ sudo cookbook sre.mysql.depool  -r "Migration to 10.11.18 T428861" --downtime 2 pc2021
Acquired lock for key /spicerack/locks/cookbooks/sre.mysql.depool:pc2021: {'concurrency': 1, 'created': '2026-06-15 05:49:30.895769', 'owner': 'marostegui@cumin1003 [3955259]', 'ttl': 60}
START - Cookbook sre.mysql.depool depool pc2021: Migration to 10.11.18 T428861
Using parsercache cookbook
The whole 'pc1' section will be depooled
Exception raised while parsing arguments for cookbook sre.mysql.parsercache:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 359, in _safe_call
    ret_value = func(*args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/argparse.py", line 1874, in parse_args
    args, argv = self.parse_known_args(args, namespace)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/argparse.py", line 1907, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/argparse.py", line 1950, in _parse_known_args
    option_tuple = self._parse_optional(arg_string)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/argparse.py", line 2240, in _parse_optional
    if not arg_string[0] in self.prefix_chars:
           ~~~~~~~~~~^^^
TypeError: 'int' object is not subscriptable
Released lock for key /spicerack/locks/cookbooks/sre.mysql.depool:pc2021: {'concurrency': 1, 'created': '2026-06-15 05:49:30.895769', 'owner': 'marostegui@cumin1003 [3955259]', 'ttl': 60}
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc2021: Migration to 10.11.18 T428861

I believe the fix is just:

458 -            cmar.extend(["--downtime", self.downtime])
458 +            cmar.extend(["--downtime", str(self.downtime)])

However, please take a look

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
T429114: sre.mysql.depool failing with downtime for parsercacheCWilliamsoperations/cookbooksmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Mon, Jun 15, 5:56 AM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptMon, Jun 15, 5:56 AM
Marostegui triaged this task as Medium priority.Mon, Jun 15, 5:56 AM
Marostegui moved this task from Triage to Ready on the DBA board.

We probably may want to also update: https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting#Depooling_a_replica

Marostegui added a parent task: T426318: Add support for automatic downtime when depooling instances using sre.mysql.depool.Mon, Jun 15, 8:18 AM
CWilliams-WMF added a comment.Mon, Jun 15, 9:01 AM

@Marostegui that takes an int

diff --git a/cookbooks/sre/mysql/parsercache.py b/cookbooks/sre/mysql/parsercache.py
index b44697e..2215733 100644
--- a/cookbooks/sre/mysql/parsercache.py
+++ b/cookbooks/sre/mysql/parsercache.py
@@ -53,7 +53,8 @@ def argument_parser() -> ArgumentParser:
     pool.add_argument("--skip-icinga-checks", action="store_true", help="Skip checks before pooling")
 
     depool = subs.add_parser("depool", help="Depool")
-    depool.add_argument("--downtime-hours", type=int, help="Create downtime (default: no)", default=0)
+    depool.add_argument("--downtime", "--downtime-hours", dest="downtime_hours", type=int,
+                        help="Create downtime (default: no)", default=0)
     return ap
CWilliams-WMF added a comment.Mon, Jun 15, 9:10 AM

@Marostegui please can you add the full command that you executed?

Marostegui added a comment.Mon, Jun 15, 9:11 AM
In T429114#12017684, @CWilliams-WMF wrote:

@Marostegui please can you add the full command that you executed?

It is there on the task description :)

CWilliams-WMF renamed this task from sre.mysql.depool failing with downtime to sre.mysql.depool failing with downtime for parsercache.Mon, Jun 15, 9:29 AM
gerritbot added a comment.Mon, Jun 15, 9:32 AM

Change #1302098 had a related patch set uploaded (by CWilliams; author: CWilliams):

[operations/cookbooks@master] T429114: sre.mysql.depool failing with downtime for parsercache

https://gerrit.wikimedia.org/r/1302098

gerritbot added a project: Patch-For-Review.Mon, Jun 15, 9:32 AM
CWilliams-WMF changed the task status from Open to In Progress.Mon, Jun 15, 9:32 AM

It is there on the task description :)

So it is ... line 1 was processed whilst the coffee kicked in :)

gerritbot added a comment.Mon, Jun 15, 10:19 AM

Change #1302098 merged by jenkins-bot:

[operations/cookbooks@master] T429114: sre.mysql.depool failing with downtime for parsercache

https://gerrit.wikimedia.org/r/1302098

CWilliams-WMF closed this task as Resolved.Mon, Jun 15, 10:23 AM
Marostegui added a comment.Mon, Jun 15, 10:26 AM

Thanks! \o/

Maintenance_bot removed a project: Patch-For-Review.Mon, Jun 15, 10:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits