Context
We encountered the following exception while running 00-downtime-db-readonly-checksin dry run mode.
[]: Exception raised while executing cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 265, in _run
raw_ret = runner.run()
^^^^^^^^^^^^
File "/srv/deployment/spicerack/cookbooks/sre/switchdc/mediawiki/__init__.py", line 145, in run
ret = self.action()
^^^^^^^^^^^^^
File "/srv/deployment/spicerack/cookbooks/sre/switchdc/mediawiki/00-downtime-db-readonly-checks.py", line 19, in action
hosts = mysql.get_core_dbs(replication_role="master")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/spicerack/mysql.py", line 926, in get_core_dbs
raise MysqlError(f"Matched {len(mysql_hosts)} masters, expected {dc_multipler * section_multiplier}")
spicerack.mysql.MysqlError: Matched 24 masters, expected 22Some investigation (thanks to @Scott_French) suggests that the assembled cumin query returned more primary hosts than expected. The math in dc_multiplier = 2 and section_multiplier=11 suggests an expected 22 sections while we actually match for 24.
~/puppet$ for db in db1163 db1181 db1189 db1193 db1201 db1210 db1220 db1222 db1244 db1258 es1035 es1038; do echo $db $(grep shard hieradata/hosts/$db.yaml); done db1163 mariadb::shard: 's1' db1181 mariadb::shard: 's7' db1189 mariadb::shard: 's3' db1193 mariadb::shard: 's8' db1201 mariadb::shard: 's6' db1210 mariadb::shard: 's5' db1220 mariadb::shard: 'x1' db1222 mariadb::shard: 's2' db1244 mariadb::shard: 's4' db1258 mariadb::shard: 'x3' es1035 mariadb::shard: 'es7' es1038 mariadb::shard: 'es6'
This appears to be related to the recent work to split s8 into a core cluster and a dedicated cluster for term store (T351820) which resulted in a new section added just after the last switchover in March.
Next steps
To resolve this we’d need to update the CORE_SECTIONS in spicerack to reflect the additional section (x3) and release a new spicerack library version.
This will also resolve the second exception raised when calling mysql.verify_core_masters_readonly(self.dc_to, True) which checks for expected sections in a single dc.
DRY-RUN: Exception raised while executing cookbook sre.switchdc.mediawiki.03-set-db-readonly:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 265, in _run
raw_ret = runner.run()
^^^^^^^^^^^^
File "/srv/deployment/spicerack/cookbooks/sre/switchdc/mediawiki/__init__.py", line 145, in run
ret = self.action()
^^^^^^^^^^^^^
File "/srv/deployment/spicerack/cookbooks/sre/switchdc/mediawiki/03-set-db-readonly.py", line 23, in action
mysql.verify_core_masters_readonly(self.dc_to, True)
File "/usr/lib/python3/dist-packages/spicerack/mysql.py", line 978, in verify_core_masters_readonly
target = self.get_core_dbs(datacenter=datacenter, replication_role="master")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/spicerack/mysql.py", line 926, in get_core_dbs
raise MysqlError(f"Matched {len(mysql_hosts)} masters, expected {dc_multipler * section_multiplier}")
spicerack.mysql.MysqlError: Matched 12 masters, expected 11