Page MenuHomePhabricator

spicerack.mysql_legacy errors on get_core_masters_heartbeats when checking x2
Closed, ResolvedPublic

Description

The query generated in get_core_masters_heartbeats is 'A:db-core and A:eqiad and A:db-section-x2 and A:db-role-master'
This creates an exception because the code seems to expect the hosts for A:db-section-x2 to also be in A:db-core, and it's not the case:

cgoubert@cumin1001:~$ sudo cumin 'A:eqiad and A:db-core and A:db-role-master'
11 hosts will be targeted:
db[1100,1103,1109,1118,1131,1136,1138,1157,1162].eqiad.wmnet,es[1021,1024].eqiad.wmnet
DRY-RUN mode enabled, aborting
cgoubert@cumin1001:~$ sudo cumin 'A:eqiad and A:db-section-x2 and A:db-role-master'
1 hosts will be targeted:
db1151.eqiad.wmnet
DRY-RUN mode enabled, aborting

Should db1151 be in A:db-core or should the check skip x2 because it's active/active?

Stacktrace:

1DRY-RUN: Exception raised while executing cookbook sre.switchdc.mediawiki.03-set-db-readonly:
2Traceback (most recent call last):
3 File "/usr/lib/python3/dist-packages/cumin/query.py", line 61, in execute
4 hosts = self._query_default_backend(query_string)
5 File "/usr/lib/python3/dist-packages/cumin/query.py", line 97, in _query_default_backend
6 return query.execute(query_string)
7 File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 47, in execute
8 self._build(query_string)
9 File "/usr/lib/python3/dist-packages/cumin/backends/puppetdb.py", line 308, in _build
10 super()._build(query_string)
11 File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 76, in _build
12 parsed = self.grammar.parseString(query_string.strip(), parseAll=True)
13 File "/usr/lib/python3/dist-packages/pyparsing.py", line 1955, in parseString
14 raise exc
15 File "/usr/lib/python3/dist-packages/pyparsing.py", line 3814, in parseImpl
16 raise ParseException(instring, loc, self.errmsg, self)
17pyparsing.ParseException: Expected end of text, found ':' (at char 1), (line:1, col:2)
18
19During handling of the above exception, another exception occurred:
20
21Traceback (most recent call last):
22 File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 335, in query
23 hosts = query.Query(self._config).execute(query_string)
24 File "/usr/lib/python3/dist-packages/cumin/query.py", line 64, in execute
25 hosts = super().execute(query_string)
26 File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 48, in execute
27 return self._execute()
28 File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 121, in _execute
29 self._loop_stack(hosts, self.stack) # The hosts NodeSet is updated in place while looping the stack
30 File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 167, in _loop_stack
31 self._loop_stack(element_hosts, child)
32 File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 171, in _loop_stack
33 self._aggregate_hosts(hosts, element_hosts, stack_element['bool'])
34 File "/usr/lib/python3/dist-packages/cumin/backends/__init__.py", line 189, in _aggregate_hosts
35 raise InvalidQueryError("Unexpected boolean operator '{boolean}' with hosts '{hosts}'".format(
36cumin.backends.InvalidQueryError: Unexpected boolean operator 'and' with hosts ''
37
38The above exception was the direct cause of the following exception:
39
40Traceback (most recent call last):
41 File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 234, in run
42 raw_ret = runner.run()
43 File "/usr/lib/python3/dist-packages/spicerack/_module_api.py", line 18, in run
44 return self._run(self.args, self.spicerack)
45 File "/srv/deployment/spicerack/cookbooks/sre/switchdc/mediawiki/03-set-db-readonly.py", line 31, in run
46 mysql.check_core_masters_in_sync(args.dc_from, args.dc_to)
47 File "/usr/lib/python3/dist-packages/spicerack/mysql_legacy.py", line 265, in check_core_masters_in_sync
48 heartbeats = self.get_core_masters_heartbeats(dc_from, dc_from)
49 File "/usr/lib/python3/dist-packages/spicerack/mysql_legacy.py", line 287, in get_core_masters_heartbeats
50 core_dbs = self.get_core_dbs(datacenter=datacenter, section=section, replication_role="master")
51 File "/usr/lib/python3/dist-packages/spicerack/mysql_legacy.py", line 177, in get_core_dbs
52 mysql_hosts = MysqlLegacyRemoteHosts(self._remote.query(" and ".join(query_parts)))
53 File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 337, in query
54 raise RemoteError("Failed to execute Cumin query") from e
55spicerack.remote.RemoteError: Failed to execute Cumin query

Event Timeline

Clement_Goubert triaged this task as High priority.
Clement_Goubert moved this task from Incoming 🐫 to Doing 😎 on the serviceops-deprecated board.
Clement_Goubert updated the task description. (Show Details)

They should probably be skipped as we have two masters being written (one per DC) and they need to remain like that - it doesn't matter which DC is active.

@Marostegui Just to clarify and avoid confusion, are you suggesting to remove x2 entirely from the spicerack module and not consider it a core section at all?

FYI the current implementation of the mysql_legacy spicerack module has x2 as part of the CORE_SECTIONS constant and then has an ACTIVE_ACTIVE_SECTIONS constant with just x2 in it to exclude it in some cases.
But the problem is that the get_core_dbs() method assumes that all DBs are part of the A:db-core cumin alias and that's not the case of x2's hosts.
So basically the current implementation of get_core_dbs() is broken with some parameter combination and we need to fix it one way or the other.

[EDIT]
Removing it would mean that it would not be possible to run for x2 methods like check_core_masters_in_sync(), get_core_masters_heartbeats() and check_core_masters_heartbeats()

@Volans I am unsure. How do we treat parsercache at the moment?
x2 is special in the sense that it does have slaves but they're not used (so lag doesn't matter if there any - although it is unlikely there will be any if they are operating normally). It is also special because both of them are actively being written in each DC. So there is no need to go RO for them if we switch DC.
Finally, we don't care about data consistency on those hosts, we are fine if we skip transactions (like we do with parsercache) to a certain extent.

Not sure if this actually helps or not :)

@Volans I am unsure. How do we treat parsercache at the moment?

Parsercache is totally ignored and not considered by the module as core sections.

x2 is special in the sense that it does have slaves but they're not used (so lag doesn't matter if there any - although it is unlikely there will be any if they are operating normally). It is also special because both of them are actively being written in each DC. So there is no need to go RO for them if we switch DC.
Finally, we don't care about data consistency on those hosts, we are fine if we skip transactions (like we do with parsercache) to a certain extent.

Not sure if this actually helps or not :)

Given the above I'd say to remove the whole logic about x2 from the module and ignore them completely. If there is nothing to be done and nothing to be checked not worth the added complexity to keep considering them IMHO.

@Volans I am unsure. How do we treat parsercache at the moment?

Parsercache is totally ignored and not considered by the module as core sections.

x2 is special in the sense that it does have slaves but they're not used (so lag doesn't matter if there any - although it is unlikely there will be any if they are operating normally). It is also special because both of them are actively being written in each DC. So there is no need to go RO for them if we switch DC.
Finally, we don't care about data consistency on those hosts, we are fine if we skip transactions (like we do with parsercache) to a certain extent.

Not sure if this actually helps or not :)

Given the above I'd say to remove the whole logic about x2 from the module and ignore them completely. If there is nothing to be done and nothing to be checked not worth the added complexity to keep considering them IMHO.

Let's do that then!

Change 889490 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/software/spicerack@master] mysql_legacy: remove x2 handling logic

https://gerrit.wikimedia.org/r/889490

The above patch removes x2 from the core databases, and removes the now unused ACTIVE_ACTIVE_SECTIONS constant while keeping the exclusion logic if we ever need it.

Can I get a firm answer from Data-Persistence that this is the way we want to go before merging and having @Volans do a spicerack release please?

Yes, that's the way we should do it given Manuel's comment above and my basic understanding of how x2 works.

Change 889777 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/cookbooks@master] sre.switchdc.mediawiki: Remove ACTIVE_ACTIVE_SECTIONS

https://gerrit.wikimedia.org/r/889777

Thanks @Ladsgroup once the spicerack release is done I'll test the cookbook patch and we should be good on that front.

Change 889490 merged by jenkins-bot:

[operations/software/spicerack@master] mysql_legacy: remove x2 handling logic

https://gerrit.wikimedia.org/r/889490

Change 889777 merged by jenkins-bot:

[operations/cookbooks@master] sre.switchdc.mediawiki: Remove ACTIVE_ACTIVE_SECTIONS

https://gerrit.wikimedia.org/r/889777

Clement_Goubert claimed this task.

dry-run looks good, resolving