Page MenuHomePhabricator

Split mariadb::dbstore_multiinstance into 2 separate roles (backup sources and analytics)
Closed, ResolvedPublic

Description

Originally, mariadb::dbstore role was a mixture of the backup hosts and the aggregated hosts for analytics queries. This then got split and converted into mariadb::dbstore_multiinstance

Nowadays, they use different hostnames: dbstore* vs db* and complete separate functions:

  • dbstore hosts are used by analytics and researchers to run queries on a separate network
  • db hosts are used to generate backups for recovery of mediawiki hosts
  • They have different grants needs (T296274#7522989)
  • They have different users: backup generator automation vs crons/human querying

For that, I propose to split the role (while keeping for now the same profile on each role, unless later has to be different) for easy orchestration:

  • mariadb::backup_source
  • mariadb::analytics_replica

(the names are just a suggestion, they could be something else, in particular for the dbstore).

I would like the ok from all stakeholders: DBAs and analytics, and I can take care of the refactoring if needed (with some help).

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Thanks @jcrespo - I'm happy with that proposed change and with the naming convention.

...propose to split the role (while keeping for now the same profile on each role, unless later has to be different) ...

So on the whole, this is intended to be a no-op on the hosts, but it will give us greater flexibility if and when we want the dbstore* and db* configurations to diverge. Is that about right?

it will give us greater flexibility if and when we want the dbstore* and db* configurations to diverge. Is that about right?

Tecnically it is already diverged, but not maintained anywhere clear- this will allow this to be explicit.

For the scope of this ticket, this should be mostly a noop- only a role name change- it may need some (trivial) changes on cumin aliases, descriptions of roles, etc. but with no impact on current used profiles for now.

Understood, thanks. Well I'm on-board with it. 🙂

Change 740815 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Split the dbstore_multiinstance role into two others

https://gerrit.wikimedia.org/r/740815

Marostegui triaged this task as Medium priority.Nov 24 2021, 6:18 AM
Marostegui moved this task from Triage to Refine on the DBA board.

Change 740815 merged by Jcrespo:

[operations/puppet@production] mariadb: Split the dbstore_multiinstance role into two others

https://gerrit.wikimedia.org/r/740815

Deployment went as expected- but now that I thought a bit, I think btullis brought up a confusing status now:

  • Roles split into mariadb::backup_source and mariadb::analytics_replica
  • Still on the same profile (mariadb::dbstore_multiinstance) but could drift in the future if needed
  • the analytics replicas are called dbstore1* but the backup_sources are called db*

This probably makes the db-store cumin alias confusing. I don't use aliases much, what is a good change for the aliases that would make things less confusing? I think that should be decided before closing this.

Change 741689 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Remove :: from profile setup on 2 mariadb roles

https://gerrit.wikimedia.org/r/741689

Change 741689 merged by Jcrespo:

[operations/puppet@production] mariadb: Remove :: from profile setup on 2 mariadb roles

https://gerrit.wikimedia.org/r/741689

Marostegui moved this task from Refine to In progress on the DBA board.

Assigning to Jaime to reflect current status as I believe he's working on it - if that's not the case please unassign!

Split the alias, have a new db-backup-source alias.

Change 742453 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] cumin: Split out backup sources from db-store alias

https://gerrit.wikimedia.org/r/742453

Reassigning to btullis, as he was the person to bring up the alias issue, so he can evaluate/review if the patch is a good solution to his concerns or not (I don't use aliases).

Change 742453 merged by Kormat:

[operations/puppet@production] cumin: Split out backup sources from db-store alias

https://gerrit.wikimedia.org/r/742453

Cumin alias change is merged, so i'm going to optimistically resolve this. Please reopen if i've been too hasty :)

from:	SYSTEMDTIMER
to:	root@cumin2001.codfw.wmnet

Output of systemd timer for '/usr/local/sbin/check-cumin-aliases'

Alias db-backup-source matched 0 hosts
...
✔️ root@cumin1001:~$ cumin A:db-backup-source
No hosts found that matches the query

Change 743918 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] cumin: Remove obsolete+redundant alias for db-backup-source

https://gerrit.wikimedia.org/r/743918

Change 743918 merged by Kormat:

[operations/puppet@production] cumin: Remove obsolete+redundant alias for db-backup-source

https://gerrit.wikimedia.org/r/743918

from:	SYSTEMDTIMER
to:	root@cumin2001.codfw.wmnet

Output of systemd timer for '/usr/local/sbin/check-cumin-aliases'

Alias db-backup-source matched 0 hosts
...
✔️ root@cumin1001:~$ cumin A:db-backup-source
No hosts found that matches the query

Thanks for pointing that out :)

Fix now:

kormat@cumin1001:~(0:0)$ cumin A:db-backup-source
14 hosts will be targeted:
db[2097-2101,2139,2141].codfw.wmnet,db[1102,1116,1139-1140,1145,1150,1171].eqiad.wmnet
DRY-RUN mode enabled, aborting