Page MenuHomePhabricator

Standardize/centralize mapping from section to mariadb port/socket and prom-mysql-exporter port
Closed, ResolvedPublic

Description

For single-instance hosts, this is simple. The mariadb port and unix socket are :3306 and /run/mysqld/mysqld.sock, and the prom-mysql-exporter runs on :9104. For multi-instance hosts, this gets more complicated.

Each section has an 'index' that is not explicitly documented (that i could find, in any case). s1 has index 1, s8 has index 8, m5 has index 15, etc. The mariadb port for a multi-instance host is 3310 + the index of the section. So x1 (which has index 10) runs mariadb on port :3320. The prom-mysql-exporter port is 13310 + the index of the section (so staging with an index of 40 has a prom exporter port of :13350).

Mapping from a section to a mariadb unix socket is straight-forward, with one catch. The socket is /run/mysqld/mysqld.<section name>.sock. The problem, however, is that if the section name has a - in it this will break due to systemd's escaping rules. Currently we do not handle this special-case anywhere.

The above rules are implicitly encoded or relied upon in many places:

  • Puppet:
    • All the multiinstance mariadb puppet profiles. E.g. modules/profile/manifests/mariadb/dbstore_multiinstance.pp (link)
    • conftool-data/dbconfig-instance/instances.yaml (link)
    • modules/mariadb/files/check_mariadb.py (link)
    • modules/profile/files/prometheus/mysqld_exporter_config.py (link)
    • modules/profile/files/mariadb/backup_mariadb.py (link)
    • modules/profile/files/mariadb/remote_backup_mariadb.py (link)
    • Backup configs. E.g. modules/profile/templates/mariadb/backup_config/cumin1001.cnf.erb (link)
  • wmfmariadbpy:
    • wmfmariadbpy/WMFMariaDB.py (link)
    • wmfmariadbpy/WMFBackup.py (link)

Ideally, this mapping would be precisely defined in exactly one place, but even one place per language (puppet and python) would be a massive improvement. Centralising this would also have the benefit that we could fix things like the mapping of sections that contain - to unix sockets in a way that makes systemd happy.

Event Timeline

Kormat moved this task from Triage to Backlog on the DBA board.
herron triaged this task as Medium priority.Jul 27 2020, 1:44 PM

Change 620291 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfmariadbpy@master] wmfmariadbpy: Load and provide a method for section to port assignment

https://gerrit.wikimedia.org/r/620291

Change 620688 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] mariadb: escape title as '-' is causes issues

https://gerrit.wikimedia.org/r/620688

m5 has index 15

guessing this should be "m5 has index 5"?

or after reading ". So x1 (which has index 10" should

s1 has index 1, s8 has index 8, m5 has index 15

be "s1 has index 10, s8 has index 18, m5 has index 15"

The problem, however, is that if the section name has a - in it this will break

Sent a fix for this however i don't know what affect it will have on current instances

when looking at the above fix i noticed a possible bug. should the following be $socket instead of $tmpdir

if $tmpdir == 'undefined' {
    $socket_instance = "/run/mysqld/mysqld.${title}.sock"
} else {
    $socket_instance = $socket
}

m5 has index 15

guessing this should be "m5 has index 5"?

Maybe the wording isn't clear. Here's a quick overview of the mapping: https://gerrit.wikimedia.org/r/c/operations/software/wmfmariadbpy/+/620291/8/config/section_ports.csv

The problem, however, is that if the section name has a - in it this will break

Sent a fix for this however i don't know what affect it will have on current instances

Replied on the CR.

when looking at the above fix i noticed a possible bug. should the following be $socket instead of $tmpdir

if $tmpdir == 'undefined' {
    $socket_instance = "/run/mysqld/mysqld.${title}.sock"
} else {
    $socket_instance = $socket
}

Ooh yep, that's a bug. I'll send a CR to fix that.

Change 620688 abandoned by Jbond:
[operations/puppet@production] mariadb: escape title as '-' is causes issues

Reason:
see comments

https://gerrit.wikimedia.org/r/620688

m5 has index 15

guessing this should be "m5 has index 5"?

Maybe the wording isn't clear. Here's a quick overview of the mapping: https://gerrit.wikimedia.org/r/c/operations/software/wmfmariadbpy/+/620291/8/config/section_ports.csv

I think i missed the differing letters (i.e. s vs m vs x) . i follow now, thanks

Change 620722 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Setup section->port assignment on puppet

https://gerrit.wikimedia.org/r/620722

Change 620899 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Apply the list of port to the core::multiinstance class

https://gerrit.wikimedia.org/r/620899

Change 620722 merged by Jcrespo:
[operations/puppet@production] mariadb: Setup section->port assignment on puppet

https://gerrit.wikimedia.org/r/620722

Change 620291 merged by Jcrespo:
[operations/software/wmfmariadbpy@master] wmfmariadbpy: Load and provide a method for section to port assignment

https://gerrit.wikimedia.org/r/620291

Change 620899 merged by Kormat:
[operations/puppet@production] mariadb: core::multiinstance - generate sections.

https://gerrit.wikimedia.org/r/620899

@Kormat: My patches solved some of the isuess above, even if one can improve over it. Maybe we should reevaluate the current status to see if this is resolved or check what is missing?

Kormat claimed this task.

I think it's good enough to resolve at this point. Thanks for the reminder :)