Page MenuHomePhabricator
Create Task
Maniphest T257033

Standardize/centralize mapping from section to mariadb port/socket and prom-mysql-exporter port
Closed, ResolvedPublic
Actions

Description

For single-instance hosts, this is simple. The mariadb port and unix socket are :3306 and /run/mysqld/mysqld.sock, and the prom-mysql-exporter runs on :9104. For multi-instance hosts, this gets more complicated.

Each section has an 'index' that is not explicitly documented (that i could find, in any case). s1 has index 1, s8 has index 8, m5 has index 15, etc. The mariadb port for a multi-instance host is 3310 + the index of the section. So x1 (which has index 10) runs mariadb on port :3320. The prom-mysql-exporter port is 13310 + the index of the section (so staging with an index of 40 has a prom exporter port of :13350).

Mapping from a section to a mariadb unix socket is straight-forward, with one catch. The socket is /run/mysqld/mysqld.<section name>.sock. The problem, however, is that if the section name has a - in it this will break due to systemd's escaping rules. Currently we do not handle this special-case anywhere.

The above rules are implicitly encoded or relied upon in many places:

  • Puppet:
    • All the multiinstance mariadb puppet profiles. E.g. modules/profile/manifests/mariadb/dbstore_multiinstance.pp (link)
    • conftool-data/dbconfig-instance/instances.yaml (link)
    • modules/mariadb/files/check_mariadb.py (link)
    • modules/profile/files/prometheus/mysqld_exporter_config.py (link)
    • modules/profile/files/mariadb/backup_mariadb.py (link)
    • modules/profile/files/mariadb/remote_backup_mariadb.py (link)
    • Backup configs. E.g. modules/profile/templates/mariadb/backup_config/cumin1001.cnf.erb (link)
  • wmfmariadbpy:
    • wmfmariadbpy/WMFMariaDB.py (link)
    • wmfmariadbpy/WMFBackup.py (link)

Ideally, this mapping would be precisely defined in exactly one place, but even one place per language (puppet and python) would be a massive improvement. Centralising this would also have the benefit that we could fix things like the mapping of sections that contain - to unix sockets in a way that makes systemd happy.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
mariadb: core::multiinstance - generate sections.operations/puppetproduction+64 -165
wmfmariadbpy: Load and provide a method for section to port assignmentoperations/software/wmfmariadbpymaster+118 -38
mariadb: Setup section->port assignment on puppetoperations/puppetproduction+60 -3
mariadb: escape title as '-' is causes issuesoperations/puppetproduction+7 -6
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Kormat created this task.Jul 3 2020, 10:27 AM
• Kormat moved this task from Unsorted 💣 to Back Burner 🏛️ on the User-Kormat board.Jul 3 2020, 12:06 PM
• Kormat moved this task from Triage to Backlog on the DBA board.
• herron triaged this task as Medium priority.Jul 27 2020, 1:44 PM
gerritbot added a comment.Aug 14 2020, 8:42 AM

Change 620291 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software/wmfmariadbpy@master] wmfmariadbpy: Load and provide a method for section to port assignment

https://gerrit.wikimedia.org/r/620291

gerritbot added a project: Patch-For-Review.Aug 14 2020, 8:42 AM
gerritbot added a comment.Aug 17 2020, 11:14 AM

Change 620688 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] mariadb: escape title as '-' is causes issues

https://gerrit.wikimedia.org/r/620688

jbond added a comment.EditedAug 17 2020, 11:20 AM

m5 has index 15

guessing this should be "m5 has index 5"?

or after reading ". So x1 (which has index 10" should

s1 has index 1, s8 has index 8, m5 has index 15

be "s1 has index 10, s8 has index 18, m5 has index 15"

The problem, however, is that if the section name has a - in it this will break

Sent a fix for this however i don't know what affect it will have on current instances

when looking at the above fix i noticed a possible bug. should the following be $socket instead of $tmpdir

if $tmpdir == 'undefined' {
    $socket_instance = "/run/mysqld/mysqld.${title}.sock"
} else {
    $socket_instance = $socket
}
• Kormat added a comment.Aug 17 2020, 12:49 PM
In T257033#6388369, @jbond wrote:

m5 has index 15

guessing this should be "m5 has index 5"?

Maybe the wording isn't clear. Here's a quick overview of the mapping: https://gerrit.wikimedia.org/r/c/operations/software/wmfmariadbpy/+/620291/8/config/section_ports.csv

The problem, however, is that if the section name has a - in it this will break

Sent a fix for this however i don't know what affect it will have on current instances

Replied on the CR.

when looking at the above fix i noticed a possible bug. should the following be $socket instead of $tmpdir

if $tmpdir == 'undefined' {
    $socket_instance = "/run/mysqld/mysqld.${title}.sock"
} else {
    $socket_instance = $socket
}

Ooh yep, that's a bug. I'll send a CR to fix that.

gerritbot added a comment.Aug 17 2020, 12:52 PM

Change 620688 abandoned by Jbond:
[operations/puppet@production] mariadb: escape title as '-' is causes issues

Reason:
see comments

https://gerrit.wikimedia.org/r/620688

jbond added a comment.Aug 17 2020, 12:55 PM
In T257033#6388644, @Kormat wrote:
In T257033#6388369, @jbond wrote:

m5 has index 15

guessing this should be "m5 has index 5"?

Maybe the wording isn't clear. Here's a quick overview of the mapping: https://gerrit.wikimedia.org/r/c/operations/software/wmfmariadbpy/+/620291/8/config/section_ports.csv

I think i missed the differing letters (i.e. s vs m vs x) . i follow now, thanks

gerritbot added a comment.Aug 17 2020, 3:34 PM

Change 620722 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Setup section->port assignment on puppet

https://gerrit.wikimedia.org/r/620722

gerritbot added a comment.Aug 18 2020, 9:39 AM

Change 620899 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Apply the list of port to the core::multiinstance class

https://gerrit.wikimedia.org/r/620899

gerritbot added a comment.Aug 26 2020, 12:56 PM

Change 620722 merged by Jcrespo:
[operations/puppet@production] mariadb: Setup section->port assignment on puppet

https://gerrit.wikimedia.org/r/620722

gerritbot added a comment.Aug 26 2020, 1:02 PM

Change 620291 merged by Jcrespo:
[operations/software/wmfmariadbpy@master] wmfmariadbpy: Load and provide a method for section to port assignment

https://gerrit.wikimedia.org/r/620291

jcrespo mentioned this in rOSWB36505976ad9f: wmfmariadbpy: Load and provide a method for section to port assignment.Sep 1 2020, 8:36 AM
gerritbot added a comment.Oct 15 2020, 1:44 PM

Change 620899 merged by Kormat:
[operations/puppet@production] mariadb: core::multiinstance - generate sections.

https://gerrit.wikimedia.org/r/620899

jbond moved this task from Unsorted 💣 to Watching 👀 on the User-jbond board.Nov 20 2020, 11:34 AM
jcrespo subscribed.Nov 25 2020, 6:29 PM

@Kormat: My patches solved some of the isuess above, even if one can improve over it. Maybe we should reevaluate the current status to see if this is resolved or check what is missing?

• Kormat closed this task as Resolved.Nov 26 2020, 9:25 AM
• Kormat claimed this task.

I think it's good enough to resolve at this point. Thanks for the reminder :)

• Kormat mentioned this in T269097: Standardize extra-port for mariadb instances.Dec 1 2020, 10:21 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits