Page MenuHomePhabricator
Create Task
Maniphest T289488

Systemd enhancements for mariadb and prometheus-mysql-exporter
Closed, ResolvedPublic
Actions

Description

This task captures some improvements to the systemd handling of the mariadb and prometheus-mysql-exporter ("PME") services; the aim being to simplify their management. Particularly, the problem that currently you have to remember to restart PME after re-starting mariadb; and that on multi-instance hosts you have to handle each instance separately.

  • Couple PME and mariadb services on single-instance hosts
  • Couple PME and mariadb services on multi-instance hosts
  • systemd targets so systemctl restart mariadb.target DTRT on single and multi-instance hosts (optional)

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
control-mariadb-*: Bump versionMarosteguioperations/softwaremaster+2 -2
prometheus: couple mysqld export service to mariadb (multi-instance)MVernonoperations/puppetproduction+13 -7
dbtools: make mariadb service Wants prometheus-mysqld-exporterMVernonoperations/softwaremaster+8 -0
prometheus: couple mysqld exporter service to mariadb serviceMVernonoperations/puppetproduction+23 -7
Customize query in gerrit

Related Objects

Event Timeline

MatthewVernon created this task.Aug 23 2021, 12:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 23 2021, 12:30 PM
LSobanski subscribed.Aug 23 2021, 12:34 PM
MatthewVernon mentioned this in T252761: Research performance changes on prometheus-mysqld-exporter after buster/mariadb upgrade.Aug 23 2021, 12:38 PM
MatthewVernon updated the task description. (Show Details)Aug 23 2021, 12:50 PM
gerritbot added a comment.Aug 23 2021, 1:01 PM

Change 714358 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] prometheus: couple mysqld exporter service to mariadb service

https://gerrit.wikimedia.org/r/714358

gerritbot added a project: Patch-For-Review.Aug 23 2021, 1:01 PM
Marostegui subscribed.Aug 24 2021, 6:04 AM

I know it is not totally related to this task, but maybe this can be also looked at as part of this? T257056: Add alert for prometheus-mysql-exporter failing to scrape mysql

LSobanski triaged this task as Medium priority.Aug 30 2021, 6:55 AM
LSobanski moved this task from Triage to In progress on the DBA board.
gerritbot added a comment.Sep 1 2021, 9:03 AM

Change 715926 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/software@master] dbtools: make mariadb service Wants prometheus-mysqld-exporter

https://gerrit.wikimedia.org/r/715926

gerritbot added a comment.Sep 2 2021, 10:24 AM

Change 714358 merged by MVernon:

[operations/puppet@production] prometheus: couple mysqld exporter service to mariadb service

https://gerrit.wikimedia.org/r/714358

• Kormat mentioned this in T256972: Refactor mariadb puppet code.Sep 2 2021, 1:10 PM
gerritbot added a comment.Sep 2 2021, 1:15 PM

Change 715926 merged by MVernon:

[operations/software@master] dbtools: make mariadb service Wants prometheus-mysqld-exporter

https://gerrit.wikimedia.org/r/715926

gerritbot added a comment.Sep 2 2021, 1:39 PM

Change 716306 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] prometheus: couple mysqld export service to mariadb (multi-instance)

https://gerrit.wikimedia.org/r/716306

gerritbot added a comment.Sep 6 2021, 12:17 PM

Change 716306 merged by MVernon:

[operations/puppet@production] prometheus: couple mysqld export service to mariadb (multi-instance)

https://gerrit.wikimedia.org/r/716306

MatthewVernon updated the task description. (Show Details)Sep 6 2021, 1:15 PM
MatthewVernon added a comment.Sep 6 2021, 1:25 PM
In T289488#7303806, @Marostegui wrote:

I know it is not totally related to this task, but maybe this can be also looked at as part of this? T257056: Add alert for prometheus-mysql-exporter failing to scrape mysql

I will have at least a look at that, but I think that I'm not going to make resolving it part of this task, IYSWIM?

MatthewVernon added a subscriber: • Kormat.Sep 6 2021, 1:27 PM

I think we concluded that the mariadb.target idea isn't all that useful (since mostly folk don't stop and start >1 instance at once I think @Kormat said).

So I'm inclined to view this task as done now? We have the coupling of prometheus-mysqld-exporter and mariadb done by system on both single and multi-instance hosts.

• Kormat added a comment.Sep 6 2021, 1:27 PM
In T289488#7334427, @MatthewVernon wrote:
In T289488#7303806, @Marostegui wrote:

I know it is not totally related to this task, but maybe this can be also looked at as part of this? T257056: Add alert for prometheus-mysql-exporter failing to scrape mysql

I will have at least a look at that, but I think that I'm not going to make resolving it part of this task, IYSWIM?

👍

Marostegui added a comment.Sep 6 2021, 1:29 PM
In T289488#7334427, @MatthewVernon wrote:
In T289488#7303806, @Marostegui wrote:

I know it is not totally related to this task, but maybe this can be also looked at as part of this? T257056: Add alert for prometheus-mysql-exporter failing to scrape mysql

I will have at least a look at that, but I think that I'm not going to make resolving it part of this task, IYSWIM?

Yes, it can be tracked on the other task itself. Not a blocker for closing this one as soon as you think it is appropriate to do so.

Thanks for your work!

MatthewVernon closed this task as Resolved.Sep 6 2021, 1:38 PM
• Kormat reopened this task as Open.Sep 10 2021, 9:11 AM

(Re-opening as until the new mariadb packages are built and deployed we're in a slightly worse situation then before the work started)

Tl;DR: Until we have new packages build/deployed, prometheus-mysqld-exporter will need to be manually started on hosts after a reboot.

In the original state the exporter would be running, but would fail to connect to mariadb. This has the minor benefit that systemctl will tab-complete to the service correctly.
In the intermediate state we're currently in, systemd won't start PME on its own initiative, and the new Wants=PME in mariadb's .service isn't present yet.

gerritbot added a comment.Sep 15 2021, 11:42 AM

Change 721284 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/software@master] control-mariadb-*: Bump version

https://gerrit.wikimedia.org/r/721284

gerritbot added a comment.Sep 15 2021, 11:44 AM

Change 721284 merged by jenkins-bot:

[operations/software@master] control-mariadb-*: Bump version

https://gerrit.wikimedia.org/r/721284

Marostegui added a comment.Sep 15 2021, 11:46 AM

The new package 10.4.21-2 includes this patch.
It has been installed on db1125 (test host), so feel free to play with it if needed. I have done several stop/start and all works fine.

MatthewVernon added a comment.Sep 15 2021, 12:29 PM

I've also checked the stop/start/restart behaviour, which is as expected.
Also, that on reboot PME isn't started, but when you start mariadb, it does then get started for you.

Marostegui added a comment.Sep 15 2021, 12:30 PM

We can test tomorrow on a multi instance host in codfw if you like (as today we are still not touching production to let the dust settle from yesterday's switchover)

MatthewVernon closed this task as Resolved.Sep 15 2021, 1:16 PM

Marking this as resolved - as we deploy 10.4.21-2 everywhere, the fix will get rolled out.

MatthewVernon mentioned this in rOALE3921c3d2300c: data-protection: add alerting for prometheus-mysqld-exporter failing.Oct 5 2021, 2:32 PM
MatthewVernon mentioned this in T292957: Investigate the root cause of prometheus-mysqld-exporter alert on db1119.Oct 11 2021, 10:34 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits