Page MenuHomePhabricator
Create Task
Maniphest T217542

prometheus-mysqld-exporter package 0.11.0 options changed
Closed, ResolvedPublic
Actions

Description

I did a apt full-upgrade on labsdb1010 and I noticed the prometheus-mysqld-exporter was not able to come back.

Mar  4 09:42:23 labsdb1010 prometheus-mysqld-exporter[19334]: prometheus-mysqld-exporter: error: unknown short flag '-c', try --help
Mar  4 09:42:23 labsdb1010 systemd[1]: prometheus-mysqld-exporter.service: Main process exited, code=exited, status=1/FAILURE
Mar  4 09:42:23 labsdb1010 systemd[1]: prometheus-mysqld-exporter.service: Unit entered failed state.
Mar  4 09:42:23 labsdb1010 systemd[1]: prometheus-mysqld-exporter.service: Failed with result 'exit-code'.
Mar  4 09:42:24 labsdb1010 systemd[1]: prometheus-mysqld-exporter.service: Service hold-off time over, scheduling restart.

Comparing labsdb1010 with labsdb1009 (not upgraded) looks like the collect options have changed:
labsdb1009

root@labsdb1009:~# /usr/bin/prometheus-mysqld-exporter --version
mysqld_exporter, version 0.10.0+ds (branch: debian/sid, revision: 0.10.0+ds-1~wmf1)

root@labsdb1009:~# /usr/bin/prometheus-mysqld-exporter --help | grep -i collect
Usage of /usr/bin/prometheus-mysqld-exporter:
  -collect.auto_increment.columns
    	Collect auto_increment columns and max values from information_schema
  -collect.binlog_size
    	Collect the current size of all registered binlog files
  -collect.engine_innodb_status
    	Collect from SHOW ENGINE INNODB STATUS
  -collect.engine_tokudb_status
    	Collect from SHOW ENGINE TOKUDB STATUS
  -collect.global_status
    	Collect from SHOW GLOBAL STATUS (default true)
<snip>

labsdb1010:

root@labsdb1010:~# /usr/bin/prometheus-mysqld-exporter --version
mysqld_exporter, version 0.11.0+ds (branch: debian/sid, revision: 0.11.0+ds-1~bpo9+1)

root@labsdb1010:~# /usr/bin/prometheus-mysqld-exporter --help
usage: prometheus-mysqld-exporter [<flags>]

Flags:
  -h, --help                   Show context-sensitive help (also try --help-long and --help-man).
      --exporter.lock_wait_timeout=2
                               Set a lock_wait_timeout on the connection to avoid long metadata locking.
      --exporter.log_slow_filter
                               Add a log_slow_filter to avoid slow query logging of scrapes. NOTE: Not supported by Oracle MySQL.
      --collect.heartbeat.database="heartbeat"
                               Database from where to collect heartbeat data
      --collect.heartbeat.table="heartbeat"

<snip>

Related Objects

Event Timeline

• Marostegui created this task.Mar 4 2019, 9:56 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 4 2019, 9:56 AM
• Marostegui added a comment.Mar 4 2019, 10:15 AM

I was able to start it by changing the default file with these options:

root@labsdb1010:~# diff -u /etc/default/prometheus-mysqld-exporter /root/prometheus-mysqld-exporter
--- /etc/default/prometheus-mysqld-exporter	2019-03-04 10:05:28.412282454 +0000
+++ /root/prometheus-mysqld-exporter	2019-03-04 10:04:50.100798097 +0000
@@ -1 +1 @@
-ARGS="-collect.global_status -collect.global_variables -collect.info_schema.processlist -collect.info_schema.processlist.min_time 0 -collect.slave_status -collect.info_schema.tables false "
\ No newline at end of file
+ARGS="--collect.global_status --collect.global_variables --collect.info_schema.processlist --collect.info_schema.processlist.min_time 0 --collect.slave_status "
Mar 04 10:13:51 labsdb1010 systemd[1]: Started Prometheus exporter for MySQL server.
Mar 04 10:13:51 labsdb1010 prometheus-mysqld-exporter[29759]: time="2019-03-04T10:13:51Z" level=info msg="Starting mysqld_exporter (version=0.11.0+ds, branch=debian/sid, revision=0.11.0+ds-1~bpo9+1)" source="mysqld_exporter.go:206"
Mar 04 10:13:51 labsdb1010 prometheus-mysqld-exporter[29759]: time="2019-03-04T10:13:51Z" level=info msg="Build context (go=go1.7.4, user=pkg-go-maintainers@lists.alioth.debian.org, date=20181206-16:58:47)" source="mysqld_exporter.go:207"
Mar 04 10:13:51 labsdb1010 prometheus-mysqld-exporter[29759]: time="2019-03-04T10:13:51Z" level=info msg="Enabled scrapers:" source="mysqld_exporter.go:218"
Mar 04 10:13:51 labsdb1010 prometheus-mysqld-exporter[29759]: time="2019-03-04T10:13:51Z" level=info msg=" --collect.info_schema.processlist" source="mysqld_exporter.go:222"
Mar 04 10:13:51 labsdb1010 prometheus-mysqld-exporter[29759]: time="2019-03-04T10:13:51Z" level=info msg=" --collect.info_schema.tables" source="mysqld_exporter.go:222"
Mar 04 10:13:51 labsdb1010 prometheus-mysqld-exporter[29759]: time="2019-03-04T10:13:51Z" level=info msg=" --collect.global_status" source="mysqld_exporter.go:222"
Mar 04 10:13:51 labsdb1010 prometheus-mysqld-exporter[29759]: time="2019-03-04T10:13:51Z" level=info msg=" --collect.global_variables" source="mysqld_exporter.go:222"
Mar 04 10:13:51 labsdb1010 prometheus-mysqld-exporter[29759]: time="2019-03-04T10:13:51Z" level=info msg=" --collect.slave_status" source="mysqld_exporter.go:222"
Mar 04 10:13:51 labsdb1010 prometheus-mysqld-exporter[29759]: time="2019-03-04T10:13:51Z" level=info msg="Listening on :9104" source="mysqld_exporter.go:232"

I also removed -collect.info_schema.tables false as looks like it is not supported anymore either.

#  --collect.info_schema.tables.databases="*"
#       The list of databases to collect table stats for, or '*' for all
fgiunchedi added a comment.Mar 4 2019, 10:24 AM

I also noticed 0.10.0+ds-1~wmf1 for some reason wasn't in stretch-wikimedia and I've fixed it now, thus 0.10 should always take precedence now:

labsdb1009:~$ apt-cache policy prometheus-mysqld-exporter
prometheus-mysqld-exporter:
  Installed: 0.10.0+ds-1~wmf1
  Candidate: 0.10.0+ds-1~wmf1
  Version table:
     0.11.0+ds-1~bpo9+1 100
        100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
 *** 0.10.0+ds-1~wmf1 1001
       1001 http://apt.wikimedia.org/wikimedia stretch-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status
     0.9.0+ds-3+b2 500
        500 http://mirrors.wikimedia.org/debian stretch/main amd64 Packages

Aside from the immediate issue moving to 0.11 would be good of course!

MoritzMuehlenhoff added a comment.Mar 4 2019, 10:26 AM

Buster has 0.11.0+ds-1, so this will also be an issue once the first DB host runs it

• Marostegui added a comment.Mar 4 2019, 10:37 AM

Thanks @fgiunchedi - I have done another apt full-upgrade and it downgraded the package and now it works without any manual workaround.
Feel free to close this task or keep open if you consider it should remain open for the future upgrade to 0.11.
Up to you!

Thanks again!

jcrespo added a comment.Mar 4 2019, 10:41 AM

If --collect is supported on 0.10, we may almost only need that change.

fgiunchedi added a comment.Mar 4 2019, 11:56 AM

Yeah I think it makes sense to resolve this @Marostegui as we have T161296: Upgrade mysqld_exporter in production already. @jcrespo no afaict in 0.10 command line options had one - whereas in 0.11 they have two.

• Marostegui closed this task as Resolved.Mar 4 2019, 1:33 PM
• Marostegui assigned this task to fgiunchedi.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits