Page MenuHomePhabricator
Create Task
Maniphest T213708

Upgrade production prometheus-node-exporter to >= 0.16
Closed, ResolvedPublic
Actions

Description

This task is to track the upgrade of node-exporter to >= 0.16. Note that we'd want to upgrade to a version that matches Debian Buster, to ease migration (Buster tracking task is T213527).

Prior upgrades: T152580 and T166561. Though in this case there's a bunch of metrics renamed, so we'll have to be backwards compatible with the old names at least for a little while, see also https://github.com/prometheus/node_exporter/blob/v0.16.0/docs/V0_16_UPGRADE_GUIDE.md.

Proposed plan of attack:

  • Make sure Buster/Stretch/Jessie all have the same node-exporter version available (in practice this means backporting 0.17 to jessie, which shouldn't pose particular problems)
  • Test the upgrade in beta:
    • Deploy the compatibility recording rules above
    • Upgrade node-exporter
    • Verify dashboards still report data as expected under old names
  • Extend the upgrade to production, one site at a time (if easy to do)
  • (Optional, can be postponed) audit/change dashboards to use new metric names, and retire compatibility recording rules -- moved to new task

Details

SubjectRepoBranchLines +/-
hiera: install node exporter 0.17 in betaoperations/puppetproduction+2 -0
hiera: upgrade prometheus-node-exporter to 0.17 in esamsoperations/puppetproduction+1 -0
hiera: upgrade prometheus-node-exporter to 0.17 in esamsoperations/puppetproduction+1 -0
prometheus: clean up node exporter transition codeoperations/puppetproduction+32 -88
prometheus: clean up node exporter transition codeoperations/puppetproduction+37 -83
hiera: upgrade prometheus-node-exporter to 0.17 in eqiadoperations/puppetproduction+1 -0
Don't set a fixed package version for prometheus-node-exporter on busteroperations/puppetproduction+12 -8
prometheus: post-upgrade node-exporter cleanupoperations/puppetproduction+26 -44
prometheus: require package before masking servicesoperations/puppetproduction+2 -0
hiera: upgrade prometheus-node-exporter to 0.17 in codfwoperations/puppetproduction+1 -0
prometheus: disable shipped node-exporter ipmitool and smartmon timersoperations/puppetproduction+15 -0
hiera: upgrade prometheus-node-exporter to 0.17 in labsoperations/puppetproduction+1 -1
hiera: upgrade prometheus-node-exporter to 0.17 in labsoperations/puppetproduction+2 -0
hiera: upgrade prometheus-node-exporter to 0.17 in ulsfooperations/puppetproduction+1 -1
hiera: upgrade prometheus-node-exporter to 0.17 in esamsoperations/puppetproduction+1 -0
prometheus: attempt to force apt updateoperations/puppetproduction+7 -1
prometheus: do not change trusty hostsoperations/puppetproduction+52 -34
prometheus: use web_listen_address with node-exporter 0.17operations/puppetproduction+1 -1
prometheus: use non-namespaced hiera key to enable site lookupoperations/puppetproduction+2 -2
hiera: upgrade prometheus-node-exporter to 0.17 in eqsinoperations/puppetproduction+1 -0
prometheus: upgrade prometheus-node-exporter to latest patchsetoperations/puppetproduction+1 -1
prometheus: upgrade to node-exporter 0.17operations/puppetproduction+86 -16
role: add backwards-compatibility rules to prometheusoperations/puppetproduction+385 -2
aptrepo: add prometheus-node-exporter components for all distsoperations/puppetproduction+2 -2
role: add prometheus2 backwards-compatibility rulesoperations/puppetproduction+595 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi triaged this task as Medium priority.Jan 14 2019, 2:57 PM
fgiunchedi created this task.
gerritbot subscribed.Jan 16 2019, 11:42 PM

Change 484793 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] role: add prometheus2 rules (new format)

https://gerrit.wikimedia.org/r/484793

gerritbot added a project: Patch-For-Review.Jan 16 2019, 11:42 PM
colewhite subscribed.Jan 17 2019, 5:09 PM

Changeset [1] contains a current snapshot of the converted rules files. These will be needed for prometheus server to not only maintain current behavior, but it has added the backwards-compatibility rules from [2].

I manually tested the buster packages on stretch and jessie and they appear to run in default configuration. They may work as-is if we load it into the distro-specific repositories. TODO: test our custom configuration.

[1]: https://gerrit.wikimedia.org/r/484793
[2]: https://github.com/prometheus/node_exporter/blob/v0.17.0/docs/example-16-compatibility-rules.yml

fgiunchedi added a comment.Jan 17 2019, 5:18 PM

I also just realized that the compatibility rules are in Prometheus v2 format, though we'll need them in v1 format as well to decouple the v1 -> v2 migration from node-exporter upgrade

colewhite added a comment.EditedJan 17 2019, 10:18 PM

After deploying the rules and node-exporter v0.17 to deployment-prometheus02, it appears the rules are not for backwards compatibility, but for forwards compatibility. Dashboards will need to be updated before node-exporter 0.17 is deployed.

colewhite added a comment.EditedJan 17 2019, 10:58 PM

Tested command line flags for prometheus-node-exporter v0.17

ARGS='--collector.diskstats.ignored-devices=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvmed+nd+p)d+$ --collector.filesystem.ignored-fs-types=^(overlay|autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|nsfs|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$ --collector.filesystem.ignored-mount-points=^/(sys|proc|dev|var/lib/docker|var/lib/kubelet)($|/) --collector.textfile.directory=/var/lib/prometheus/node.d --collector.buddyinfo --collector.conntrack --collector.diskstats --collector.edac --collector.entropy --collector.filefd --collector.filesystem --collector.hwmon --collector.loadavg --collector.mdadm --collector.meminfo --collector.netdev --collector.netstat --collector.netstat.fields="^(.*)" --collector.sockstat --collector.stat --collector.tcpstat --collector.textfile --collector.time --collector.uname --collector.vmstat --collector.vmstat.fields="^(.*)" --web.listen-address=:9100'
gerritbot added a comment.Jan 22 2019, 8:17 PM

Change 485889 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] role: add forwards-compatibility rules to prometheus

https://gerrit.wikimedia.org/r/485889

gerritbot added a comment.Jan 23 2019, 11:31 AM

Change 484793 merged by Filippo Giunchedi:
[operations/puppet@production] role: add prometheus2 backwards-compatibility rules

https://gerrit.wikimedia.org/r/484793

gerritbot added a comment.Jan 24 2019, 12:02 AM

Change 486192 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] prometheus: upgrade to node-exporter 0.17

https://gerrit.wikimedia.org/r/486192

fgiunchedi mentioned this in T214558: Puppet failure on deployment-prometheus01.deployment-prep.eqiad.wmflabs.Jan 24 2019, 10:07 AM
colewhite updated the task description. (Show Details)Jan 24 2019, 6:21 PM
gerritbot added a comment.Jan 25 2019, 4:24 PM

Change 486493 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] aptrepo: add prometheus-node-exporter component for jessie

https://gerrit.wikimedia.org/r/486493

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 11:04 PM
jbond subscribed.Feb 6 2019, 12:49 PM
gerritbot added a comment.Feb 6 2019, 6:25 PM

Change 486493 merged by Cwhite:
[operations/puppet@production] aptrepo: add prometheus-node-exporter components for all dists

https://gerrit.wikimedia.org/r/486493

gerritbot added a comment.Feb 6 2019, 10:15 PM

Change 488593 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] hiera: install node exporter 0.17 in beta

https://gerrit.wikimedia.org/r/488593

Stashbot subscribed.Feb 6 2019, 10:34 PM

Mentioned in SAL (#wikimedia-cloud) [2019-02-06T22:34:32Z] <shdubsh> Deploy node-exporter 0.17 T213708

gerritbot added a comment.Feb 7 2019, 7:04 PM

Change 485889 merged by Cwhite:
[operations/puppet@production] role: add backwards-compatibility rules to prometheus

https://gerrit.wikimedia.org/r/485889

gerritbot added a comment.Feb 7 2019, 7:25 PM

Change 486192 merged by Cwhite:
[operations/puppet@production] prometheus: upgrade to node-exporter 0.17

https://gerrit.wikimedia.org/r/486192

fgiunchedi moved this task from Inbox to In progress on the observability board.Feb 8 2019, 11:37 AM
colewhite updated the task description. (Show Details)Feb 8 2019, 8:16 PM
gerritbot added a comment.Feb 8 2019, 10:16 PM

Change 489325 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] prometheus: post-upgrade node-exporter cleanup

https://gerrit.wikimedia.org/r/489325

MoritzMuehlenhoff subscribed.Feb 11 2019, 2:21 PM

We currently pin prometheus-node-exporter to 0.17.0+ds-2 on the selected hosts and for buster, but yesterday 0.17.0+ds-3 migrated to testing/buster. I could change the puppet code to pick -3 on buster, but I'd say we upgrade the components for jessie and stretch also to -3 and bump it in general? https://packages.qa.debian.org/p/prometheus-node-exporter/news/20190131T180815Z.html lists a number of fixes and at least the TMPDIR change seems relevant as for those as well.

fgiunchedi added a comment.Feb 11 2019, 4:05 PM

Uploading -3 internally and changing puppet to install that version sounds good to me!

gerritbot added a comment.Feb 11 2019, 6:19 PM

Change 489753 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] hiera: upgrade prometheus-node-exporter to 0.17 in labs

https://gerrit.wikimedia.org/r/489753

gerritbot added a comment.Feb 11 2019, 6:21 PM

Change 489754 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] hiera: upgrade prometheus-node-exporter to 0.17 in eqsin

https://gerrit.wikimedia.org/r/489754

gerritbot added a comment.Feb 11 2019, 6:25 PM

Change 489756 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] prometheus: upgrade prometheus-node-exporter to latest patchset

https://gerrit.wikimedia.org/r/489756

gerritbot added a comment.Feb 12 2019, 8:37 AM

Change 489756 merged by Muehlenhoff:
[operations/puppet@production] prometheus: upgrade prometheus-node-exporter to latest patchset

https://gerrit.wikimedia.org/r/489756

gerritbot added a comment.Feb 12 2019, 4:33 PM

Change 489754 merged by Cwhite:
[operations/puppet@production] hiera: upgrade prometheus-node-exporter to 0.17 in eqsin

https://gerrit.wikimedia.org/r/489754

gerritbot added a comment.Feb 12 2019, 4:57 PM

Change 490095 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] prometheus: use non-namespaced hiera key to enable site lookup

https://gerrit.wikimedia.org/r/490095

gerritbot added a comment.Feb 12 2019, 5:02 PM

Change 490095 merged by Cwhite:
[operations/puppet@production] prometheus: use non-namespaced hiera key to enable site lookup

https://gerrit.wikimedia.org/r/490095

gerritbot added a comment.Feb 12 2019, 11:31 PM

Change 490203 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] prometheus: do not change trusty hosts

https://gerrit.wikimedia.org/r/490203

gerritbot added a comment.Feb 12 2019, 11:37 PM

Change 490204 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] prometheus: attempt to force apt update

https://gerrit.wikimedia.org/r/490204

gerritbot added a comment.Feb 13 2019, 12:14 AM

Change 490229 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] hiera: upgrade prometheus-node-exporter to 0.17 in esams

https://gerrit.wikimedia.org/r/490229

colewhite updated the task description. (Show Details)Feb 13 2019, 3:04 AM
Joe subscribed.Feb 13 2019, 10:29 AM

I've just noticed, based on a diffscan email, that the new version of prometheus-node-exporter ALSO binds to :::9100 on ipv6 and listens to all ipv6 clients, while the old node exporter version would only bind to a specific interface on ipv4 and listen on that interface.

This means that on publicly-exposed hosts, we expose node-exporter to the world over ipv6 if there is no firewall rule. Some servers have no firewall by design, so node-exporter is actually reachable from the open internet.

The solution is probably to just not listen over ipv6.

gerritbot added a comment.Feb 13 2019, 10:59 AM

Change 490304 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: use web_listen_address with node-exporter 0.17

https://gerrit.wikimedia.org/r/490304

gerritbot added a comment.Feb 13 2019, 11:02 AM

Change 490304 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: use web_listen_address with node-exporter 0.17

https://gerrit.wikimedia.org/r/490304

fgiunchedi added a comment.Feb 13 2019, 11:04 AM
In T213708#4950462, @Joe wrote:

I've just noticed, based on a diffscan email, that the new version of prometheus-node-exporter ALSO binds to :::9100 on ipv6 and listens to all ipv6 clients, while the old node exporter version would only bind to a specific interface on ipv4 and listen on that interface.

This means that on publicly-exposed hosts, we expose node-exporter to the world over ipv6 if there is no firewall rule. Some servers have no firewall by design, so node-exporter is actually reachable from the open internet.

The solution is probably to just not listen over ipv6.

Indeed, this was a regression in the commandline options for node-exporter, fixed now in https://gerrit.wikimedia.org/r/490304

gerritbot added a comment.Feb 13 2019, 11:11 PM

Change 490203 merged by Cwhite:
[operations/puppet@production] prometheus: do not change trusty hosts

https://gerrit.wikimedia.org/r/490203

colewhite claimed this task.Feb 13 2019, 11:36 PM
gerritbot added a comment.Feb 14 2019, 5:12 PM

Change 490204 merged by Cwhite:
[operations/puppet@production] prometheus: attempt to force apt update

https://gerrit.wikimedia.org/r/490204

gerritbot added a comment.Feb 14 2019, 5:20 PM

Change 490229 merged by Cwhite:
[operations/puppet@production] hiera: upgrade prometheus-node-exporter to 0.17 in esams

https://gerrit.wikimedia.org/r/490229

gerritbot added a comment.Feb 14 2019, 5:33 PM

Change 490651 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] hiera: upgrade prometheus-node-exporter to 0.17 in ulsfo

https://gerrit.wikimedia.org/r/490651

gerritbot added a comment.Feb 14 2019, 5:42 PM

Change 490651 merged by Cwhite:
[operations/puppet@production] hiera: upgrade prometheus-node-exporter to 0.17 in ulsfo

https://gerrit.wikimedia.org/r/490651

gerritbot added a comment.Feb 14 2019, 8:28 PM

Change 490689 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] hiera: upgrade prometheus-node-exporter to 0.17 in codfw

https://gerrit.wikimedia.org/r/490689

gerritbot added a comment.Feb 14 2019, 8:36 PM

Change 490690 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] hiera: upgrade prometheus-node-exporter to 0.17 in eqiad

https://gerrit.wikimedia.org/r/490690

gerritbot added a comment.Feb 14 2019, 8:41 PM

Change 489753 merged by Cwhite:
[operations/puppet@production] hiera: upgrade prometheus-node-exporter to 0.17 in labs

https://gerrit.wikimedia.org/r/489753

gerritbot added a comment.Feb 14 2019, 8:45 PM

Change 490693 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] hiera: upgrade prometheus-node-exporter to 0.17 in labs

https://gerrit.wikimedia.org/r/490693

gerritbot added a comment.Feb 14 2019, 8:46 PM

Change 490693 merged by Cwhite:
[operations/puppet@production] hiera: upgrade prometheus-node-exporter to 0.17 in labs

https://gerrit.wikimedia.org/r/490693

fgiunchedi added a comment.Feb 20 2019, 10:57 AM

Noticed this today on bast3002, probably harmless but needs investigation:

Feb 20 10:26:50 bast3002 systemd[1]: Starting Collect ipmitool sensor metrics for prometheus-node-exporter...
Feb 20 10:26:50 bast3002 sh[14417]: awk: not an option: -nf
colewhite added a comment.Feb 21 2019, 4:51 PM

On further investigation, the log messages appear to be from the shebang of the ipmitool awk script.

Patch submitted upstream and a Debian bug has been filed.

CDanis added a comment.Feb 21 2019, 4:57 PM

Looks like -n is/was a gawk option?

-n
--non-decimal-data
Enable automatic interpretation of octal and hexadecimal values in input
data (see Nondecimal Data).
CAUTION: This option can severely break old programs. Use with care. Also
note that this option may disappear in a future version of gawk.
https://www.gnu.org/software/gawk/manual/html_node/Options.html

gerritbot added a comment.Feb 23 2019, 12:12 AM

Change 492408 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] prometheus: disable shipped node-exporter ipmitool and smartmon timers

https://gerrit.wikimedia.org/r/492408

gerritbot added a comment.Feb 25 2019, 4:08 PM

Change 492408 merged by Cwhite:
[operations/puppet@production] prometheus: disable shipped node-exporter ipmitool and smartmon timers

https://gerrit.wikimedia.org/r/492408

gerritbot added a comment.Feb 26 2019, 3:09 PM

Change 490689 merged by Cwhite:
[operations/puppet@production] hiera: upgrade prometheus-node-exporter to 0.17 in codfw

https://gerrit.wikimedia.org/r/490689

gerritbot added a comment.Feb 26 2019, 3:21 PM

Change 493047 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] prometheus: require package before disabling services

https://gerrit.wikimedia.org/r/493047

gerritbot added a comment.Feb 26 2019, 3:26 PM

Change 493047 merged by Cwhite:
[operations/puppet@production] prometheus: require package before masking services

https://gerrit.wikimedia.org/r/493047

gerritbot added a comment.Feb 27 2019, 4:31 PM

Change 493269 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] prometheus: clean up node exporter transition code

https://gerrit.wikimedia.org/r/493269

gerritbot added a comment.Feb 27 2019, 9:58 PM

Change 489325 abandoned by Cwhite:
prometheus: post-upgrade node-exporter cleanup

https://gerrit.wikimedia.org/r/489325

gerritbot added a comment.Mar 18 2019, 9:42 AM

Change 497252 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Don't set a fixed package version for prometheus-node-exporter on buster

https://gerrit.wikimedia.org/r/497252

gerritbot added a comment.Mar 18 2019, 10:08 AM

Change 497252 merged by Muehlenhoff:
[operations/puppet@production] Don't set a fixed package version for prometheus-node-exporter on buster

https://gerrit.wikimedia.org/r/497252

fgiunchedi mentioned this in T208875: Update prometheus-node-exporter NTP metrics.Mar 18 2019, 2:18 PM
gerritbot added a comment.Mar 26 2019, 4:48 PM

Change 490690 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] hiera: upgrade prometheus-node-exporter to 0.17 in eqiad

https://gerrit.wikimedia.org/r/490690

gerritbot added a comment.Mar 26 2019, 5:31 PM

Change 490690 merged by Cwhite:
[operations/puppet@production] hiera: upgrade prometheus-node-exporter to 0.17 in eqiad

https://gerrit.wikimedia.org/r/490690

gerritbot added a comment.Mar 27 2019, 10:09 PM

Change 499667 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] prometheus: clean up node exporter transition code

https://gerrit.wikimedia.org/r/499667

gerritbot added a comment.Mar 27 2019, 10:11 PM

Change 493269 abandoned by Cwhite:
prometheus: clean up node exporter transition code

Reason:
superseded

https://gerrit.wikimedia.org/r/493269

gerritbot added a comment.Mar 27 2019, 10:15 PM

Change 488593 abandoned by Cwhite:
hiera: install node exporter 0.17 in beta

Reason:
no longer needed

https://gerrit.wikimedia.org/r/488593

gerritbot added a comment.Mar 28 2019, 3:46 PM

Change 499667 merged by Cwhite:
[operations/puppet@production] prometheus: clean up node exporter transition code

https://gerrit.wikimedia.org/r/499667

colewhite updated the task description. (Show Details)Mar 28 2019, 11:13 PM
colewhite closed this task as Resolved.Apr 1 2019, 6:43 PM
colewhite updated the task description. (Show Details)
CDanis added a comment.Jul 1 2019, 8:40 PM

I think I just found another one that was missed: https://grafana.wikimedia.org/d/000000545/ganeti

colewhite added a comment.Jul 1 2019, 10:52 PM

@CDanis thanks for the heads up. Should look better now.

gerritbot added a comment.Jul 26 2019, 8:13 PM

Change 490223 abandoned by Cwhite:
hiera: upgrade prometheus-node-exporter to 0.17 in esams

https://gerrit.wikimedia.org/r/490223

gerritbot added a comment.Jul 26 2019, 8:13 PM

Change 490224 abandoned by Cwhite:
hiera: upgrade prometheus-node-exporter to 0.17 in esams

https://gerrit.wikimedia.org/r/490224

colewhite mentioned this in T275618: Kubernetes 1.16 dropped deprecated cadvisor metric labels pod_name and container_name.Feb 24 2021, 4:04 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits