Page MenuHomePhabricator

Upgrade statsd_exporter to 0.9
Closed, ResolvedPublic

Description

We'll need to upgrade statsd_exporter (production and k8s) to ship an important bugfix for summaries, i.e. scaling by a factor of 1000: https://github.com/prometheus/statsd_exporter/pull/178. This is affecting amongst other things, service-runner gc metrics. Once done we'll need to audit/change dashboards too.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 11 2019, 3:25 PM
Ottomata added a project: EventBus.
Ottomata added a subscriber: Pchelolo.
Restricted Application added a project: Analytics. · View Herald TranscriptApr 11 2019, 3:28 PM
fdans moved this task from Incoming to Radar on the Analytics board.Apr 11 2019, 4:17 PM
jijiki added a subscriber: jijiki.Apr 11 2019, 8:58 PM
akosiaris triaged this task as Normal priority.Apr 12 2019, 7:09 AM
fgiunchedi moved this task from Backlog to In progress on the observability board.Apr 15 2019, 2:39 PM
fgiunchedi moved this task from Backlog to Up next on the User-fgiunchedi board.Thu, May 2, 3:01 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-07T09:51:25Z] <godog> test statsd-exporter 0.9 upgrade on deployment-imagescaler03 - T220709

Looks good so far in deployment-prep, there's a deb on boron for testing /var/cache/pbuilder/result/stretch-amd64/prometheus-statsd-exporter_0.9.0+ds1-1_amd64.deb cc @Ottomata @akosiaris

Great! I guess it just needs to go into the WMF base docker image somehow?

Great! I guess it just needs to go into the WMF base docker image somehow?

Indeed, I'm not sure about the exact procedure for that, I'll upload 0.9 to stretch-wikimedia though and upgrade production hosts.

Mentioned in SAL (#wikimedia-operations) [2019-05-08T08:55:59Z] <godog> upload prometheus-statsd-exporter 0.9.0+ds1-1 to stretch-wikimedia T220709

Change 508798 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: change swift statsd-exporter units

https://gerrit.wikimedia.org/r/508798

Change 508803 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/docker-images/production-images@master] Use prometheus-statsd-exporter 0.9

https://gerrit.wikimedia.org/r/508803

Change 508798 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: change swift statsd-exporter units

https://gerrit.wikimedia.org/r/508798

Mentioned in SAL (#wikimedia-operations) [2019-05-08T11:54:57Z] <akosiaris> bump prometheus-statsd-exporter for cxserver to 0.0.5 T220709

Change 508803 merged by Alexandros Kosiaris:
[operations/docker-images/production-images@master] Use prometheus-statsd-exporter 0.9

https://gerrit.wikimedia.org/r/508803

fgiunchedi moved this task from Up next to Doing on the User-fgiunchedi board.Mon, May 13, 8:57 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-13T13:07:34Z] <akosiaris> bump cxserver chart to 0.0.7. Renames nodejs GC stats to microseconds and bumps the biggest bucket to 100ms. T220709

Change 509836 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] eventgate: Switch GC metric to microseconds, update buckets

https://gerrit.wikimedia.org/r/509836

Change 509836 merged by Alexandros Kosiaris:
[operations/deployment-charts@master] eventgate: Switch GC metric to microseconds, update buckets

https://gerrit.wikimedia.org/r/509836

Mentioned in SAL (#wikimedia-operations) [2019-05-13T13:39:22Z] <akosiaris> bump eventgate-analytics chart to 0.0.36. Renames nodejs GC stats to microseconds and bumps the biggest bucket to 100ms. T220709

Mentioned in SAL (#wikimedia-operations) [2019-05-14T09:02:28Z] <godog> statsd_exporter 0.9 upgrade on logstash - T220709

Change 510089 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] thumbor: statsd_exporter mappings to seconds

https://gerrit.wikimedia.org/r/510089

Mentioned in SAL (#wikimedia-operations) [2019-05-14T09:15:47Z] <godog> statsd_exporter 0.9 upgrade on ores - T220709

Change 510089 merged by Filippo Giunchedi:
[operations/puppet@production] thumbor: statsd_exporter mappings to seconds

https://gerrit.wikimedia.org/r/510089

Mentioned in SAL (#wikimedia-operations) [2019-05-14T16:22:30Z] <godog> statsd_exporter 0.9 upgrade on thumbor - T220709

All production has been updated ! Leaving open for now in case there's still upgrades to be done in k8s (cc @akosiaris )

fgiunchedi moved this task from Doing to Radar on the User-fgiunchedi board.Wed, May 15, 7:30 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-15T11:29:20Z] <akosiaris> upgrade to statsd_export 0.9 for citoid T220709

Mentioned in SAL (#wikimedia-operations) [2019-05-16T08:52:40Z] <akosiaris> upgrade mathoid to statsd_exporter 0.9 T220709

akosiaris closed this task as Resolved.Thu, May 16, 8:54 AM
akosiaris claimed this task.

Every deployment that uses statsd-exporter (namely zotero & blubberoid don't) in kubernetes has been upgraded. Resolving this. Many thanks!