Page MenuHomePhabricator

mtail 3.0.0-rc35 doesn't support the histogram type in -oneshot mode.
Closed, ResolvedPublic

Description

The current production version of mtail doesn't support histograms in -oneshot mode, thus making it impossible to use our current testing framework:

$ mtail -one_shot -logtostderr -progs mediawiki_access_log.mtail  -logs ../test/logs/mediawiki_access_log.test 
[CUT]
Metrics store:E0924 11:18:13.353454   81132 main.go:178] failed to marshal metrics into json: json: unsupported value: +Inf

This happens because json marshalling automatically assumes the bucket field of the map to be an integer, and "+Inf" isn't an integer. See https://github.com/google/mtail/issues/223

The fix is pretty simple https://github.com/google/mtail/commit/4e8b829b9333346fc82c8a5ee253c07c8a87e105 and is included in mtail 3.0.0-rc36 AFAICT.

The work on the API gateway SLOs (T254916) is blocked on us either backporting that single fix or moving to mtail 3.0.0-rc36.

Event Timeline

Joe created this task.Sep 24 2020, 9:42 AM
ArielGlenn triaged this task as High priority.Mon, Sep 28, 9:52 AM
colewhite moved this task from Inbox to In progress on the observability board.

Change 631501 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] mtail: upgrade mtail across the fleet to 3.0.0~rc35-3+wmf3

https://gerrit.wikimedia.org/r/631501

Change 631502 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[integration/config@master] Bump mtail to 3.0.0~rc35-3+wmf3

https://gerrit.wikimedia.org/r/631502

Change 631502 merged by jenkins-bot:
[integration/config@master] Bump mtail to 3.0.0~rc35-3+wmf3

https://gerrit.wikimedia.org/r/631502

Change 631734 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Revert "Bump mtail to 3.0.0~rc35-3+wmf3"

https://gerrit.wikimedia.org/r/631734

Change 631734 merged by jenkins-bot:
[integration/config@master] Revert "Bump mtail to 3.0.0~rc35-3+wmf3"

https://gerrit.wikimedia.org/r/631734

Change 631735 had a related patch set uploaded (by Hashar; owner: Cwhite):
[integration/config@master] Bump mtail to 3.0.0~rc35-3+wmf3

https://gerrit.wikimedia.org/r/631735

Change 631735 merged by jenkins-bot:
[integration/config@master] Bump mtail to 3.0.0~rc35-3+wmf3

https://gerrit.wikimedia.org/r/631735

Mentioned in SAL (#wikimedia-releng) [2020-10-02T14:39:32Z] <hashar> Successfully tagged docker-registry.discovery.wmnet/releng/operations-puppet:0.7.7 # T263728

Change 631787 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: update operations-puppet to 0.7.7 image

https://gerrit.wikimedia.org/r/631787

Change 631787 merged by jenkins-bot:
[integration/config@master] jjb: update operations-puppet to 0.7.7 image

https://gerrit.wikimedia.org/r/631787

Change 631501 merged by Cwhite:
[operations/puppet@production] mtail: upgrade mtail across the fleet to 3.0.0~rc35-3+wmf3

https://gerrit.wikimedia.org/r/631501

colewhite closed this task as Resolved.Tue, Oct 6, 2:37 PM

Patched mtail rolling out to the fleet this morning. Please let me know if you encounter any related issue.