On parent task it became evident that pybal metrics are not removed upon monitor stop()
Description
Description
Details
Details
Related Changes in Gerrit:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | fgiunchedi | T314118 Reduce IRC flood/spam during incidents | |||
| Resolved | fgiunchedi | T320627 Alert on individual pybal backend hosts being down for a long time | |||
| Resolved | fgiunchedi | T321191 Cleanup pybal Prometheus metrics on monitor stop() |
Event Timeline
Comment Actions
Change 844469 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/debs/pybal@1.15] Clean up monitor metrics on stop()
Comment Actions
Change 845001 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/debs/pybal@1.15] Add 'pybal_server_pooled' metric
Comment Actions
Change 845001 abandoned by Filippo Giunchedi:
[operations/debs/pybal@1.15] Add 'pybal_server_pooled' metric
Reason:
Given Brandon's point about confusion re: "pool" I'm going to abandon this change for now, it can be restored at any time though
Comment Actions
Change 844469 merged by Filippo Giunchedi:
[operations/debs/pybal@1.15] Clean up monitor metrics on stop()
Comment Actions
Change 879839 had a related patch set uploaded (by Ssingh; author: Ssingh):
[operations/debs/pybal@1.15-stretch] Release 1.15.10
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2023-01-19T15:30:51Z] <sukhe> reprepro -C main include buster-wikimedia pybal_1.15.10_amd64.changes: T321191
Comment Actions
^ For posterity: building pybal on build2001 is broken because of the Python 2 dependency (among others things), so @Muehlenhoff was able to do a local build and that's what we uploaded. Thanks to him!
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2023-01-19T15:55:25Z] <sukhe> update pybal to 1.15.10 on lvs4010: T321191
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2023-01-23T14:42:20Z] <sukhe> rolling out pybal 1.15.10: T321191
Comment Actions
The version of PyBal with the above change (1.15.10) has now been rolled out to all A:lvs:
===== NODE GROUP ===== (20) lvs[2007-2010].codfw.wmnet,lvs[6001-6003].drmrs.wmnet,lvs[1017-1020].eqiad.wmnet,lvs[5004-5006].eqsin.wmnet,lvs[3005-3007].esams.wmnet,lvs[4008-4010].ulsfo.wmnet ----- OUTPUT of 'dpkg-query --show pybal' ----- pybal 1.15.10