Description

Grafana looks to be intermittently displaying misleading data as a result of Prometheus serving old metric data by default, in the absence of new data.

This bug is visible within the grafana dash "donations queued to civicrm", which is displaying PayPal donation messages for new data points which there have actually been no paypal donations.

To observe yourself:
Jump on to civi1001, and in your terminal run watch -n1 cat /var/spool/prometheus/donations.prom while watching in another window https://grafana.wikimedia.org/dashboard/db/fundraising-overview?orgId=1&from=now-15m&to=now&panelId=15&fullscreen&refresh=1m and compare the two sources

Event Timeline

jgleeson created this task.Dec 19 2017, 5:20 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 19 2017, 5:20 PM

jgleeson moved this task from Triage to Completed in Q2 2017-18 on the Fundraising-Backlog board.Dec 19 2017, 5:21 PM

jgleeson updated the task description. (Show Details)Dec 19 2017, 5:29 PM

jgleeson updated the task description. (Show Details)Dec 19 2017, 5:34 PM

jgleeson updated the task description. (Show Details)

jgleeson added a project: Fundraising Sprint Winter Wanderland.Dec 19 2017, 5:36 PM

@jgleeson is right, wonky behavior, looks like Prometheus doesn't deal well with a highly variant list of metrics: https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics

Apparently best practice is to export a zero for an empty queue

Ejegg moved this task from Backlog to Doing on the Fundraising Sprint Winter Wanderland board.Dec 22 2017, 5:26 PM

• DStrine moved this task from Completed in Q2 2017-18 to Completed in Q3 2017-18 on the Fundraising-Backlog board.Jan 2 2018, 6:11 PM

• DStrine added a project: Fundraising Sprint Asymmetrical Earth Theory.Jan 2 2018, 9:18 PM

jgleeson moved this task from Backlog to Doing on the Fundraising Sprint Asymmetrical Earth Theory board.Jan 8 2018, 8:38 PM

Change 403305 had a related patch set uploaded (by Jgleeson; owner: Jgleeson):
[wikimedia/fundraising/crm@master] T183275 WIP. Implementation working but not tests..due to flaky DonationStats API.

https://gerrit.wikimedia.org/r/403305

gerritbot added a project: Patch-For-Review.Jan 9 2018, 10:05 PM

jgleeson moved this task from Doing to Review on the Fundraising Sprint Asymmetrical Earth Theory board.Jan 11 2018, 3:54 PM

jgleeson moved this task from Review to Pending Deployment on the Fundraising Sprint Asymmetrical Earth Theory board.Jan 15 2018, 2:43 PM

• DStrine added a project: Fundraising Sprint Bermuda Rhombus (where things disappear then reappear).Jan 16 2018, 9:28 PM

jgleeson closed this task as Resolved.Jan 30 2018, 9:02 PM

jgleeson moved this task from Backlog to Done on the Fundraising Sprint Bermuda Rhombus (where things disappear then reappear) board.

Bugfix: Prometheus serving old data in the absence of new data, to GrafanaClosed, ResolvedPublicActions

Description

Event Timeline

Bugfix: Prometheus serving old data in the absence of new data, to Grafana
Closed, ResolvedPublic
Actions