Instead of just writing stats once at the end, it should write the processing rate stats each minute as well as once at the end.
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Export donations qc stats every minute | wikimedia/fundraising/crm | master | +26 -0 |
Related Objects
Related Objects
- Mentioned Here
- T323657: Make Donations queue consumer wait for new messages
Event Timeline
Comment Actions
Change 857074 had a related patch set uploaded (by Ejegg; author: Ejegg):
[wikimedia/fundraising/crm@master] Export donations qc stats every minute
Comment Actions
As discussed in IRC, the approach in the attached patch (writing every minute plus once at the end of the run) would risk overwriting good statistics before they're published if e.g. the run lasted 61 seconds. There would be a file with stats for the first 60 seconds, that would quickly be overwritten with a file with stats for the last second (much less informative), likely before the first file is published to prometheus. We could mitigate that by making the queue consumer wait for new messages (T323657) so it always runs the full almost-three minutes, but as mentioned on that task we would need an update to the stats collector library to discount waiting time.
Comment Actions
Adding note from Elliot in IRC - this task is stalled; was trying to fix a data visualization problem that we ended up solving just by deleting the misleading graph