As mentioned in T258013, we want to capture data on civicrm_contribution_recur so that we can monitor and possibly alert on it. This can be captured using the following query:
SELECT count(*) FROM civicrm.civicrm_contribution_recur WHERE contribution_status_id = 14;
Please find an appropriate place to collect this into prometheus and start graphing it.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | MBeat33 | T258013 Fix stuck recurring contributions, add alarm | |||
| Resolved | Dwisehaupt | T258204 Capture metric for civicrm_contribution_recur in processing state |
I have a commit staged with the fundraising_query_collector.
The collector will run mysql queries to collect stats on values desired from mysql. The queries will run using the .my.cnf of the prometheus user so appropriate db permissions are needed for that user. Individual queries can be disabled by setting enabled = 0. The initial queries it will run are ones that we are currently monitoring with icinga.
Still need to sort out how we want the db permissions structured before rolling out.
DB permissions have been fixed and this has been deployed. Current queries we are collecting stats on are visible here: https://frmon.frdev.wikimedia.org/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Prometheus%22,%7B%22expr%22:%22recurring_gc_failures_missed%22%7D,%7B%22expr%22:%22recurring_gc_jobs_required%22%7D,%7B%22expr%22:%22recurring_gc_schedule_sanity%22%7D,%7B%22expr%22:%22missing_thank_yous%22%7D,%7B%22expr%22:%22civicrm_contribution_recur%22%7D,%7B%22ui%22:%5Btrue,true,true,%22none%22%5D%7D%5D
This is just using the same naming as the current nagios alert collection at this point. We may want to adjust them to a hierarchy that makes more sense in the near future.
[frack::puppet::private] 79dc600 Shift exporter account to use dbstats user on staging [frack::puppet::private] 498b70f Fix password for dbstats user [frack::puppet] 8607bb22 Add fundraising query collector for prometheus [frack::puppet] 375849f3 Enable fundraising_query_collector on staging role [frack::puppet::private] 50948b7 Add dbstats selects to civicrm.wmf_contribution_extra