Page MenuHomePhabricator

Measure donation processing rate (once we decide what that is)
Open, MediumPublic

Description

Currently, we use this graph to indicate our donation processing rate however, the data behind is graph is actually measuring how quickly previously processed donation queue messages are imported into CiviCRM. This doesn't feel like a typical donation processing rate metric. Instead, we should rename this to something more accurate, like our 'Save To CRM rate.'

Now, to more accurately measure our donation processing rate, we need to decide what we mean by donation processing rate. More specifically, we need to decide which part of the donation pipeline we want to measure.

Here are some suggestions of points in the flow that we could measure to derive processing rates from:

  1. Gateway Result To CRM Save Rate - The time between the gateway giving us a result back and the donation being imported into the CRM
    • We currently capture this under the metric average_transaction_age in Prometheus. I added a test graph for this here Experimental Processing Rate, but the output looks questionable and needs verifying.
  2. Submission to Save Rate - The time between the donor clicking submit and the completed donation being imported into the CRM
    • This feels like the truest measure of our overall processing rate. However, this would also include the time the donation is sitting on the Redis queue in between the front-end gateway transaction and the backend CRM save.
  3. Submission to Thank You Email Rate - The same as above but adding on the period afterwards, which captures the time it takes for the thank you email to be sent
    • This feels like overkill, but it could give us timings for the full donor's donation lifecycle.
  4. Gateway Processing Rate - The time between the donor clicking submit and the response coming back from a gateway
    • This one isn't really measuring how fast we do things, but it could be useful to indicate one of our gateways is backing up.
  5. Frontend Processing Rate - The time between the donor clicking submit and the donation ending up on the Redis queue
  6. Backend Processing Rate - AKA our Save-To-CRM rate!

We could discuss other options or even consider slicing and dicing different timings to come up with a more desirable metric.

Related Objects

Event Timeline

Restricted Application added a subscriber: Aklapper. ยท View Herald TranscriptJul 17 2023, 3:23 PM
jgleeson updated the task description. (Show Details)
jgleeson updated the task description. (Show Details)

I think we have historically focused on 'Save to CRM' rate because that's where we do the most database activity, so it's the limiting factor in how fast we can show results and get the TY letters out. For a while, the most frequent causes of these things slowing down were DB changes in CiviCRM that slowed our usual queries or changes in our code that made us do more DB work for each transaction. Focusing on just the 'Save to CRM' rate made those slowdowns easier to detect.

Naming-wise, I agree that 'donation processing rate' is a bit misleading for just that step. I guess I would consider your #3 (Submission to Thank You Email Rate) as a good metric for 'donation processing rate'

AKanji-WMF subscribed.

Aiming to have in place well before 6enC

XenoRyet lowered the priority of this task from High to Medium.Nov 1 2023, 6:01 PM

We scrape some metrics from logs that could be informative and/or useful for comparison re. this task: https://frmon.wikimedia.org/d/xbcwsjY4k/payment-transaction-outcomes?orgId=1&refresh=1m

The metrics aren't perfect, for example I'm not sure they handle donations from Amazon and some of the recurring stuff. But they're based on log messages per minute instead of per queue run, which could be useful.

Dwisehaupt subscribed.

Pushing out to sprint +2 but need to review if/how we want to proceed on this.