Page MenuHomePhabricator

Estimate the proportion of people who send us complaints vs are affected by an issue
Closed, ResolvedPublic

Description

It would be nice to have a ballpark multiplier we can use when reasoning about how many people are affected by issues. An easy first step would be to look at a few cases where there was a way to get a count of exact number of people affected, and compare how many complaints we received.

Event Timeline

awight raised the priority of this task from to Needs Triage.
awight updated the task description. (Show Details)
awight added subscribers: awight, MBeat33, CCogdill_WMF.

I suspect we need a couple of different measurements here. Some errors
don't display the instructions for reporting an issue.

For starters here are two batches of GC100 Auth Fail errors (usually a bank doing maintenance causes this). These errors occur after donors submit their data. On 1/28/2016, 67 donors got this error and 2 contacted us, for a 2.98% complaint rate. On 10/18/2015, 16 out of 542 donors contacted us, which is 2.95% - consistent! I will keep looking for more unique errors like this to compare. It's also possible that errors that occur before donors submit their data, like iframe errors, may have a different complaint rate.

Two recent datapoints from recurring donation errors:

{T142312} 1243 donors affected, 25 tickets in Zendesk: 2% contact rate

T144489: Date bug on Aug 31st causing recurring Ingenico donations to process in error 1303 donors affected, 10 tickets (so far): .77% contact rate

For {T145776} of the 261 affected donors, only 2 (.76%) have so far contacted us via Zendesk to confirm their donations.

In my time managing problems like this, Ive never seen a CS/QA/Engineering team actually accomplish this model. You can't really predict an accurate number of affected users. you can measure quick changes in a complaint to indicate something is happening. I would like to decline this.

I still like this idea. T243356: Civi: January 2020 Ingenico recurrings processing in duplicate? had 427 affected donors, from which we received 3 complaints so far, for duplicate donations on the 9th of January. We may catch more complaints end of month, when people look at their statements.

MBeat33 added a project: FR-donorservices.
MBeat33 claimed this task.

Seems like we can close this.

For the Ingenico duplicate error, 15310 trx were sent twice to WMF, with ~440 Zendesk inquiries from affected donors = 2.87% rate