Page MenuHomePhabricator

GC error causing donors to create multiple recurring donations
Closed, ResolvedPublic4 Story Points

Description

Since 8/24 we have seen 11 duplicate GC recurrings caused by donors attempting multiple donations after their initial attempt (which they don't know were successful). Some donors indicated that they received error messages on the first attempt, others say they received no feedback from the form. The messages received include:

GC 6322742090 - 550 Permission Denied,
GC 362210930 - "I received a notice that access to that page was denied."
GC 4600617163 and 3976209062 - “Forbidden Page"

This may possibly overlap with https://phabricator.wikimedia.org/T105041. I will scan to make sure this is not also affecting one-time donations, but the 11 tickets we've seen about this have all come from recurring donors.

Updating story points to 4 to reflect work on automating refunds

Details

Related Gerrit Patches:
wikimedia/fundraising/crm : civi-4.6.9Make refund import logic reusable
wikimedia/fundraising/crm : civi-4.6.9Use Civi method to cancel recurring via qc
wikimedia/fundraising/crm : masterMake refund import logic reusable
wikimedia/fundraising/crm : masterUse Civi method to cancel recurring via qc
mediawiki/extensions/DonationInterface : masterMore refund logic tweaks
mediawiki/extensions/DonationInterface : masterCancel subscriptions programatically
mediawiki/extensions/DonationInterface : masterTweak the cancel/refund logic
mediawiki/extensions/DonationInterface : masterGlobalCollect refund API
mediawiki/extensions/DonationInterface : deploymentLimit OID reset on recurring changes, log resets
mediawiki/extensions/DonationInterface : masterLimit OID reset on recurring changes, log resets

Event Timeline

MBeat33 created this task.Aug 26 2015, 6:26 PM
MBeat33 raised the priority of this task from to Needs Triage.
MBeat33 updated the task description. (Show Details)
MBeat33 added a project: Fundraising-Backlog.
MBeat33 added a subscriber: MBeat33.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2015, 6:26 PM
Ejegg claimed this task.Aug 26 2015, 10:13 PM
Ejegg triaged this task as High priority.
Ejegg set Security to None.
Ejegg edited a custom field.

Yep, this is definitely a result of the fix to the linked ticket. I had it resetting the order id in session any time the donor comes in with a different value for 'recurring'. This was a little too aggressive and triggered when the donor returned to our site, which makes us reject them. I'm dialing back the reset so it only triggers if there is a 'recurring' parameter on the URL.

Great, thank you!

Change 234177 had a related patch set uploaded (by Ejegg):
Limit OID reset on recurring changes, log resets

https://gerrit.wikimedia.org/r/234177

Change 234177 merged by jenkins-bot:
Limit OID reset on recurring changes, log resets

https://gerrit.wikimedia.org/r/234177

Change 234200 had a related patch set uploaded (by Ejegg):
Limit OID reset on recurring changes, log resets

https://gerrit.wikimedia.org/r/234200

Change 234200 merged by jenkins-bot:
Limit OID reset on recurring changes, log resets

https://gerrit.wikimedia.org/r/234200

Ejegg added a comment.Aug 27 2015, 2:33 AM

Code fix seems good, but I see a whole lot of these in the logs. I will get a list of all txn IDs that might have been affected by this, and especially those with multiple donations. We may have to confirm some of the hanging transactions with GlobalCollect.

We haven't seen any new instances of this error. @Ejegg, if DS can help process any of the batch in the logs by communicating with donors or cancelling/refunding any duplicates, please let me know.

awight added a subscriber: awight.Aug 28 2015, 7:39 AM

We'll have to cancel the subscriptions, too.

Change 234474 had a related patch set uploaded (by Awight):
Tweak the cancel/refund logic

https://gerrit.wikimedia.org/r/234474

Change 234476 had a related patch set uploaded (by Awight):
Cancel subscriptions programatically

https://gerrit.wikimedia.org/r/234476

I've smoke tested the branch head, it seems to do the job! Time to mail ourselves more pizza, we've been talking about adding this functionality for years...

Change 234672 had a related patch set uploaded (by Ejegg):
More refund logic tweaks

https://gerrit.wikimedia.org/r/234672

Ejegg updated the task description. (Show Details)Aug 28 2015, 9:32 PM
Ejegg edited a custom field.

Change 234459 had a related patch set uploaded (by Ejegg):
GlobalCollect refund API

https://gerrit.wikimedia.org/r/234459

Change 234459 merged by jenkins-bot:
GlobalCollect refund API

https://gerrit.wikimedia.org/r/234459

Change 234474 merged by jenkins-bot:
Tweak the cancel/refund logic

https://gerrit.wikimedia.org/r/234474

Change 234476 merged by jenkins-bot:
Cancel subscriptions programatically

https://gerrit.wikimedia.org/r/234476

Change 234672 merged by jenkins-bot:
More refund logic tweaks

https://gerrit.wikimedia.org/r/234672

Leaving this in review till we make sure we closed all the associated subscriptions so we can't double-charge them next week

Perhaps, pending deployment?

Ejegg added a comment.Sep 9 2015, 11:04 PM

Update from GC
+++ Email 1:
I believe END_ORDER is the call you need.
+++ Email 2:
We are still investigating this issue. We ran some tests in Staging and we are seeing similar issues to what you reported. I should be able to get back to you soon.

Pinging on this--turns out we can't close the subscriptions on GC's side, hopefully nothing is pending in Civi...

Ejegg added a comment.Sep 16 2015, 6:44 PM

Leaving this open till I can make sure all of these are canceled in Civi

Change 239900 had a related patch set uploaded (by Ejegg):
Use Civi method to cancel recurring via qc

https://gerrit.wikimedia.org/r/239900

Ejegg added a comment.Sep 21 2015, 7:19 PM

Update: 47 of the refunded donations still need the associated subscription canceled (SQL sent to fr-tech for review). Also, none of the refunded donations show up as refunded in Civi because our GC audit parser doesn't handle refunds on recurring donations.

Ejegg added a comment.Sep 21 2015, 7:56 PM

All subscriptions have been canceled, but I still need to mark the initial payments in Civi as refunded on 8/28

Change 239900 merged by Ejegg:
Use Civi method to cancel recurring via qc

https://gerrit.wikimedia.org/r/239900

Change 240630 had a related patch set uploaded (by Ejegg):
Make refund import logic reusable

https://gerrit.wikimedia.org/r/240630

Ejegg closed this task as Resolved.Sep 24 2015, 7:05 PM
Ejegg moved this task from Pending Deployment to Done on the Fundraising Sprint Tom Waits board.

All of the refunded contributions have been marked as such in Civi, and all duplicate subscriptions are canceled.

Change 240630 merged by Ejegg:
Make refund import logic reusable

https://gerrit.wikimedia.org/r/240630

Change 249564 had a related patch set uploaded (by Eileen):
Use Civi method to cancel recurring via qc

https://gerrit.wikimedia.org/r/249564

Change 249624 had a related patch set uploaded (by Eileen):
Make refund import logic reusable

https://gerrit.wikimedia.org/r/249624

Change 249564 abandoned by Eileen:
Use Civi method to cancel recurring via qc

https://gerrit.wikimedia.org/r/249564

Change 249624 abandoned by Eileen:
Make refund import logic reusable

https://gerrit.wikimedia.org/r/249624

mmodell removed a subscriber: awight.Jun 22 2017, 9:37 PM