Page MenuHomePhabricator
Create Task
Maniphest T110367

GC error causing donors to create multiple recurring donations
Closed, ResolvedPublic4 Estimated Story Points
Actions

Description

Since 8/24 we have seen 11 duplicate GC recurrings caused by donors attempting multiple donations after their initial attempt (which they don't know were successful). Some donors indicated that they received error messages on the first attempt, others say they received no feedback from the form. The messages received include:

GC 6322742090 - 550 Permission Denied,
GC 362210930 - "I received a notice that access to that page was denied."
GC 4600617163 and 3976209062 - “Forbidden Page"

This may possibly overlap with https://phabricator.wikimedia.org/T105041. I will scan to make sure this is not also affecting one-time donations, but the 11 tickets we've seen about this have all come from recurring donors.

Updating story points to 4 to reflect work on automating refunds

Details

SubjectRepoBranchLines +/-
Make refund import logic reusablewikimedia/fundraising/crmcivi-4.6.9+89 -74
Use Civi method to cancel recurring via qcwikimedia/fundraising/crmcivi-4.6.9+68 -68
Make refund import logic reusablewikimedia/fundraising/crmmaster+89 -74
Use Civi method to cancel recurring via qcwikimedia/fundraising/crmmaster+25 -13
More refund logic tweaksmediawiki/extensions/DonationInterfacemaster+64 -15
Cancel subscriptions programaticallymediawiki/extensions/DonationInterfacemaster+83 -0
Tweak the cancel/refund logicmediawiki/extensions/DonationInterfacemaster+41 -17
GlobalCollect refund APImediawiki/extensions/DonationInterfacemaster+159 -3
Limit OID reset on recurring changes, log resetsmediawiki/extensions/DonationInterfacedeployment+18 -3
Limit OID reset on recurring changes, log resetsmediawiki/extensions/DonationInterfacemaster+18 -3
Customize query in gerrit

Related Objects

Event Timeline

MBeat33 created this task.Aug 26 2015, 6:26 PM
MBeat33 raised the priority of this task from to Needs Triage.
MBeat33 updated the task description. (Show Details)
MBeat33 added a project: Fundraising-Backlog.
MBeat33 subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2015, 6:26 PM
• atgo added a project: Recurring-Donations.Aug 26 2015, 6:32 PM
• atgo subscribed.
Ejegg claimed this task.Aug 26 2015, 10:13 PM
Ejegg triaged this task as High priority.
Ejegg added a project: Fundraising Sprint Rowlf the Dog.
Ejegg set Security to None.
Ejegg edited a custom field.
Ejegg added a comment.Aug 26 2015, 10:53 PM

Yep, this is definitely a result of the fix to the linked ticket. I had it resetting the order id in session any time the donor comes in with a different value for 'recurring'. This was a little too aggressive and triggered when the donor returned to our site, which makes us reject them. I'm dialing back the reset so it only triggers if there is a 'recurring' parameter on the URL.

MBeat33 added a comment.Aug 26 2015, 10:57 PM

Great, thank you!

gerritbot subscribed.Aug 26 2015, 11:09 PM

Change 234177 had a related patch set uploaded (by Ejegg):
Limit OID reset on recurring changes, log resets

https://gerrit.wikimedia.org/r/234177

gerritbot added a project: Patch-For-Review.Aug 26 2015, 11:09 PM
gerritbot added a comment.Aug 26 2015, 11:28 PM

Change 234177 merged by jenkins-bot:
Limit OID reset on recurring changes, log resets

https://gerrit.wikimedia.org/r/234177

Ejegg mentioned this in rMEXTe72459331370: Updated mediawiki/extensions Project: mediawiki/extensions/DonationInterface….Aug 26 2015, 11:28 PM
Ejegg mentioned this in rEDOIe7d7638b028b: Limit OID reset on recurring changes, log resets.
gerritbot added a comment.Aug 27 2015, 1:24 AM

Change 234200 had a related patch set uploaded (by Ejegg):
Limit OID reset on recurring changes, log resets

https://gerrit.wikimedia.org/r/234200

gerritbot added a comment.Aug 27 2015, 1:26 AM

Change 234200 merged by jenkins-bot:
Limit OID reset on recurring changes, log resets

https://gerrit.wikimedia.org/r/234200

Ejegg mentioned this in rEDOI712cfe216b9e: Limit OID reset on recurring changes, log resets.Aug 27 2015, 1:26 AM
Ejegg mentioned this in rMW59a08b48d2af: Updated mediawiki/core Project: mediawiki/extensions/DonationInterface….
Ejegg mentioned this in rMW0765a0cd8b02: Updated mediawiki/core Project: mediawiki/extensions/DonationInterface….
Ejegg mentioned this in rMWff27724f1aa4: Updated mediawiki/core Project: mediawiki/extensions/DonationInterface….
awight moved this task from Backlog to Doing on the Fundraising Sprint Rowlf the Dog board.Aug 27 2015, 1:46 AM
awight moved this task from Triage to Closed Sprint Work Q1 1516 on the Fundraising-Backlog board.Aug 27 2015, 2:20 AM
Ejegg added a comment.Aug 27 2015, 2:33 AM

Code fix seems good, but I see a whole lot of these in the logs. I will get a list of all txn IDs that might have been affected by this, and especially those with multiple donations. We may have to confirm some of the hanging transactions with GlobalCollect.

MBeat33 added a comment.Aug 27 2015, 1:57 PM

We haven't seen any new instances of this error. @Ejegg, if DS can help process any of the batch in the logs by communicating with donors or cancelling/refunding any duplicates, please let me know.

awight subscribed.Aug 28 2015, 7:39 AM

We'll have to cancel the subscriptions, too.

gerritbot added a comment.Aug 28 2015, 7:47 AM

Change 234474 had a related patch set uploaded (by Awight):
Tweak the cancel/refund logic

https://gerrit.wikimedia.org/r/234474

gerritbot added a comment.Aug 28 2015, 7:59 AM

Change 234476 had a related patch set uploaded (by Awight):
Cancel subscriptions programatically

https://gerrit.wikimedia.org/r/234476

awight added a comment.Aug 28 2015, 8:02 AM

I've smoke tested the branch head, it seems to do the job! Time to mail ourselves more pizza, we've been talking about adding this functionality for years...

gerritbot added a comment.Aug 28 2015, 8:59 PM

Change 234672 had a related patch set uploaded (by Ejegg):
More refund logic tweaks

https://gerrit.wikimedia.org/r/234672

Ejegg updated the task description. (Show Details)Aug 28 2015, 9:32 PM
Ejegg edited a custom field.
gerritbot added a comment.Aug 28 2015, 11:17 PM

Change 234459 had a related patch set uploaded (by Ejegg):
GlobalCollect refund API

https://gerrit.wikimedia.org/r/234459

Ejegg moved this task from Doing to Review on the Fundraising Sprint Rowlf the Dog board.Aug 31 2015, 4:36 PM
gerritbot added a comment.Aug 31 2015, 5:36 PM

Change 234459 merged by jenkins-bot:
GlobalCollect refund API

https://gerrit.wikimedia.org/r/234459

Ejegg mentioned this in rMEXT3af927496ef6: Updated mediawiki/extensions Project: mediawiki/extensions/DonationInterface….Aug 31 2015, 5:36 PM
awight mentioned this in rEDOI64ad7e908fd3: GlobalCollect refund API.Aug 31 2015, 5:37 PM
gerritbot added a comment.Aug 31 2015, 5:39 PM

Change 234474 merged by jenkins-bot:
Tweak the cancel/refund logic

https://gerrit.wikimedia.org/r/234474

awight mentioned this in rMEXT7f7a39e2f4c6: Updated mediawiki/extensions Project: mediawiki/extensions/DonationInterface….Aug 31 2015, 5:40 PM
awight mentioned this in rEDOI215fd211e830: Tweak the cancel/refund logic.
gerritbot added a comment.Aug 31 2015, 5:41 PM

Change 234476 merged by jenkins-bot:
Cancel subscriptions programatically

https://gerrit.wikimedia.org/r/234476

awight mentioned this in rMEXT4973c2ef9143: Updated mediawiki/extensions Project: mediawiki/extensions/DonationInterface….Aug 31 2015, 5:41 PM
awight mentioned this in rEDOI0576848149c5: Cancel subscriptions programatically.
gerritbot added a comment.Aug 31 2015, 6:02 PM

Change 234672 merged by jenkins-bot:
More refund logic tweaks

https://gerrit.wikimedia.org/r/234672

Ejegg mentioned this in rMEXT5853c96059b3: Updated mediawiki/extensions Project: mediawiki/extensions/DonationInterface….Aug 31 2015, 6:02 PM
awight mentioned this in rEDOI575b2eb80fcb: More refund logic tweaks.Aug 31 2015, 6:02 PM
Ejegg moved this task from Review to Done on the Fundraising Sprint Rowlf the Dog board.Sep 2 2015, 10:19 PM
Ejegg moved this task from Done to Review on the Fundraising Sprint Rowlf the Dog board.Sep 2 2015, 10:27 PM

Leaving this in review till we make sure we closed all the associated subscriptions so we can't double-charge them next week

awight added a comment.Sep 2 2015, 10:43 PM

Perhaps, pending deployment?

Ejegg added a project: Fundraising Sprint Snoop (Dogg|Lion).Sep 3 2015, 8:15 PM
Ejegg moved this task from Backlog to Review on the Fundraising Sprint Snoop (Dogg|Lion) board.Sep 3 2015, 8:22 PM
Ejegg added a comment.Sep 9 2015, 11:04 PM

Update from GC
+++ Email 1:
I believe END_ORDER is the call you need.
+++ Email 2:
We are still investigating this issue. We ran some tests in Staging and we are seeing similar issues to what you reported. I should be able to get back to you soon.

awight added a comment.Sep 14 2015, 10:23 PM

Pinging on this--turns out we can't close the subscriptions on GC's side, hopefully nothing is pending in Civi...

Ejegg added a comment.Sep 16 2015, 6:44 PM

Leaving this open till I can make sure all of these are canceled in Civi

• DStrine added a project: Fundraising Sprint Tom Waits.Sep 17 2015, 1:12 AM
Ejegg moved this task from Backlog to Review on the Fundraising Sprint Tom Waits board.Sep 17 2015, 2:36 PM
Ejegg moved this task from Review to Doing on the Fundraising Sprint Tom Waits board.Sep 21 2015, 5:00 PM
gerritbot added a comment.Sep 21 2015, 6:40 PM

Change 239900 had a related patch set uploaded (by Ejegg):
Use Civi method to cancel recurring via qc

https://gerrit.wikimedia.org/r/239900

Ejegg added a comment.Sep 21 2015, 7:19 PM

Update: 47 of the refunded donations still need the associated subscription canceled (SQL sent to fr-tech for review). Also, none of the refunded donations show up as refunded in Civi because our GC audit parser doesn't handle refunds on recurring donations.

Ejegg added a comment.Sep 21 2015, 7:56 PM

All subscriptions have been canceled, but I still need to mark the initial payments in Civi as refunded on 8/28

Ejegg moved this task from Doing to Review on the Fundraising Sprint Tom Waits board.Sep 21 2015, 9:56 PM
gerritbot added a comment.Sep 24 2015, 12:17 AM

Change 239900 merged by Ejegg:
Use Civi method to cancel recurring via qc

https://gerrit.wikimedia.org/r/239900

awight mentioned this in rWFCGcb6248e1a8dd: Use Civi method to cancel recurring via qc.Sep 24 2015, 12:19 AM
Ejegg mentioned this in rWFCG795d4b6df437: Import refund data from CSV.Sep 24 2015, 1:12 AM
gerritbot added a comment.Sep 24 2015, 1:29 AM

Change 240630 had a related patch set uploaded (by Ejegg):
Make refund import logic reusable

https://gerrit.wikimedia.org/r/240630

Ejegg moved this task from Review to Pending Deployment on the Fundraising Sprint Tom Waits board.Sep 24 2015, 5:18 PM
Ejegg closed this task as Resolved.Sep 24 2015, 7:05 PM
Ejegg moved this task from Pending Deployment to Done on the Fundraising Sprint Tom Waits board.

All of the refunded contributions have been marked as such in Civi, and all duplicate subscriptions are canceled.

Eileenmcnaughton mentioned this in rWFCG5d6845dd08fa: Use Civi method to cancel recurring via qc.Sep 25 2015, 4:40 AM
Eileenmcnaughton mentioned this in rWFCGfbd71036a7f3: Import refund data from CSV.
awight mentioned this in rWFCGf195411e6fdc: Merge from master : Revert commits on wrong branch.Sep 28 2015, 9:25 PM
gerritbot added a comment.Oct 15 2015, 7:42 PM

Change 240630 merged by Ejegg:
Make refund import logic reusable

https://gerrit.wikimedia.org/r/240630

gerritbot added a comment.Oct 28 2015, 9:28 PM

Change 249564 had a related patch set uploaded (by Eileen):
Use Civi method to cancel recurring via qc

https://gerrit.wikimedia.org/r/249564

gerritbot added a comment.Oct 28 2015, 11:22 PM

Change 249624 had a related patch set uploaded (by Eileen):
Make refund import logic reusable

https://gerrit.wikimedia.org/r/249624

gerritbot added a comment.Oct 29 2015, 12:19 AM

Change 249564 abandoned by Eileen:
Use Civi method to cancel recurring via qc

https://gerrit.wikimedia.org/r/249564

gerritbot added a comment.Oct 29 2015, 12:29 AM

Change 249624 abandoned by Eileen:
Make refund import logic reusable

https://gerrit.wikimedia.org/r/249624

• mmodell removed a subscriber: awight.Jun 22 2017, 9:37 PM
• DStrine edited projects, added Fr-tech-archived-from-FY-2015/16; removed Fundraising-Backlog.Jan 11 2018, 9:39 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits