Page MenuHomePhabricator

Investigate data quality payment information in donations
Closed, ResolvedPublic2 Estimated Story Points

Description

  • Create a list of donations that can't be migrated (legacy CC provider or missing legacy data) and ask PM what to do about them. Options are
    • Delete
    • Keep with fake payment entry (maybe reuse payment ID)
    • Keep and hope they are never loaded with payment information (which would explode)

Implementation notes:

Suggested approach: create a class that takes donation data (including legacy blob), determines the legacy mapper according to payment type and booking status and tries to create a new payment instance / validates the transaction data according to the rules in the payment domain.

Create a small script that reads the donation table and calls migrator for each row, returning a list/CSV of "bad" donations so we can determine what to do with them

This is a one-off script, no need for code quality

Event Timeline

gabriel-wmde renamed this task from Create database migration for donations to Investigate data quality payment information in donations.Mar 30 2022, 1:00 PM
gabriel-wmde updated the task description. (Show Details)
gabriel-wmde set the point value for this task to 8.
gabriel-wmde changed the point value for this task from 8 to 2.May 2 2022, 9:22 AM

Result of investigation

ErrorDonations affectedDate Range
Legacy transfer code pattern, omitting :533971 (12.84%)(2011-09-30 - 2018-11-12)
Skipped MBK payment :33165 (0.80%)(2011-10-12 - 2013-09-19)
Booked Paypal payment without timestamp, restored from log :7903 (0.19%)(2011-11-25 - 2015-10-07)
Booked Paypal payment without timestamp, assumed donation date:393 (0.01%)(2011-11-25 - 2015-10-07)
Booked Paypal without payer ID, booked by admins :1 (0.00%)(2011-11-25 - 2011-11-25)
Converted empty amount to 0 :76 (0.00%)(2011-11-26 - 2015-10-07)
Invalid date format for booked PayPal, reformatted existing date:2157097 (51.85%)(2011-11-27 - 2020-12-29)
Booked Credit card without transaction ID, booked by admins :5 (0.00%)(2012-11-24 - 2013-11-14)
Booked Paypal without payer ID, 2015 error period :125 (0.00%)(2015-09-28 - 2015-10-07)
Booked Credit card without transaction ID, 2015 error period:24 (0.00%)(2015-09-30 - 2015-10-07)
Sofort donation with empty valuation date, using donation date:274 (0.01%)(2017-10-13 - 2017-11-08)
Recurring interval for sofort payment, set to one-time :597 (0.01%)(2017-10-26 - 2020-11-19)

Open questions:

  • What to do with MBK donations? - Create Credit card donations instead of skipping?
  • Should we change the expected date format for Paypal to the one that PayPal actually sends?
  • What to do about transfer codes? Suggestion: Copy them to new transfer code table, anonymise them in payments
kai.nissen claimed this task.