Background

Continuing T416949 we will merge a first version of the maintenance script. This will allow us to demonstrate the transfer of information from CiviCRM end to end.

User story

As a user I want my donor status synced with my account so that the interface tailors itself to me.

Precursors

• The CSV output will be finalized (refer to https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/fundraising/tools/+/refs/heads/master/mediawiki_donor_export/)
• The donor segments will be confirmed
• There will be a user preference that can be written to (T416946: Add a preference for receiving donor information from CiviCRM)
• Discuss whether we should store a JSON blob or an integer (given in future we will likely need to store the date of consent as well)

Requirements

• Code should live in WikimediaCustomizations as we will need to share constants with preferences.
• The script is merged and can be manually run in production

The script will process every file in the provided CSV as follows:

• Make the maintenance script accept the CSV pathe on standard input. This is really important for making it k8s friendly (it will likely be located at /var/lib/fundraising-data-uploader)
  • If the e-mail address does not correspond with any user in the user table, the row will be discarded.
• If the e-mail address corresponds with a user who has not consented to donor identification the row will be discarded.
• If the e-mail address corresponds with a user that has not confirmed their email address the row will be discarded.
  • If the email address corresponds with a user that has confirmed their email address and given consent, a donor preference will be derived from the donor segment provided by the CSV. This will be a json with the donor status value (since preferences are stored as a blob Amir recommended this is preferable to multiple preferences). In future the blob will allow us to also store date of consent and possibly version number if we need to make breaking changes
• After the CSV has been fully processed (and is not using dry-run), the CSV will be discarded (or will notify an external service that this has been done)
• The script takes a dry-run parameter to allow us to update

Descoped for follow up in T424089: Discuss batching requirements for finalizing the donor preference sync script

[] The script can operate safely on ~250k contacts normally, ~1m when donation volume is higher. See {T418164#11643063}
[] The script should be able to handle large CSVs e.g. 1 million rows in a way that doesn't cause problems for SRE.

BDD

• For QA engineer to fill out

Test Steps

• For QA engineer to fill out

Design

• Add mockups and design requirements

Acceptance criteria

• We have a merged patch that is capable of parsing the file exported by CiviCRM
• The script can process large CSVs and has safeguards in place to avoid writing / reading too many rows from the database.

Communication criteria - does this need an announcement or discussion?

• Add communication criteria

Rollback plan

• What is the rollback plan in production for this task if something goes wrong?

This task was created by Version 1.2.0 of the Web team task template using phabulous