Page MenuHomePhabricator

CiviCRM should export a CSV that can be used in a MediaWiki maintenance script
Closed, ResolvedPublic

Description

Background

As part of the donor identification project we will need to prepare a script that exports e-mail addresses and their corresponding CiviCRM

User story

As a user who has consent donor identification I want MediaWiki have awareness of how recently I last donated so that it can tailor my experience accordingly.

Requirements

  • The export should not contain any duplicate rows. I defer to you on how to handle duplicates (e.g. consolidate them or discard rows based on segment priority)
  • The export only transfers records that have changed since the last run
  • The export can be transferred securely to a TBC machine for processing in MediaWiki
  • We have finalized the expected structure of the export and its columns. It should be a CSV that is formatted such that it can consistently parsed by another script (e.g. we can ensure column 1 always corresponds to an email address and won't change 3 weeks later)
  • The export should include only the donor segment number
  • A decision should be made about whether to include the donors "do not solicit" field in case a future version wants to make use of this (either as a donor segment or a distinct field)
  • The exported file doesn't impede us transferring information in the opposite direction in future

Out of scope

  • We can assume that encryption will be handled at a later date.
  • We currently do know where the file will be exported to. This will be worked out later.
  • We do not need to make decisions around how the export will be used
  • Optimizations - we will evaluate these later once we understand the volume and cadence we are dealing with.

BDD

  • For QA engineer to fill out

Test Steps

  • For QA engineer to fill out

Communication criteria - does this need an announcement or discussion?

  • Add communication criteria

This task was created by Version 1.2.0 of the Web team task template using phabulous

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change #1237965 had a related patch set uploaded (by Jgleeson; author: Jgleeson):

[wikimedia/fundraising/tools@master] Add MediaWiki donor status export prototype

https://gerrit.wikimedia.org/r/1237965

Change #1240084 had a related patch set uploaded (by Lars SG; author: Lars SG):

[wikimedia/fundraising/tools@master] Include opted out contacts in silverpop_export & silverpop_export_view_full

https://gerrit.wikimedia.org/r/1240084

Change #1240381 had a related patch set uploaded (by Jgleeson; author: Jgleeson):

[wikimedia/fundraising/dev@master] Add tools and mediawiki-donor-export helper scripts

https://gerrit.wikimedia.org/r/1240381

@Jdlrobson-WMF Just FYI, it looks like we'll be planning to refresh donor_status_id, likely with significant changes, launching at the start of the fiscal.

Change #1240818 had a related patch set uploaded (by Lars SG; author: Lars SG):

[wikimedia/fundraising/tools@master] Add do_not_solicit to silverpop_export_view_full for MW export

https://gerrit.wikimedia.org/r/1240818

Change #1240381 merged by Damilare Adedoyin:

[wikimedia/fundraising/dev@master] Add tools and mediawiki-donor-export helper scripts

https://gerrit.wikimedia.org/r/1240381

Change #1237965 merged by jenkins-bot:

[wikimedia/fundraising/tools@master] Add MediaWiki donor status export

https://gerrit.wikimedia.org/r/1237965

Change #1241018 had a related patch set uploaded (by Lars SG; author: Lars SG):

[wikimedia/fundraising/tools@master] Add do_not_solicit MW export

https://gerrit.wikimedia.org/r/1241018

Change #1240084 merged by jenkins-bot:

[wikimedia/fundraising/tools@master] Include opted out contacts in silverpop_export & silverpop_export_view_full

https://gerrit.wikimedia.org/r/1240084

Change #1240818 merged by jenkins-bot:

[wikimedia/fundraising/tools@master] Add do_not_solicit to silverpop_export_view_full for MW export

https://gerrit.wikimedia.org/r/1240818

Change #1241018 merged by jenkins-bot:

[wikimedia/fundraising/tools@master] Add do_not_solicit MW export

https://gerrit.wikimedia.org/r/1241018

Change #1247609 had a related patch set uploaded (by Jgleeson; author: Jgleeson):

[wikimedia/fundraising/tools@master] Add silverpop prereq data freshness check to mediawiki export job

https://gerrit.wikimedia.org/r/1247609

Change #1247614 had a related patch set uploaded (by Jgleeson; author: Jgleeson):

[wikimedia/fundraising/dev@master] Allow test suite to be passed when running python tools unit tests

https://gerrit.wikimedia.org/r/1247614

Change #1248017 had a related patch set uploaded (by Jgleeson; author: Jgleeson):

[wikimedia/fundraising/tools@master] Add --limit option to mediawiki donor export

https://gerrit.wikimedia.org/r/1248017

Change #1247614 merged by Jgleeson:

[wikimedia/fundraising/dev@master] Allow test suite to be passed when running python tools unit tests

https://gerrit.wikimedia.org/r/1247614

Change #1247609 merged by jenkins-bot:

[wikimedia/fundraising/tools@master] Add silverpop data freshness check to mediawiki export job

https://gerrit.wikimedia.org/r/1247609

Change #1248017 merged by jenkins-bot:

[wikimedia/fundraising/tools@master] Add --limit option to mediawiki donor export

https://gerrit.wikimedia.org/r/1248017

@jgleeson Feels like we should probably do something to rotate the export files, while we are doing this, in line with what we do for silverpop.

@Dwisehaupt, we need to do the following:

ALTER TABLE silverpop_export
ADD COLUMN opted_out TINYINT(1) AFTER opted_in,
ADD COLUMN do_not_solicit TINYINT(1) AFTER opted_out,
ADD COLUMN sms_consent TINYINT(1) AFTER do_not_solicit;

@Lars alter complete:

MariaDB [silverpop]> ALTER TABLE silverpop_export
    -> ADD COLUMN opted_out TINYINT(1) AFTER opted_in,
    -> ADD COLUMN do_not_solicit TINYINT(1) AFTER opted_out,
    -> ADD COLUMN sms_consent TINYINT(1) AFTER do_not_solicit;
Query OK, 0 rows affected (0.016 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [silverpop]> describe silverpop_export;
+---------------------------------------------------+---------------------+------+-----+---------+-------+
| Field                                             | Type                | Null | Key | Default | Extra |
+---------------------------------------------------+---------------------+------+-----+---------+-------+
| id                                                | int(10) unsigned    | NO   | PRI | NULL    |       |
| modified_date                                     | datetime            | YES  | MUL | NULL    |       |
| contact_id                                        | int(10) unsigned    | YES  | UNI | NULL    |       |
| contact_hash                                      | varchar(32)         | YES  |     | NULL    |       |
| first_name                                        | varchar(128)        | YES  |     | NULL    |       |
| last_name                                         | varchar(128)        | YES  |     | NULL    |       |
| preferred_language                                | varchar(12)         | YES  |     | NULL    |       |
| email                                             | varchar(255)        | YES  | UNI | NULL    |       |
| opted_in                                          | tinyint(1)          | YES  |     | NULL    |       |
| opted_out                                         | tinyint(1)          | YES  |     | NULL    |       |
| do_not_solicit                                    | tinyint(1)          | YES  |     | NULL    |       |
| sms_consent                                       | tinyint(1)          | YES  |     | NULL    |       |
| employer_id                                       | int(10) unsigned    | YES  |     | NULL    |       |
| employer_name                                     | varchar(255)        | YES  |     | NULL    |       |
| direct_mail_latest_appeal                         | varchar(255)        | YES  |     | NULL    |       |
| matching_gifts_provider_info_url                  | varchar(255)        | YES  |     | NULL    |       |
| guide_url                                         | varchar(255)        | YES  |     | NULL    |       |
| online_form_url                                   | varchar(255)        | YES  |     | NULL    |       |
| foundation_has_recurred_donation                  | tinyint(1)          | YES  |     | NULL    |       |
| foundation_has_active_recurring_donation          | tinyint(1)          | YES  |     | NULL    |       |
| foundation_recurring_first_donation_date          | datetime            | YES  |     | NULL    |       |
| foundation_recurring_latest_donation_date         | datetime            | YES  |     | NULL    |       |
| foundation_recurring_active_count                 | tinyint(3) unsigned | YES  |     | NULL    |       |
| foundation_recurring_latest_contribution_recur_id | int(10)             | YES  |     | NULL    |       |
| recurring_has_upgrade_activity                    | tinyint(1)          | YES  |     | NULL    |       |
| foundation_highest_usd_amount                     | decimal(20,2)       | YES  |     | NULL    |       |
| foundation_highest_native_amount                  | decimal(20,2)       | YES  |     | NULL    |       |
| foundation_highest_native_currency                | varchar(3)          | YES  |     | NULL    |       |
| foundation_highest_donation_date                  | datetime            | YES  |     | NULL    |       |
| lifetime_usd_total                                | decimal(20,2)       | YES  |     | NULL    |       |
| donation_count                                    | int(10) unsigned    | NO   |     | 0       |       |
| all_funds_total_2018_2019                         | decimal(20,2)       | YES  |     | NULL    |       |
| all_funds_total_2019_2020                         | decimal(20,2)       | YES  |     | NULL    |       |
| all_funds_total_2020_2021                         | decimal(20,2)       | YES  |     | NULL    |       |
| all_funds_total_2021_2022                         | decimal(20,2)       | YES  |     | NULL    |       |
| all_funds_total_2022_2023                         | decimal(20,2)       | YES  |     | NULL    |       |
| all_funds_total_2023_2024                         | decimal(20,2)       | YES  |     | NULL    |       |
| all_funds_total_2024_2025                         | decimal(20,2)       | YES  |     | NULL    |       |
| all_funds_total_2025_2026                         | decimal(20,2)       | YES  |     | NULL    |       |
| endowment_last_donation_date                      | datetime            | YES  |     | NULL    |       |
| endowment_first_donation_date                     | datetime            | YES  |     | NULL    |       |
| endowment_number_donations                        | int(10) unsigned    | NO   |     | 0       |       |
| endowment_highest_usd_amount                      | decimal(20,2)       | YES  |     | NULL    |       |
| foundation_latest_currency                        | varchar(3)          | YES  |     | NULL    |       |
| foundation_latest_currency_symbol                 | varchar(8)          | YES  |     | NULL    |       |
| foundation_latest_native_amount                   | decimal(20,2)       | YES  |     | NULL    |       |
| foundation_last_donation_date                     | datetime            | YES  |     | NULL    |       |
| foundation_first_donation_date                    | datetime            | YES  |     | NULL    |       |
| city                                              | varchar(128)        | YES  |     | NULL    |       |
| country                                           | varchar(2)          | YES  |     | NULL    |       |
| state                                             | varchar(64)         | YES  |     | NULL    |       |
| postal_code                                       | varchar(128)        | YES  |     | NULL    |       |
| donor_segment_id                                  | int(11)             | YES  |     | NULL    |       |
| donor_status_bin                                  | int(10) unsigned    | YES  |     | NULL    |       |
| double_opt_in_activity                            | tinyint(1)          | YES  |     | 0       |       |
+---------------------------------------------------+---------------------+------+-----+---------+-------+
55 rows in set (0.002 sec)

MariaDB [silverpop]>

We tested the new MediaWiki donor export today in production, and the export generation works as expected.

jgleeson@civi1002:/srv/process-control/civi1002$ run-job -wj mediawiki_donor_export
Writing lockfile.
Slow starting job Mediawiki Donor Export - Build export files (mediawiki_donor_export)
Running command: /usr/bin/python3 -m mediawiki_donor_export.export --days=1 --limit=50
Logging to /var/log/process-control/mediawiki_donor_export/mediawiki_donor_export-20260306-142156.log
===========
/usr/bin/python3 -m mediawiki_donor_export.export --days=1 --limit=50 (3951738)
-----------
Lockfile found!
Lockfile is stale.
Lockfile is stale.
Removing old lockfile.
Exporting donor status (delta: last 1 days)
Wrote 50 rows to /srv/tmp/mediawiki_donor_export/MediaWikiDonorStatus-20260306142156.csv
----------- end command output
Successfully completed mediawiki_donor_export.
Clearing lockfile.

The file contained 50 real records with id, email, and donor status. This serves as a good smoke test.

The next step is to add the encryption post-generation, and then add the SFTP target to push the file to.

Hi @CDanis, @Jdrewniak and @SToyofuku-WMF, I'm looking over my visual todo list for this work and one thing we don't have is an SFTP target to push to. Do we have that yet?

Slack-bot changed the task status from Open to In Progress.Mar 16 2026, 12:49 PM
Slack-bot closed this task as Resolved.

Change #1253624 had a related patch set uploaded (by Jgleeson; author: Jgleeson):

[wikimedia/fundraising/tools@master] Add optional AES encryption for mediawiki donor export output

https://gerrit.wikimedia.org/r/1253624

Hi folks. We've just pushed the work to encrypt the export using AES-256 in this patch. If you would like to test this at your end simulating processing the encrypted file, you can encrypt test data with:

OPENSSL_PASS=YOUR_KEY openssl enc -aes-256-cbc -salt -pbkdf2 -in FILE.csv -out FILE.enc -pass env:OPENSSL_PASS

This works with any language's OpenSSL bindings or the CLI tool directly.

Change #1255036 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] Add fundraising-data-uploader role user

https://gerrit.wikimedia.org/r/1255036

Change #1256389 had a related patch set uploaded (by Jgleeson; author: Jgleeson):

[wikimedia/fundraising/tools@master] Limit MW donor export CSV to email and donor_status_id only

https://gerrit.wikimedia.org/r/1256389

Jdlrobson-WMF lowered the priority of this task from High to Medium.Mar 30 2026, 9:20 PM

Change #1253624 merged by jenkins-bot:

[wikimedia/fundraising/tools@master] Add optional AES encryption for mediawiki donor export output

https://gerrit.wikimedia.org/r/1253624

Change #1256389 merged by jenkins-bot:

[wikimedia/fundraising/tools@master] Limit MW donor export CSV to email and donor_status_id only

https://gerrit.wikimedia.org/r/1256389

Change #1255036 merged by CDanis:

[operations/puppet@production] deployment_server: fundraising-data-uploader role

https://gerrit.wikimedia.org/r/1255036

Change #1268688 had a related patch set uploaded (by Lars SG; author: Lars SG):

[wikimedia/fundraising/tools@master] Remove unneeded details from export test

https://gerrit.wikimedia.org/r/1268688

Change #1268688 merged by jenkins-bot:

[wikimedia/fundraising/tools@master] Remove unneeded details from export test

https://gerrit.wikimedia.org/r/1268688

Change #1271028 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] fundraising_data_import maintenance script wrapper & timer

https://gerrit.wikimedia.org/r/1271028

Change #1271028 merged by CDanis:

[operations/puppet@production] fundraising_data_import maintenance script wrapper & timer

https://gerrit.wikimedia.org/r/1271028

Change #1304104 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] fundraising_data_import: hostname != FQDN

https://gerrit.wikimedia.org/r/1304104

Change #1304104 merged by CDanis:

[operations/puppet@production] fundraising_data_import: hostname != FQDN

https://gerrit.wikimedia.org/r/1304104