Due to us needing to search this data with substring queries we probably wanna put this data into secondary location capable of supporting that feature (e.g SQLite import)

It looks like there's a builtin php function that could help here: https://www.php.net/manual/en/function.preg-grep.php

With preg_grep, we could just cache the big array of id->name in memcache, snag that, and grep it.

My current thinking on this is for generating the file we can either:

1. Add a new drupal module command called something like civicrm-employers-export maybe to the wmf_civicrm module that we call via process-control
2. OR we could do a similar sorta thing inside the python fundraising-tools project which already has scripts for building csv files from MySQL database tables.

I guess the drupal upgrade on the horizon might make option 1 less favourable if it will need refactoring although the option 2 fundraising-tools project is also kinda legacyee so we might wanna not put it there?

What do other @fr-tech folks think?

@jgleeson this commit creates an API call within the new Matching Gifts civi extension: https://gerrit.wikimedia.org/r/607633

once you've installed the extension, added the credentials and called MatchingGiftPolicies.Sync, just call this to create the new file:

drush cvapi MatchingGiftPolicies.Export path=/tmp/newEmployers.csv

So I think the work in this task would be to configure a wrapper script that calls the new cvapi call, diffs the new CSV with the previous one, emails if there is a change, then probably moves the new file to the old file path.

Ok great. Thanks for doing the heavy lifting on the CSV generation part!

@Ejegg do you see the sync and export processes running in tandem here or would there be a case to have the sync running independently to keep things fresh elsewhere and then we just call the export action to "get the latest" at that point in time.

@jgleeson maybe the parent job runs the sync and then the export?

Change 608089 had a related patch set uploaded (by Jgleeson; owner: Jgleeson):
[wikimedia/fundraising/crm@master] WIP: Command to diff matching gifts employer data file

https://gerrit.wikimedia.org/r/608089

The directory for storing these downloads has been added to the civicrm role manifests. A puppet run on both hosts has created the directory.

[frack::puppet] 5ee37ccf Add matching gifts employer for HEPdata download

To Test:

1. pull down https://gerrit.wikimedia.org/r/608089
2. connect to Wikimedia VPN
3. run drush dis wmf_civicrm
4. run drush en wmf_civicrm, wmf_communication, exchange_rates, queue2civicrm, wmf_audit, adyen_audit, amazon_audit, astropay_audit, banner_history, ingenico_audit, thank_you, large_donation, offline2civicrm, orphan_slayer, recurring, recurring_globalcollect, wmf_campaigns, wmf_ct_qc, wmf_eoy_receipt, wmf_fredge_qc, wmf_optin_qc, wmf_refund_qc, wmf_unsubscribe_qc
5. open http://crm.local.wmftest.net:8080/#overlay=admin/config/wmf_civicrm/matching_gifts and set path to /tmp/employers.csv and save
6. run drush cc all
7. run drush cvmgc (this is the new command)
8. update notifications emails should be visible in mailcatcher http://localhost:1080/

Change 608089 merged by jenkins-bot:
[wikimedia/fundraising/crm@master] Command to check matching gifts employer data file

https://gerrit.wikimedia.org/r/608089