Just updating this with what we need to do

Before the db update (note completing these is not a pre-requisite to doing the update -they might continue on)

• Action fr-tech with help from @bsisolak - Review whether any other api code needs updating - I have listed all our api calls below in https://phabricator.wikimedia.org/T365826#10198474 - we need to fill in the needs update column with help from @bsisolak
• update api calls T376689
• Action fr-tech As part of a separate request, add preference_tags to our export T366312
• Update segments to include annual recurring - this does not need to tie in with the window as it is touching existing fields - but it might be helpful for QA purposes to align it T372378
• Possible fix on first name fields - this does not need to tie in with the window as it is touching existing fields - but it might be helpful as we would likely do a full db upload if there is a substantive fix in there T376347
• Brian / Erik to confirm how RML works for emails post convert (they have converted they need to update the form) - added T376685

The db update

• Action - trilogy / accoustic - Starting 9 Oct Acoustic will migrate the Acoustic database so that email is no longer a unique key - this necessitates changing some of our integrations

After the db update

• Action - fr-tech - Update the mappings for our nightly exports from CIvi to Acoustic . We are currently trying to make it possible to do that in code - but failing that we need to work with Natalie to recreate the imports T376690
• Action fr-tech Look at renaming 3 fields in the csv per T376360
• Action @NNgu-WMF QA on the exports & tags

Before we start doing SMS campaigns

• Action @NNgu-WMF -with support from fr-tech update Acoustic list 18468760 to exclude contacts with no emails - this list is the list of remind me later contacts in Acoustic (that have no CiviCRM ID) and they are imported back to CiviCRM & added to the group 302. This should be done before we start
• Confirm the process & additional functionality required for the RML flow (this might be largely done in the scoping of T376686)
• Write queue consumption for these requests when they come in T376686 - see https://phabricator.wikimedia.org/T365826#10201384

Just thinking about the process when we start SMS - let's imagine someone fills in remind me later - they can either use their email or (new) their phone number.

Email flow (current)

• a new record is created in Acoustic with their email
• this record is pulled into CiviCRM
• the nightly export picks up this new record as 'recently modifier' and pushes it to Acoustic - thus giving it a Contact ID in CiviCRM

Question - what does Acoustic do when the form is filled in for a contact that already exists in Acoustic

Phone flow (new)

• a new record is created in Acoustic with just a phone
• * we don't want to pull the record into CiviCRM at this stage *
• a SMS is sent to the contact. The url would include the Acoustic recipient ID
• when the contact donates would would get their email - this might be a pre-existing email or a new email
• on import our queue would detect the presence of the recipient ID and either
• not find an existing contact in which case a new record would be created - we could maybe at this point fire off an update to add the email address to that recipient ID in Acoustic so that on later upload the dots would be connected. Probably similar to snooze we would queue this via coworker
• find an existing contact - in this case we would ideally merge the records using the merge api - but it's hard to follow that page to see what that would look like

Note that we also discussed this flow in our call - but I feel like we rejected having orphan phones in civi

Potential flow via RML
RML offers supporters choice of submit your phone and/or email
Opt-in text goes to supporter within legal timeframe[talk to legal]
SMS sign up goes into Acoustic
Passed to Civi to get a CiviID, which is then sent back to Acoustic
Before the SMS message goes out, so any hyperlink included in the message can include a cid in the URL to match back.

Questions for call tomorrow

• I'm struggling with the importList mapping - help
• check[[ https://phabricator.wikimedia.org/T365826#10198474 | this list ]]t of api calls to re-confirm which need updating as our notes are not good enough
• If time discuss the flow again https://phabricator.wikimedia.org/T365826#10198466

As for what calls I think need to be updated:

GetSentMailingsForOrgRequest - No
ExportListRequest - No
CreateContactListRequest - No
ImportListRequest - No (API call does not change, file format does. https://developer.goacoustic.com/acoustic-campaign/reference/importlist)
RawRecipientDataExportRequest - No

GetQueryRequest - Not sure what this API call is

AddContactToContactList - no, as you already use Contact ID here (https://developer.goacoustic.com/acoustic-campaign/reference/addcontacttocontactlist)

AddRecipient - Yes, SYNC_FIELDS (https://developer.goacoustic.com/acoustic-campaign/reference/add-a-contact)

SelectRecipientData - Yes (https://developer.goacoustic.com/acoustic-campaign/reference/selectrecipientdata)

PrivacyInformationRequest - Depends
PrivacyDeleteRequest - Depends
https://developer.goacoustic.com/acoustic-campaign/docs/perform-ccpa-gdpr-right-of-access-with-acoustic-campaign-automation-apis

We had a helpful call with Brian & Eric & some updates

1. It seems we only need to update the addRecipient, and select RecipientData calls - neither of these are used in our primary data processes so the impact of them being fixed slightly later is low

2. Brian / Eric will confirm whether our existing Remind me later form needs any changes as a result of the database transformation to ensure that anyone who fills in the RML form who has an existing email in Acoustic updates the existing record rather than creates a new one

3. We are still a bit stuck on what is happening with the mapping - currently best guess is something to do with the user account - it is

a) uploading to the sftp site
b) being removed from the sftp site & ?being moved to the user directory?
c) the api call is being received & fails differently if the file is present on the sftp or not.

• we will try with the live user on prod once we can get the code out & Brian will also check in with Acoustic

4. Regarding the flow when we start doing text rmls - the idea would be that the recipient id would be in the url & we would need to
5. update the donation form to take the recipient id from the rml and add it to the message
6. create 2 new custom fields in Civi extending the 'phone' entity - source and recipient ID (source options TBD - but probably similar to address)
7. in Acoustic create a custom field to denote the 'master record' - (would it hold contact ID, recipient ID or email?) to be populated with an RML phone has led to a donation & thus led to us linking it to a different contact
8. In the Message class normalization - when recipient_id is present add to the message array these 2 new custom fields (apiv4 style syntax) plus phone_primary.phone = 999999
9. In the WMFContact.Save class ensure the custom fields get save
10. Create a job that looks for this 99999 phone number and sends a query to Acoustic to get the actual phone number based on the recipient ID. At this point it would ALSO look up whether there is an existing record in Acoustic for the email of the contact. If there is not then if will 'claim' the phone-only record by pushing the email into it. If there is then it will soft-cancel the Acoustic record by pushing the master record value into it
11. the tools repo export should be update to not push any records with a phone of 99999 - to avoid the nightly export doing anything before the reconcilitiation job has had a chance

Regarding #2, we have confirmed we will need to update the existing Acoustic RML forms to match existing records using the email field when the database type is updated. This is a simple checkbox selection and we (Brian and I) can make sure it's completed when the update is complete.

Regarding #3, we're still digging into the error you're seeing when testing in the sandbox and will send an update when we have more information. Do let us know how it goes when you test on the production database.

Moving to 'done' as I think the scoping aspect is done & we have split out other phabs