Page MenuHomePhabricator

Analytics request for 5 years of donor segment & status fields
Open, Needs TriagePublic

Description

In order to do year-on-year metrics using consistent definitions @JMando has requested that we store 5 years worth of donor segment & status fields for Analytics use.

Timing wise we would want to work on this a sprint or 2 before a planned maintenance window - although we may not require an outage if we use storage options 1 or 3 (it is adding fields to the large wmf_donor table that is slow)

It is worth noting that the fields are 'static' except when we merge contacts. When we merge 2 contacts with different giving histories the status for each year is recalculated.

Storage Options are

  1. new custom data table in CiviCRM
  2. add the options to wmf_donor (this only works if @NNichols finds enough fields to remove from there)
  3. stored in superset only (doesn't address merges etc)

Update options are

  1. update all the fields, new & old, via triggers - this is unlikely to be possible if the fields are not in the wmf_donor table but would capture the outcome of all merges. This *might* have an impact on queue speed but generally it is best to test for queue speed impacts rather than assume
  2. update the old fields by script, perhaps running the script on a rolling basis over all contacts
  3. update the old fields during maintenance window or other one-off or infrequent update methos

Event Timeline

@JMando FYI I'm looking at planning for our July 1 handling of the segmentation / status fields at the moment - my current thinking is to add an additional table that would hold the fields & would be populated on one-off basis by script (on the assumption that the fields are used for aggregation & it doesn't matter if merges make them a bit wrong over time)

The field for the current fiscal would be created at the same time but not populated until 1 July - my thinking being we can do it by script over a few days & would update the 'current' segment at the same time (ie the emptiness of the field in the wmf_segmentation_history field would be our WHERE on which contacts to run the script on)

Still working through it but wanted to put in on your radar in case you want to have input. Otherwise I'll just look forward to seeing you person soon

image.png (386×970 px, 114 KB)