Page MenuHomePhabricator
Create Task
Maniphest T347724

Removing some WMF donor fields
Closed, ResolvedPublic
Actions

Description

Not high priority.
After the data audit we've concluded that when it comes to the WMF donor fields we can remove:

  • CY fields (including WMF only, Endowment specific, and ALL FUNDS) other than the immediate CY - These are used by MG for CY giving totals are helpful for those donors who give monthly (like Brave, etc.), so that they can prepare annual year end giving receipts that capture the full CY (donors typically care about CY, not FY)

Online focuses on FY - does not need CY at all

  • Change amount fields other than the current and immediate year

This should be about 43 fields

I hope this helps with efficiency!

Details

SubjectRepoBranchLines +/-
Re-run custom field removal - I forgot the civicrm_custom_field deletewikimedia/fundraising/crmmaster+16 -2
Update silverpop with removal of fields to be removedwikimedia/fundraising/toolsmaster+19 -91
Update calculation of WMF Donor fields to exclude fields for removalwikimedia/fundraising/crmmaster+215 -206
Fix last querywikimedia/fundraising/crmmaster+1 -1
Oops - our endowment totals are transposed...wikimedia/fundraising/crmmaster+2 -2
Disable fields to be removed during maitenance windowwikimedia/fundraising/crmmaster+64 -0
Update calculation of WMF Donor fields to exclude fields for removalwikimedia/fundraising/crmmaster+88 -78
Fix CustomField handling to stop logging disabled custom fieldswikimedia/fundraising/crmmaster+32 -0
Customize query in gerrit

Related Objects

Event Timeline

NNichols created this task.Sep 29 2023, 3:41 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 29 2023, 3:41 PM
KHaggard added a comment.EditedSep 29 2023, 11:44 PM

Just a quick little reminder, please hold off on deleting the few CY fields that are still in the export to Acoustic until we replace them with FY... and then we could blow them away after. :)
I'm pretty sure deleting them will cause the import to fail until we remap Acoustic (even with deletions) and I want to avoid remapping anything until after Big English.

Eileenmcnaughton added a comment.Sep 30 2023, 8:09 AM

Re the accordian export - we can just send 0 out in the accordian export - that should't block us from deleting from CiviCRM

KHaggard added a comment.Oct 2 2023, 3:57 PM

Oh ok that works. Thanks @Eileenmcnaughton

AKanji-WMF moved this task from Triage to Sprint +1 on the Fundraising-Backlog board.Oct 2 2023, 8:08 PM
AKanji-WMF subscribed.

Should schedule in advance of next maintenance window; Aim for Sprint T

Eileenmcnaughton added a comment.Oct 2 2023, 10:10 PM

Let's also remove the disabled field from this custom group - https://civicrm.wikimedia.org/civicrm/admin/custom/group/fields#/?gid=25 - maybe there are other disabled fields

AKanji-WMF moved this task from Sprint +1 to Current Sprint on the Fundraising-Backlog board.Oct 10 2023, 8:07 PM
AKanji-WMF added a project: Fundraising Sprint This LAN is your LAN.Oct 10 2023, 11:09 PM
Eileenmcnaughton moved this task from Backlog to Doing on the Fundraising Sprint This LAN is your LAN board.Oct 11 2023, 3:42 AM
gerritbot added a comment.Oct 12 2023, 4:40 AM

Change 965274 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Fix CustomField handling to stop logging disabled custom fields

https://gerrit.wikimedia.org/r/965274

gerritbot added a project: Patch-For-Review.Oct 12 2023, 4:40 AM
Eileenmcnaughton added a subscriber: ERoden-WMF.EditedOct 16 2023, 11:41 PM

OK so summarising what is going to happen during the upcoming maintenance window

  1. Remove all change fields up to & including 2021-22
  • Rationale / confirmation** - in this phab @NNichols states we can remove "Change amount fields other than the current and immediate year". I think this means we need to retain 2022-23 (2023 is our current calendar year so that makes sense).
  • Fields to remove

All Funds Change 2018-2019 total all_funds_change_2018_2019
All Funds Change 2019-2020 total all_funds_change_2019_2020
All Funds Change 2020-2021 total all_funds_change_2020_2021
All Funds Change 2021-2022 total all_funds_change_2021_2022
Change 2017-2018 total Annual fund change_2017_2018
Change 2018-2019 total Annual fund change_2018_2019
Change 2019-2020 total Annual fund change_2019_2020
Change 2020-2021 total Annual fund change_2020_2021
Change 2021-2022 total Annual fund change_2021_2022

  1. Remove all calendar year fields up to & including 2022
  • Rationale / confirmation - in this phab @NNichols states "CY fields (including WMF only, Endowment specific, and ALL FUNDS) other than the immediate CY ". I think this means we need to retain 2022-23 (2023 is our current calendar year so that makes sense).
  • Fields to remove

total_2006
total_2007
total_2008
total_2009
total_2010
total_2011
total_2012
total_2013
total_2014
total_2015
total_2016
total_2017
total_2018
total_2019
total_2020
total_2021
total_2022
endowment_total_2018
endowment_total_2019
endowment_total_2020
endowment_total_2021
endowment_total_2022

  1. Remove financial year fields up to & including 2017_2018
  2. Rationale / confirmation - in this phab @ERoden-WMF confirms that the proposal on that phab is agreed. https://phabricator.wikimedia.org/T339314#8954622 & also https://phabricator.wikimedia.org/T331919#8934843 - note that it's a bit unclear what 5 years means but since we are in 2023-2024 I guess 2018-2019 should be kept but not 2017-2018

total_2006_2007
total_2007_2008
total_2008_2009
total_2009_2010
total_2010_2011
total_2011_2012
total_2012_2013
total_2013_2014
total_2014_2015
total_2015_2016
total_2016_2017

  1. Replace all total_ fields in the acoustic export with zero amounts in the final export, remove from internal workings
gerritbot added a comment.Oct 16 2023, 11:58 PM

Change 966312 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/tools@master] Update silverpop with removal of fields to be removed

https://gerrit.wikimedia.org/r/966312

gerritbot added a comment.Oct 17 2023, 3:00 AM

Change 966318 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Disable fields to be removed during maitenance window

https://gerrit.wikimedia.org/r/966318

gerritbot added a comment.Oct 17 2023, 3:00 AM

Change 966319 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Update calculation of WMF Donor fields to exclude fields for removal

https://gerrit.wikimedia.org/r/966319

KHaggard added a comment.Oct 17 2023, 6:10 PM

Thanks @Eileenmcnaughton . Yes, just confirming that any fields currently in the export file that are being removed / changed / added should (for now) remain in the export file with value=0 until we can find a time after Big English to roll out the changes in Acoustic (I think just swapping the old CY totals with the FY totals). Reasoning is because changes in the export file, no matter how minuscule, tends to make the job fail until Acoustic matches it exactly.

gerritbot added a comment.Oct 18 2023, 3:06 AM

Change 966654 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Oops - our endowment totals are transposed...

https://gerrit.wikimedia.org/r/966654

Eileenmcnaughton moved this task from Doing to Ready for Review on the Fundraising Sprint This LAN is your LAN board.Oct 18 2023, 3:19 AM
gerritbot added a comment.Oct 18 2023, 2:01 PM

Change 965274 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Fix CustomField handling to stop logging disabled custom fields

https://gerrit.wikimedia.org/r/965274

gerritbot added a comment.Oct 18 2023, 2:15 PM

Change 966319 abandoned by Ejegg:

[wikimedia/fundraising/crm@master] Update calculation of WMF Donor fields to exclude fields for removal

Reason:

duplicated as I27221484d2cf6edeac3d9d8f5988314d6dc6595c

https://gerrit.wikimedia.org/r/966319

gerritbot added a comment.Oct 18 2023, 2:19 PM

Change 966318 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Disable fields to be removed during maitenance window

https://gerrit.wikimedia.org/r/966318

gerritbot added a comment.Oct 18 2023, 2:28 PM

Change 966654 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Oops - our endowment totals are transposed...

https://gerrit.wikimedia.org/r/966654

gerritbot added a comment.Oct 18 2023, 11:21 PM

Change 966938 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Fix last query

https://gerrit.wikimedia.org/r/966938

gerritbot added a comment.Oct 18 2023, 11:40 PM

Change 966938 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Fix last query

https://gerrit.wikimedia.org/r/966938

gerritbot added a comment.Oct 23 2023, 6:33 PM

Change 966655 had a related patch set uploaded (by Ejegg; author: Eileen):

[wikimedia/fundraising/crm@master] Update calculation of WMF Donor fields to exclude fields for removal

https://gerrit.wikimedia.org/r/966655

gerritbot added a comment.Oct 23 2023, 6:52 PM

Change 966655 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Update calculation of WMF Donor fields to exclude fields for removal

https://gerrit.wikimedia.org/r/966655

AKanji-WMF edited projects, added Fundraising Sprint Uptown cURL; removed Fundraising Sprint This LAN is your LAN.Oct 24 2023, 8:24 PM
Eileenmcnaughton moved this task from Backlog to Ready for Review on the Fundraising Sprint Uptown cURL board.Oct 25 2023, 1:35 AM
Damilare moved this task from Ready for Review to Pending Deployment on the Fundraising Sprint Uptown cURL board.Oct 26 2023, 1:10 PM
gerritbot added a comment.Oct 27 2023, 10:13 PM

Change 966312 merged by jenkins-bot:

[wikimedia/fundraising/tools@master] Update silverpop with removal of fields to be removed

https://gerrit.wikimedia.org/r/966312

Maintenance_bot removed a project: Patch-For-Review.Oct 27 2023, 10:31 PM
Eileenmcnaughton moved this task from Pending Deployment to Done on the Fundraising Sprint Uptown cURL board.Oct 31 2023, 9:51 PM
gerritbot added a comment.Nov 1 2023, 12:09 AM

Change 970464 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Re-run custom field removal - I forgot the civicrm_custom_field delete

https://gerrit.wikimedia.org/r/970464

gerritbot added a project: Patch-For-Review.Nov 1 2023, 12:09 AM
gerritbot added a comment.Nov 1 2023, 12:48 AM

Change 970464 merged by Eileen:

[wikimedia/fundraising/crm@master] Re-run custom field removal - I forgot the civicrm_custom_field delete

https://gerrit.wikimedia.org/r/970464

Eileenmcnaughton added a subscriber: JMando.Nov 1 2023, 12:53 AM
Maintenance_bot removed a project: Patch-For-Review.Nov 1 2023, 1:10 AM
XenoRyet closed this task as Resolved.Nov 6 2023, 8:25 PM
XenoRyet set Final Story Points to 4.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits