actually now I look at it - several of these are not existing CiviCRM fields so it might take some work to figure out how to calculate them - however, if need be we can use place holders for the new fields initially so that we can update the mapping in July in advance of us necessarily having all the data

I'm not familiar with the last 2 - are they recently added fields?

@MDemosWMF - I am working with Coleman (core team) not in SC & had a go at this with some help from him. I've deployed the patch onto staging - although it can't fully be tested on staging because coworker isn't running there - but if you do want to test it on staging @Dwisehaupt might be able to get coworker going

@JMando I don't think so cos it tracks the last fetch date rather than using a fetch date relative to when it runs

cool!

Not too sure if this is the only task - it is medium priority as it has come up a few times but is not immediately problematic but I might shoot the breeze on ideas for it while here

I tried deduping this group and was able to find a bunch of duplicate using deduper.

Here is the url to the group...
https://civicrm.wikimedia.org/civicrm/group/search?force=1&context=smog&gid=1193

OK - Manage Groups form we had in place was being delivered by the Admin UI extension - the query it comes up with is slow & I logged an upstream issue
https://lab.civicrm.org/dev/core/-/issues/5247

Hi Melanie - I removed the phone number from that row & it imported - I think the phone number field is mapped to the country field but most of the time it doesn't matter because the phone field is blank so it defaults to 'United States' - which is what it should be

I think all of us dug into this in some way today but I feel we got to the bottom of it. In the end I found 2 fixes & Elliott found 1 and I think any of them are OK for a short-term fix (we have merged Elliott's & can deploy it tomorrow).

great - thanks for updating me @KHaggard

@KHaggard the ones that have gone up over the past few days should have the new number / name

Actually this is not quite the same issue - we calculate the Donor Statuses based on the donations they have in CiviCRM. It's a pretty complex calculation & when we update to Acoustic they just get the highest priority status from the contacts being 'Acoustic Merged' - the New status is higher priority that the Current status so that gets exported.

I guess we will be pushing up the right statuses now but not re-actively updating them all. @KHaggard has plans for us to add some additional fields in an upcoming maintenance window so we will do a full DB push then & sweep up any retroactive ones. Also, come 1 July we will be updating the donor statuses so if that happens first most of them will be updated that way

I had a go at logging into Acoustic ... and failed - I'm assuming the issue here is the credentials have expired - but so have the credentials I normally use to login I guess.

Yep - also on staging

OK on Prod we have

So this was the eventual define

@KHaggard well it would only need to be done once as long as it is also possible to set a default value for the field in Acoustic so that any new records that do not have values get them when created (which seems like RML)

@KHaggard from today's run the non donor number should be changed to 1000 fr donor_status_id

This should be fixed by our renumbering - when contact's are Acoustic-merged (not merged in civi but combined during export to acoustic regardless of the reason they are not merged in civi) they get the LOWEST status number - we renumbered the recurring ones yesterday so they would have a lower number than the others & hence would take precedence

This should be fixed by our renumbering of the statuses

@KHaggard we are also pushing up slightly different words - Non donor vs Non Donors - should we consolidate that? if so how should we capitilize?

@KHaggard so if someone donates via RML & has a non-donor status we do pretty much suck them into Civi & update Acoustic with the CiviCRM contact ID - but we would leave that non-donor status in place. If they later donate then their status is updated like any other donor - ok with 1000!

thanks @MDemosWMF - I'll close this then & we can do a new phab if we need to

@KHaggard - yes we can change the value to 1000 or 1001 instead of 100 - ideally right away as we are in maintenance week - it's fine for them to be 1000 like the other if that is more sane. There was some idea that the numbers could be added together to get a number that represents a combined segment but not sure if that is worth considering

Putting this in pending deployment as there is one last patch to deploy https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1032895 - it's not +2 yet only because there might be some need to do last checks before deployment but it is +2 really.....

@KHaggard @JMando @ERoden-WMF we have updated the numbering for the statuses (the recurring ones) - the documentation updates automatically - https://civicrm.wikimedia.org/civicrm/wmf-segment when we make changes

@KHaggard the places where one of donor_status & donor_segment being set & not the other will be from when we first added the fields - I ran an update to ensure that if one is set both are set to reduce the confusion. I also fixed our export such that if a contact does not have a segment / status in CiviCRM the export will set it to non-donor rather than blank.

it should have been

@Dwisehaupt are the first query was to add it - but looking above it seems I posted the second query twice

@JMando - per conversation

After some digging on this I think that now that Joseph doesn't want the existing data stashed we should

Just noting that I think this is DONE now - @Cstone ?

grep -r "Updating legacy" *
recurring_queue_consume/recurring_queue_consume-20240521-101001.log:2024-05-21 10:10:18,204 ERROR civicrm.wmf.INFO: Updating legacy paypal contribution recur row

This is almost done - just a couple of tidy ups including https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1034559 & https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1034525

This is the same issue as the non-donor one - https://phabricator.wikimedia.org/T363959#9819404 - when we 'Acoustic-merge' a contact (that may be a non-duplicate or a not-yet-merged contact) we are using MAX() to get the status - because of the way the recurring ones are higher priority but are not numbered in such a way we can use MIN - it would be better to fix up that numbering - if there are no blockers

So the first problem here is that when we export to Acoustic we merge contacts regardless of whether they are merged in CiviCRM - they could be set to deliberately not merge in CiviCRM or they might be just not-yet-merged.

OK - I ran through this & cleared all of them out - should be closable now

Well I wouldn't say we ran out of disk space - it's more like adding 22GB to our database will bring forward the time frame in which we need to replace our server / create other work.

Running this

I re-ran the segment update on the first 3 & they all updated to Deep Lapsed so the calculation is correct. It's a long time since we populated these so I can't recall but my best guess is we tweaked the definition after these ones were populated

@MDemosWMF I have now imported them to https://civicrm.wikimedia.org/civicrm/group/search?force=1&context=smog&gid=2109

This one is as long as a piece of string & there is def more that could be done but I think maybe under another phab

I put this in pending deployment since the next step is probably to disable on live & push out the process control update (currently on frpm) - then I guess the module just needs to be removed

Plan is to deploy on Wed 22nd (Yours)
https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1027983

@NNichols - we only geocode by post code - are you entering a post code into that where clause?

this was because the footer template wasn't updated to use the url token - I just did it

Once this patch https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1031626 is merged they will be pushed to the queue - although we could consider