Page MenuHomePhabricator
Create Task
Maniphest T383194

Fix dedupe merge wrapper script to track modified date
Open, Needs TriagePublic
Actions

Description

Our automated dedupe script tracks the contact ID it got up to each run & then does the next chunk the next run. This used to be fine as we didn't do deduping on ingress so we were always adding - but it feels like maybe we should consider looking at modified_date now - so that when a contact is updated they are then queued for dedupe

This could be a bit tricksy when we do large contact updates - ie at fiscal year end - because when we do that many contacts will have a recently modified date....

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Clarify test commentwikimedia/fundraising/crmmaster+3 -4
Cleaner / more complete command outputwikimedia/fundraising/crmmaster+5 -12
Avoid group_concat when fetching idswikimedia/fundraising/crmmaster+7 -1
Improve dedupe loggingwikimedia/fundraising/crmmaster+22 -10
Debug output to generate a command to try for replicationwikimedia/fundraising/crmmaster+12 -0
Fix query retrieving max idwikimedia/fundraising/crmmaster+2 -1
Respect permissions on dedupe criteriawikimedia/fundraising/crmmaster+2 -2
Further queue_task fixeswikimedia/fundraising/crmmaster+3 -3
Customize query in gerrit

Related Objects

Event Timeline

Eileenmcnaughton created this task.Jan 8 2025, 1:47 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 8 2025, 1:47 AM
Damilare moved this task from Triage to Current Sprint on the Fundraising-Backlog board.Jan 28 2025, 8:32 PM
Damilare added a project: Fundraising Sprint Bibimbap.Jan 28 2025, 9:00 PM
Eileenmcnaughton claimed this task.Feb 5 2025, 12:09 AM
Eileenmcnaughton moved this task from Backlog to Doing on the Fundraising Sprint Bibimbap board.
AKanji-WMF edited projects, added Fundraising Sprint: Churros; removed Fundraising Sprint Bibimbap.Feb 11 2025, 9:34 PM
Eileenmcnaughton moved this task from Backlog to Doing on the Fundraising Sprint: Churros board.Feb 12 2025, 2:18 AM
AKanji-WMF edited projects, added Fundraising Sprint: Dodos; removed Fundraising Sprint: Churros.Feb 25 2025, 8:56 PM
Eileenmcnaughton moved this task from Backlog to Doing on the Fundraising Sprint: Dodos board.Mar 11 2025, 1:09 AM
AKanji-WMF edited projects, added Fundraising Sprint: Elephant Toothpaste; removed Fundraising Sprint: Dodos.Mar 11 2025, 8:06 PM
Eileenmcnaughton moved this task from Backlog to Doing on the Fundraising Sprint: Elephant Toothpaste board.Mar 17 2025, 7:36 PM
AKanji-WMF edited projects, added Fundraising Sprint: Frito Pie; removed Fundraising Sprint: Elephant Toothpaste.Mar 25 2025, 7:49 PM
Eileenmcnaughton moved this task from Backlog to Doing on the Fundraising Sprint: Frito Pie board.Mar 25 2025, 9:58 PM
Eileenmcnaughton moved this task from Doing to Ready for Review on the Fundraising Sprint: Frito Pie board.Apr 6 2025, 12:24 AM
AKanji-WMF edited projects, added Fundraising Sprint: Grits & Catfish; removed Fundraising Sprint: Frito Pie.Apr 8 2025, 8:11 PM
Eileenmcnaughton moved this task from Backlog to Ready for Review on the Fundraising Sprint: Grits & Catfish board.Apr 15 2025, 10:48 PM
AKanji-WMF edited projects, added Fundraising Sprint: Hot Pot!; removed Fundraising Sprint: Grits & Catfish.Apr 23 2025, 8:33 PM
Eileenmcnaughton moved this task from Backlog to Ready for Review on the Fundraising Sprint: Hot Pot! board.Apr 24 2025, 5:03 AM
XenoRyet added a project: Fundraising Sprint Impossible shepherds pie.May 6 2025, 10:49 PM
Eileenmcnaughton moved this task from Backlog to Ready for Review on the Fundraising Sprint Impossible shepherds pie board.May 12 2025, 7:42 PM
AKanji-WMF edited projects, added Fundraising Sprint: Jollof Rice; removed Fundraising Sprint Impossible shepherds pie, Fundraising Sprint: Hot Pot!.May 20 2025, 7:18 PM
AKanji-WMF edited projects, added Fundraising Sprint: Kimchee Pancake; removed Fundraising Sprint: Jollof Rice.Jun 3 2025, 8:18 PM
Eileenmcnaughton moved this task from Backlog to Doing on the Fundraising Sprint: Kimchee Pancake board.Jun 4 2025, 6:03 AM
AKanji-WMF added a project: fr-current-sprint.Jun 13 2025, 7:16 PM
AKanji-WMF edited projects, added Fundraising Sprint: Left-Handed Churro ; removed Fundraising Sprint: Kimchee Pancake.Jun 17 2025, 7:51 PM
XenoRyet added a project: Fundraising Sprint Michelin Star.Jul 1 2025, 7:15 PM
AKanji-WMF removed a project: Fundraising Sprint: Left-Handed Churro .Jul 7 2025, 2:35 PM
Eileenmcnaughton moved this task from Backlog to Ready for Review on the Fundraising Sprint Michelin Star board.Jul 7 2025, 7:46 PM
AKanji-WMF removed projects: Fundraising-Backlog, Wikimedia-Fundraising-CiviCRM, FR-Civi-Dedupe.Jul 8 2025, 7:03 PM
AKanji-WMF added a project: Fundraising-Backlog.
AKanji-WMF edited projects, added Fundraising Sprint: Naan-Sequitur; removed Fundraising Sprint Michelin Star.Jul 15 2025, 8:02 PM
AKanji-WMF moved this task from Backlog to General Civi on the fr-current-sprint board.Jul 15 2025, 8:12 PM
Eileenmcnaughton moved this task from Backlog to Ready for Review on the Fundraising Sprint: Naan-Sequitur board.Jul 15 2025, 10:00 PM
AKanji-WMF edited projects, added Fundraising Sprint: Omakase; removed Fundraising Sprint: Naan-Sequitur.Jul 29 2025, 7:48 PM
AKanji-WMF moved this task from Backlog to Ready for Review on the Fundraising Sprint: Omakase board.
Damilare edited projects, added Fundraising Sprint: Pirate's Booty; removed Fundraising Sprint: Omakase.Aug 12 2025, 7:50 PM
Damilare moved this task from Backlog to Doing on the Fundraising Sprint: Pirate's Booty board.Aug 18 2025, 8:04 PM
AKanji-WMF mentioned this in T402571: Fundraising Sprint Q - Priorities.Aug 26 2025, 6:45 PM
AKanji-WMF edited projects, added Fundraising Sprint: Queuesadilla; removed Fundraising Sprint: Pirate's Booty.Aug 26 2025, 6:50 PM
AKanji-WMF moved this task from Backlog to Doing on the Fundraising Sprint: Queuesadilla board.
AKanji-WMF edited projects, added Fundraising Sprint: rich, the; removed Fundraising Sprint: Queuesadilla.Sep 9 2025, 9:00 PM
AKanji-WMF moved this task from Backlog to Doing on the Fundraising Sprint: rich, the board.
AKanji-WMF edited projects, added Fundraising Sprint: Spaghetti code and makefiles; removed Fundraising Sprint: rich, the.Sep 23 2025, 7:59 PM
AKanji-WMF edited projects, added Fundraising Sprint: Temp file pura; removed Fundraising Sprint: Spaghetti code and makefiles.Oct 7 2025, 8:12 PM
AKanji-WMF edited projects, added Fundraising Sprint: UTM_key lime pie; removed Fundraising Sprint: Temp file pura.Oct 21 2025, 8:18 PM
AKanji-WMF edited projects, added Fundraising Sprint: Vixen; removed Fundraising Sprint: UTM_key lime pie.Nov 4 2025, 9:21 PM
Eileenmcnaughton moved this task from Backlog to Ready for Review on the Fundraising Sprint: Vixen board.Nov 5 2025, 11:43 PM
Eileenmcnaughton added a comment.Nov 6 2025, 12:39 AM

I pushed out a fix to the limit & re-started the job but since it starts right back at the earliest modified_date it's not processing anything yet - I'll check in in a while

AnnWF subscribed.Nov 6 2025, 12:58 AM

https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1202287

AnnWF moved this task from Ready for Review to Pending Deployment on the Fundraising Sprint: Vixen board.Nov 6 2025, 12:58 AM
gerritbot added a comment.Nov 6 2025, 9:36 PM

Change #1202838 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Further queue_task fixes

https://gerrit.wikimedia.org/r/1202838

gerritbot added a project: Patch-For-Review.Nov 6 2025, 9:36 PM
gerritbot added a comment.Nov 6 2025, 10:18 PM

Change #1202838 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Further queue_task fixes

https://gerrit.wikimedia.org/r/1202838

Maintenance_bot removed a project: Patch-For-Review.Nov 6 2025, 10:30 PM
XenoRyet added a project: Fundraising Sprint WWaally hoo.Tue, Nov 18, 8:51 PM
Eileenmcnaughton moved this task from Backlog to Doing on the Fundraising Sprint WWaally hoo board.Thu, Nov 20, 1:29 AM
gerritbot added a comment.Tue, Nov 25, 9:34 PM

Change #1211245 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Respect permissions on dedupe criteria

https://gerrit.wikimedia.org/r/1211245

gerritbot added a project: Patch-For-Review.Tue, Nov 25, 9:34 PM
gerritbot added a comment.Tue, Nov 25, 9:36 PM

Change #1211246 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Fix query retrieving max id

https://gerrit.wikimedia.org/r/1211246

gerritbot added a comment.Tue, Nov 25, 9:56 PM

Change #1211245 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Respect permissions on dedupe criteria

https://gerrit.wikimedia.org/r/1211245

gerritbot added a comment.Tue, Nov 25, 10:00 PM

Change #1211246 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Fix query retrieving max id

https://gerrit.wikimedia.org/r/1211246

Maintenance_bot removed a project: Patch-For-Review.Tue, Nov 25, 10:31 PM
Eileenmcnaughton moved this task from Doing to Done on the Fundraising Sprint WWaally hoo board.Tue, Nov 25, 11:45 PM

This appears to be working OK now & around 600 contacts have merged since I rolled it out an hour ago.

Currently 3 jobs are running - one starting from early 2024, one from the start of the db and the main one - which is deduping new contacts in real time.

The expectation is the other 2 will be deleted once they catch up

gerritbot added a comment.Wed, Nov 26, 5:06 AM

Change #1211332 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Debug output to generate a command to try for replication

https://gerrit.wikimedia.org/r/1211332

gerritbot added a project: Patch-For-Review.Wed, Nov 26, 5:06 AM
gerritbot added a comment.Mon, Dec 1, 12:16 AM

Change #1211332 merged by Eileen:

[wikimedia/fundraising/crm@master] Debug output to generate a command to try for replication

https://gerrit.wikimedia.org/r/1211332

Maintenance_bot removed a project: Patch-For-Review.Mon, Dec 1, 12:30 AM
gerritbot added a comment.Mon, Dec 1, 1:11 AM

Change #1213132 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Improve dedupe logging

https://gerrit.wikimedia.org/r/1213132

gerritbot added a project: Patch-For-Review.Mon, Dec 1, 1:11 AM
gerritbot added a comment.Mon, Dec 1, 2:00 AM

Change #1213132 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Improve dedupe logging

https://gerrit.wikimedia.org/r/1213132

Maintenance_bot removed a project: Patch-For-Review.Mon, Dec 1, 2:30 AM
gerritbot added a comment.Mon, Dec 1, 8:10 PM

Change #1213560 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Avoid group_concat when fetching ids

https://gerrit.wikimedia.org/r/1213560

gerritbot added a project: Patch-For-Review.Mon, Dec 1, 8:10 PM

Change #1213561 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Cleaner / more complete command output

https://gerrit.wikimedia.org/r/1213561

XenoRyet moved this task from Done to Doing on the Fundraising Sprint WWaally hoo board.Mon, Dec 1, 8:32 PM
gerritbot added a comment.Mon, Dec 1, 9:16 PM

Change #1213560 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Avoid group_concat when fetching ids

https://gerrit.wikimedia.org/r/1213560

gerritbot added a comment.Tue, Dec 2, 4:09 AM

Change #1213561 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Cleaner / more complete command output

https://gerrit.wikimedia.org/r/1213561

Maintenance_bot removed a project: Patch-For-Review.Tue, Dec 2, 4:31 AM
gerritbot added a comment.Tue, Dec 2, 4:26 PM

Change #1214105 had a related patch set uploaded (by Lars SG; author: Lars SG):

[wikimedia/fundraising/crm@master] Clarify test comment

https://gerrit.wikimedia.org/r/1214105

gerritbot added a project: Patch-For-Review.Tue, Dec 2, 4:26 PM
gerritbot added a comment.Tue, Dec 2, 7:09 PM

Change #1214105 merged by Eileen:

[wikimedia/fundraising/crm@master] Clarify test comment

https://gerrit.wikimedia.org/r/1214105

Maintenance_bot removed a project: Patch-For-Review.Tue, Dec 2, 7:31 PM
AKanji-WMF edited projects, added Fundraising Sprint: Xiaolongbao; removed Fundraising Sprint WWaally hoo.Tue, Dec 2, 8:58 PM
Eileenmcnaughton moved this task from Backlog to Done on the Fundraising Sprint: Xiaolongbao board.Mon, Dec 8, 8:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits