Page MenuHomePhabricator

Fail Mail (civi1002) run-job: Silverpop emails
Closed, ResolvedPublic

Description

We received a failmail on October 2, 2025 with the following information:

Fail Mail (civi1002) run-job: Silverpop emails - Build export files failed with code 1
See the logs for more information: /var/log/process-control/silverpop_emails_build_export_files/silverpop_emails_build_export_files-20251002-020001.log

A rerun of the jobs did not suffice to resolve this issue. Adding the email thread regarding this:

Elliott Eggleston - Thu, Oct 2, 2025 at 3:03 PM
I deleted the duplicate employer relationship for the associated contact, so tonight's export should work.
Looks like the duplicate relationship was created in this merge: https://civicrm.wikimedia.org/civicrm/report/instance/553?reset=1&log_conn_id=68dd74fd4170fZWvu&cid=52677910&oid=59607377&cid=52677910&alteredBy=Ellen%20Christine%20Johnston&alteredById=14795889&section=2
If we can reproduce the error locally it might be worth filing a bug upstream to get the merge code fixed.

Elliott Eggleston - Thu, Oct 2, 11:50 PM
Looks like it's a duplicate employer relationship. There's a bunch of internal logic that's supposed to synchronize the 'current employer' field with the relationship table and make sure only one is active at a time, but it's fallible. I'll clear out one of them.

Jack Gleeson - Thu, 2 Oct 2025 at 13:03
I'm not sure what went wrong here. I can see that the ID in question relates to a contact in civicrm_email, which has two rows with the same email, and both are set to is_primary = 1. However, this is due to a contact merge that happened yesterday.
The INSERT INTO silverpop_export_staging query, which is where things are breaking, doesn't seem to care about unique emails so I'm gonna put this down to a temporary blip and rerun the job.

Jack Gleeson - Thu, 2 Oct 2025 at 12:20
Looks like we've got a duplicate email in civicrm_email
100 │ 2025-10-02 02:03:30,064 ERROR pymysql.err.IntegrityError: (1062, "Duplicate entry '68716757' for key 'PRIMARY'")
I'll see if I can work out what went wrong and remove the one we don't want.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

@Ejegg fixed this one last night.

Looks like it's a duplicate employer relationship. There's a bunch of internal logic that's supposed to synchronize the 'current employer' field with the relationship table and make sure only one is active at a time, but it's fallible. I'll clear out one of them.

I checked the logs today, and I can see that last night's job ran successfully, so that must have been it! Thanks @Ejegg

168   │ 2025-10-03 04:07:35,968 INFO    Begin Silverpop Export
169   │ 2025-10-03 04:07:36,037 INFO    Starting full data export
170   │ 2025-10-03 04:09:40,327 INFO    Wrote 460564 rows
171   │ 2025-10-03 04:09:40,384 INFO    Starting unsubscribe data export
172   │ 2025-10-03 04:10:45,008 INFO    Wrote 11228441 rows
173   │ 2025-10-03 04:10:45,067 INFO    Starting unsubscribe data export
174   │ 2025-10-03 04:11:49,559 INFO    Wrote 11228441 rows
175   │ 2025-10-03 04:11:49,615 INFO    Starting email with correspondent checksum data export
176   │ 2025-10-03 04:15:14,285 INFO    Wrote 32773661 rows
XenoRyet set Final Story Points to 2.