Page MenuHomePhabricator
Create Task
Maniphest T395011

Fail Mail (civi1002) run-job: Silverpop emails
Closed, ResolvedPublic
Actions

Description

We received failmail today for Acoustic's daily upload job:

Fail Mail (civi1002) run-job: Silverpop emails - Upload the main database file (DatabaseUpdate) failed with code 1
See the logs for more information: /var/log/process-control/silverpop_emails_upload_data_file/silverpop_emails_upload_data_file-20250522-030425.log

Attempting to fix this, we rerun the data job (ID: 250855252) but we noticed it took only 2 minutes to complete and had about 260k contacts parsed.

Usually, when run at 09:00 (GMT) this job takes around ~25 minutes and has the full 32 million contacts.
There are occasions when this job runs at 03:00 (GMT) and doesn't have the full number of contacts.

Previous 09:00 examples:

  • 250853767
  • 250840756
  • 250813723

Previous 03:00 examples:

  • 250837802
  • 250823816

*edit: the job id hyperlinks don't work as expected. Go to All Data Jobs and CTRL/CMD + F the job ids to click on each entry.

Event Timeline

ppenloglou created this task.May 22 2025, 10:41 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 22 2025, 10:41 AM
ppenloglou updated the task description. (Show Details)May 22 2025, 10:57 AM
RLopez-WMF subscribed.May 22 2025, 2:42 PM
AKanji-WMF triaged this task as High priority.May 22 2025, 4:20 PM
AKanji-WMF moved this task from Triage to Chaos Crew Backlog on the Fundraising-Backlog board.
jgleeson claimed this task.May 22 2025, 5:25 PM
jgleeson moved this task from Backlog to In Progress on the Fundraising Tech - Chaos Crew board.
Eileenmcnaughton added subscribers: • PPenloglou-WMF, Eileenmcnaughton.May 22 2025, 5:49 PM

Hi @PPenloglou-WMF this doesn't seem quite clear- the nightly job that has 30 million-ish rows is just the checksum job - the main data base update job is only a few million rows - I updated the list of the 4 of them over on the docs

https://wikitech.wikimedia.org/wiki/Fundraising/Data_and_Integrated_Processes/Acoustic_Integration

The error DOES look like it relates to the main job - but I would not expect it to normally be 32 M rows

jgleeson added a comment.May 22 2025, 6:05 PM

@Eileenmcnaughton, that looks right to me. The numbers for the main job look consistent based on what I see in the logs below, and the checksum stats are all around the 32M mark. I think we're good @ppenloglou based on this

162 │ 2025-05-21 02:57:18,835 INFO Begin Silverpop Export
163 │ 2025-05-21 02:57:18,901 INFO Starting full data export
164 │ 2025-05-21 02:58:09,906 INFO Wrote 281980 rows
165 │ 2025-05-21 02:58:09,963 INFO Starting unsubscribe data export
166 │ 2025-05-21 02:59:11,329 INFO Wrote 10985157 rows
167 │ 2025-05-21 02:59:11,380 INFO Starting unsubscribe data export
168 │ 2025-05-21 03:00:13,308 INFO Wrote 10985157 rows
169 │ 2025-05-21 03:00:13,361 INFO Starting email with correspondent checksum data export
170 │ 2025-05-21 03:03:38,452 INFO Wrote 32373728 rows

162 │ 2025-05-20 02:59:40,532 INFO Begin Silverpop Export
163 │ 2025-05-20 02:59:40,592 INFO Starting full data export
164 │ 2025-05-20 03:01:32,943 INFO Wrote 291163 rows
165 │ 2025-05-20 03:01:32,995 INFO Starting unsubscribe data export
166 │ 2025-05-20 03:02:36,571 INFO Wrote 10980952 rows
167 │ 2025-05-20 03:02:36,624 INFO Starting unsubscribe data export
168 │ 2025-05-20 03:03:40,627 INFO Wrote 10980952 rows
169 │ 2025-05-20 03:03:40,679 INFO Starting email with correspondent checksum data export
170 │ 2025-05-20 03:07:11,268 INFO Wrote 32375327 rows

162 │ 2025-05-19 02:56:33,580 INFO Begin Silverpop Export
163 │ 2025-05-19 02:56:33,646 INFO Starting full data export
164 │ 2025-05-19 02:58:20,028 INFO Wrote 295254 rows
165 │ 2025-05-19 02:58:20,082 INFO Starting unsubscribe data export
166 │ 2025-05-19 02:59:22,158 INFO Wrote 10979781 rows
167 │ 2025-05-19 02:59:22,207 INFO Starting unsubscribe data export
168 │ 2025-05-19 03:00:24,751 INFO Wrote 10979781 rows
169 │ 2025-05-19 03:00:24,802 INFO Starting email with correspondent checksum data export
170 │ 2025-05-19 03:03:47,826 INFO Wrote 32374989 rows

163 │ 2025-05-18 02:55:41,443 INFO Starting full data export
164 │ 2025-05-18 02:57:27,692 INFO Wrote 296161 rows
165 │ 2025-05-18 02:57:27,745 INFO Starting unsubscribe data export
166 │ 2025-05-18 02:58:28,750 INFO Wrote 10978731 rows
167 │ 2025-05-18 02:58:28,802 INFO Starting unsubscribe data export
168 │ 2025-05-18 02:59:30,167 INFO Wrote 10978731 rows
169 │ 2025-05-18 02:59:30,218 INFO Starting email with correspondent checksum data export
170 │ 2025-05-18 03:02:53,001 INFO Wrote 32374256 rows (edited)

ppenloglou added a comment.May 22 2025, 6:33 PM

Thanks @Eileenmcnaughton , I don't think I had a clear understanding of each job... Thanks for sharing the wikitech page as well!

jgleeson moved this task from In Progress to Done on the Fundraising Tech - Chaos Crew board.May 30 2025, 2:59 PM
XenoRyet closed this task as Resolved.Jun 3 2025, 7:12 PM
XenoRyet set Final Story Points to 2.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits