Page MenuHomePhabricator
Create Task
Maniphest T239756

Investigate why orphan-slayer job is running for hours
Closed, ResolvedPublic
Actions

Description

When looking at the process list on civi1001 we noticed that the orphan-slater for paypal and ingenico was running for 12 and 5 hours plus respectively.

jgleeson@civi1001:/var/spool/prometheus$ ps aux --sort=-%mem,-%cpu | head -n 50
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
www-data 14647  3.9  4.2 1047620 689480 ?      S    16:00  12:52 php /usr/share/php/drush/drush.php --user=1 -v -r /srv/org.wikimedia.civicrm/drupal orphan-slayer paypal_ec
www-data 26968  4.0  1.5 615740 254836 ?       S    19:00   6:01 php /usr/share/php/drush/drush.php --user=1 -v -r /srv/org.wikimedia.civicrm/drupal orphan-slayer ingenico

Details

SubjectRepoBranchLines +/-
Add batch and time_limit options to orphan slayerwikimedia/fundraising/crmmaster+25 -5
Customize query in gerrit

Related Objects
Search...

Event Timeline

jgleeson created this task.Dec 3 2019, 9:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 3 2019, 9:30 PM
• DStrine moved this task from Triage to Sprint +1 on the Fundraising-Backlog board.Dec 5 2019, 6:33 PM
• DStrine moved this task from Sprint +1 to Sprint +2 on the Fundraising-Backlog board.Jan 6 2020, 4:42 PM
• DStrine moved this task from Sprint +2 to Q3 2021-2022 on the Fundraising-Backlog board.Mar 3 2020, 4:33 PM
Ejegg subscribed.Dec 6 2021, 11:24 PM

@jgleeson I found a couple of candidates for ways to speed it up:

T249276 (processing messages that we know are failed while donor is on payments-wiki)

T205566 (making extra API calls which are guaranteed to fail after finding out that the payment is unprocessable)

Ejegg added a comment.Dec 7 2021, 10:29 PM

One more thing that might help: T297246

Ejegg added a subtask: T297246: Delete PayPal pending messages more aggressively.Dec 16 2021, 6:44 PM
Ejegg mentioned this in T298067: Donations QC seems to leave some pending rows it shouldn't.Dec 21 2021, 1:08 AM
• DStrine closed subtask T297246: Delete PayPal pending messages more aggressively as Resolved.Dec 21 2021, 8:09 PM
gerritbot added a comment.Dec 22 2021, 10:52 PM

Change 749586 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Add batch and time_limit options to orphan slayer

https://gerrit.wikimedia.org/r/749586

gerritbot added a project: Patch-For-Review.Dec 22 2021, 10:52 PM
Ejegg mentioned this in T298219: Endowment donations not being tagged as endowment.Dec 23 2021, 12:00 AM
gerritbot added a comment.Dec 23 2021, 1:42 AM

Change 749586 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Add batch and time_limit options to orphan slayer

https://gerrit.wikimedia.org/r/749586

Maintenance_bot removed a project: Patch-For-Review.Dec 23 2021, 2:10 AM
Ejegg added a project: fundraising sprint Yeet-coaster.Dec 23 2021, 2:38 AM
Ejegg moved this task from Q3 2021-2022 to Current Sprint on the Fundraising-Backlog board.
jgleeson moved this task from Backlog to Doing on the fundraising sprint Yeet-coaster board.Dec 23 2021, 4:47 PM
Ejegg moved this task from Doing to Done on the fundraising sprint Yeet-coaster board.Dec 24 2021, 12:54 AM
Ejegg merged a task: T297113: MySQL went away during Paypal Orphan Slayer run.Jan 10 2022, 9:01 PM
• DStrine triaged this task as Medium priority.Jan 14 2022, 5:15 PM
• DStrine closed this task as Resolved.Jan 18 2022, 8:14 PM
AnnWF added a project: FR-Alerts.Jul 15 2022, 9:54 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL