Page MenuHomePhabricator
Create Task
Maniphest T357470

replace our drush scripts with WMFQueue.consume api
Closed, ResolvedPublic
Actions

Description

I had a first go at the api https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1002753 - but I think ideally we want to switch to calling via cv not drush in process control - because drush is a drupal thing

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Eileenmcnaughton created this task.Feb 13 2024, 8:33 PM
Eileenmcnaughton moved this task from Triage to Current Sprint on the Fundraising-Backlog board.
Eileenmcnaughton moved this task from Current Sprint to Sprint +1 on the Fundraising-Backlog board.
AKanji-WMF moved this task from Sprint +1 to Current Sprint on the Fundraising-Backlog board.Feb 13 2024, 8:38 PM
AKanji-WMF added a project: Fundraising Sprint: canOfWorms(🪱).Feb 13 2024, 9:23 PM
Eileenmcnaughton updated the task description. (Show Details)Feb 20 2024, 3:01 AM
Eileenmcnaughton moved this task from Backlog to Ready for Review on the Fundraising Sprint: canOfWorms(🪱) board.Feb 23 2024, 8:05 PM
AKanji-WMF edited projects, added Fundraising Sprint: didAnyoneTryThis(); removed Fundraising Sprint: canOfWorms(🪱).Feb 29 2024, 5:43 PM
Eileenmcnaughton claimed this task.Mar 4 2024, 8:47 PM
Eileenmcnaughton moved this task from Backlog to Doing on the Fundraising Sprint: didAnyoneTryThis() board.
Dwisehaupt added a project: fundraising-tech-ops.Mar 4 2024, 8:49 PM
Eileenmcnaughton added subscribers: Dwisehaupt, Jgreen.Mar 4 2024, 8:50 PM

@Dwisehaupt @Jgreen

image.png (702×1 px, 313 KB)

Dwisehaupt added a comment.Mar 7 2024, 1:14 AM

Ok. This is now in place on both civi and frdev roles. frdev will probably throw an error related to watchdog() but that is known and independent of this.

dallas@civi1002:~$ wmf-cv vars:show
{
    "CIVI_CORE": "/srv/org.wikimedia.civicrm/drupal/sites/all/modules/civicrm/",
    "CIVI_DB_ARGS": "",
    "CIVI_DB_DSN": "civirpow://",

<snip>

Feel free to test it out.

Eileenmcnaughton added a comment.Mar 7 2024, 2:31 AM

I just tested the upiQueue consumer & this ran a treat

wmf-cv api4 --user=root -vv WMFQueue.Consume timeLimit=280 messageLimit=1 queueConsumer=UpiDonations queueName=upi-donations

that would be the slow start command & non-slow-start would be without the messageLimit=1

@Dwisehaupt could we update process control such that if no slow-start version is defined it just runs the main script with messageLimit=1 appended? It seems unnecessary to have to put it in the yaml once we have standardised our workings

I also tried

wmf-cv api4 --user=root -vv WMFQueue.Consume timeLimit=280 messageLimit=1 queueConsumer=OptIn queueName=opt-in
wmf-cv api4 --user=root -vv WMFQueue.Consume timeLimit=280 messageLimit=1 queueConsumer=ContributionTracking queueName=contribution-tracking
Dwisehaupt added a comment.Mar 7 2024, 5:43 PM

@Eileenmcnaughton The way process control is designed right now is to treat the command and slow-start bits of config in the same manner. It treats them both as a list since there can be multiple commands passed in a job.

An example is the silverpop_daily job:

---
name: Silverpop build and export
command:
    - /usr/bin/run-job --job silverpop_emails_build_export_files
    - /usr/bin/run-job --job silverpop_emails_upload_files

I agree that removing the duplication of entries would be good and would likely cut down on copy/paste errors. However, I think it would be a pretty big rework of process control to handle appending an argument in the case of slow start and I'm not sure of the priority of that work.

Eileenmcnaughton added a comment.Mar 7 2024, 6:08 PM

@Dwisehaupt probably doesn't make sense to do if it is a big rewrite

gerritbot added a comment.Mar 11 2024, 9:53 PM

Change 1010328 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] More detailed context

https://gerrit.wikimedia.org/r/1010328

gerritbot added a project: Patch-For-Review.Mar 11 2024, 9:53 PM
gerritbot added a comment.Mar 11 2024, 10:13 PM

Change 1010331 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Stop enabling opt in queue consumer

https://gerrit.wikimedia.org/r/1010331

gerritbot added a comment.Mar 11 2024, 10:15 PM

Change 1010332 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Remove OptIn module

https://gerrit.wikimedia.org/r/1010332

Eileenmcnaughton added a comment.Mar 11 2024, 10:15 PM

optin is switched!!!

Eileenmcnaughton mentioned this in rWFDVc582f0fdbcd5: Migrate OptIn to use cv.Mar 12 2024, 3:24 AM
XenoRyet added a project: Fundraising Sprint: explain($LikeImFive).Mar 12 2024, 7:26 PM
gerritbot added a comment.Mar 12 2024, 10:46 PM

Change 1010328 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] More detailed context

https://gerrit.wikimedia.org/r/1010328

gerritbot added a comment.Mar 12 2024, 10:47 PM

Change 1010331 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Stop enabling opt in queue consumer

https://gerrit.wikimedia.org/r/1010331

gerritbot added a comment.Mar 12 2024, 10:48 PM

Change 1010332 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Remove OptIn module

https://gerrit.wikimedia.org/r/1010332

Maintenance_bot removed a project: Patch-For-Review.Mar 12 2024, 11:30 PM
Ejegg moved this task from Backlog to Doing on the Fundraising Sprint: explain($LikeImFive) board.Mar 18 2024, 7:38 PM
AKanji-WMF edited projects, added Fundraising Sprint: gone(); removed Fundraising Sprint: explain($LikeImFive).Mar 26 2024, 8:43 PM
AKanji-WMF moved this task from Current Sprint to TEMP Sprint G on the Fundraising-Backlog board.Mar 26 2024, 8:47 PM
gerritbot added a comment.Apr 3 2024, 11:33 PM

Change #1016863 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/dev@master] Migrate RecurringModifyAmount to use cv

https://gerrit.wikimedia.org/r/1016863

gerritbot added a project: Patch-For-Review.Apr 3 2024, 11:33 PM

Change #1016864 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/dev@master] Migrate Recurring to use cv

https://gerrit.wikimedia.org/r/1016864

gerritbot added a comment.Apr 3 2024, 11:33 PM

Change #1016865 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/dev@master] Migrate BannerHistory to use cv

https://gerrit.wikimedia.org/r/1016865

gerritbot added a comment.Apr 4 2024, 12:00 AM

Change #1016889 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/dev@master] Migrate Unsubscribe to use cv

https://gerrit.wikimedia.org/r/1016889

gerritbot added a comment.Apr 4 2024, 12:00 AM

Change #1016890 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/dev@master] Migrate EmailPreferences to use cv

https://gerrit.wikimedia.org/r/1016890

gerritbot added a comment.Apr 4 2024, 12:00 AM

Change #1016891 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/dev@master] Migrate UpiDonations to use generic queue

https://gerrit.wikimedia.org/r/1016891

gerritbot added a comment.Apr 4 2024, 12:18 AM

Change #1016893 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/dev@master] Migrate Refund queue to use generic queue

https://gerrit.wikimedia.org/r/1016893

gerritbot added a comment.Apr 4 2024, 12:18 AM

Change #1016894 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/dev@master] Migrate ContributionTracking queue to use generic queue

https://gerrit.wikimedia.org/r/1016894

gerritbot added a comment.Apr 4 2024, 12:18 AM

Change #1016895 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/dev@master] Migrate Donation queue to use generic queue

https://gerrit.wikimedia.org/r/1016895

gerritbot added a comment.Apr 4 2024, 12:18 AM

Change #1016896 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/dev@master] Migrate PaymentsInit queue to use generic queue

https://gerrit.wikimedia.org/r/1016896

gerritbot added a comment.Apr 4 2024, 12:18 AM

Change #1016897 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/dev@master] Migrate Antifraud queue to use generic queue

https://gerrit.wikimedia.org/r/1016897

gerritbot added a comment.Apr 4 2024, 1:56 AM

Change #1016863 merged by Ejegg:

[wikimedia/fundraising/dev@master] Migrate RecurringModifyAmount to use cv

https://gerrit.wikimedia.org/r/1016863

Eileenmcnaughton mentioned this in rWFDVed987f2f76ce: Migrate RecurringModifyAmount to use cv.Apr 4 2024, 1:56 AM
gerritbot added a comment.Apr 8 2024, 9:33 PM

Change #1016890 merged by Eileen:

[wikimedia/fundraising/dev@master] Migrate EmailPreferences to use cv

https://gerrit.wikimedia.org/r/1016890

Eileenmcnaughton mentioned this in rWFDVd012ea39333d: Migrate EmailPreferences to use cv.Apr 8 2024, 9:34 PM
gerritbot added a comment.Apr 8 2024, 9:55 PM

Change #1016865 merged by Eileen:

[wikimedia/fundraising/dev@master] Migrate BannerHistory to use cv

https://gerrit.wikimedia.org/r/1016865

Eileenmcnaughton mentioned this in rWFDVf44ef9bc8c06: Migrate BannerHistory to use cv.Apr 8 2024, 9:55 PM
AKanji-WMF moved this task from TEMP Sprint G to Current Sprint on the Fundraising-Backlog board.Apr 9 2024, 7:45 PM
Cstone mentioned this in T357001: Standardize message and time limit options for all scripts.Apr 9 2024, 7:48 PM
gerritbot added a comment.Apr 9 2024, 10:24 PM

Change #1016864 merged by Cstone:

[wikimedia/fundraising/dev@master] Migrate Recurring to use cv

https://gerrit.wikimedia.org/r/1016864

Eileenmcnaughton mentioned this in rWFDVe679e047bbfb: Migrate Recurring to use cv.Apr 9 2024, 10:25 PM
gerritbot added a comment.Apr 9 2024, 10:27 PM

Change #1018363 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Stop enabling recurring

https://gerrit.wikimedia.org/r/1018363

gerritbot added a comment.Apr 9 2024, 10:34 PM

Change #1018364 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Stop enabling unsubscribe

https://gerrit.wikimedia.org/r/1018364

gerritbot added a comment.Apr 9 2024, 10:36 PM

Change #1016889 merged by Eileen:

[wikimedia/fundraising/dev@master] Migrate Unsubscribe to use cv

https://gerrit.wikimedia.org/r/1016889

Cstone mentioned this in rWFDV3573ee321e3f: Migrate Unsubscribe to use cv.Apr 9 2024, 10:36 PM
gerritbot added a comment.Apr 9 2024, 10:43 PM

Change #1016891 merged by Cstone:

[wikimedia/fundraising/dev@master] Migrate UpiDonations to use generic queue

https://gerrit.wikimedia.org/r/1016891

Cstone mentioned this in rWFDV3f63568841b2: Migrate UpiDonations to use generic queue.Apr 9 2024, 10:43 PM
gerritbot added a comment.Apr 9 2024, 10:47 PM

Change #1016893 merged by Cstone:

[wikimedia/fundraising/dev@master] Migrate Refund queue to use generic queue

https://gerrit.wikimedia.org/r/1016893

Cstone mentioned this in rWFDV8b6f37175a08: Migrate Refund queue to use generic queue.Apr 9 2024, 10:48 PM
gerritbot added a comment.Apr 9 2024, 10:58 PM

Change #1016894 merged by Cstone:

[wikimedia/fundraising/dev@master] Migrate ContributionTracking queue to use generic queue

https://gerrit.wikimedia.org/r/1016894

Cstone mentioned this in rWFDVef9cc769c5cb: Migrate ContributionTracking queue to use generic queue.Apr 9 2024, 10:59 PM
gerritbot added a comment.Apr 9 2024, 11:07 PM

Change #1016895 merged by Cstone:

[wikimedia/fundraising/dev@master] Migrate Donation queue to use generic queue

https://gerrit.wikimedia.org/r/1016895

Cstone mentioned this in rWFDV904c69118ccb: Migrate Donation queue to use generic queue.Apr 9 2024, 11:08 PM
gerritbot added a comment.Apr 9 2024, 11:17 PM

Change #1016896 merged by Cstone:

[wikimedia/fundraising/dev@master] Migrate PaymentsInit queue to use generic queue

https://gerrit.wikimedia.org/r/1016896

gerritbot added a comment.Apr 9 2024, 11:17 PM

Change #1016897 merged by Cstone:

[wikimedia/fundraising/dev@master] Migrate Antifraud queue to use generic queue

https://gerrit.wikimedia.org/r/1016897

Cstone mentioned this in rWFDV1692814c25a4: Migrate Antifraud queue to use generic queue.Apr 9 2024, 11:17 PM
Cstone mentioned this in rWFDV33ab6535918a: Migrate PaymentsInit queue to use generic queue.
gerritbot added a comment.Apr 10 2024, 12:23 AM

Change #1018390 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Stop enabling remaining queue2civicrm drush modules

https://gerrit.wikimedia.org/r/1018390

gerritbot added a comment.Apr 10 2024, 1:37 AM

Change #1018363 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Stop enabling recurring

https://gerrit.wikimedia.org/r/1018363

gerritbot added a comment.Apr 10 2024, 1:41 AM

Change #1018364 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Stop enabling unsubscribe

https://gerrit.wikimedia.org/r/1018364

Cstone moved this task from Backlog to Done on the Fundraising Sprint: gone() board.Apr 11 2024, 9:10 PM
gerritbot added a comment.Apr 17 2024, 7:59 PM

Change #1018390 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Stop enabling remaining queue2civicrm drush modules

https://gerrit.wikimedia.org/r/1018390

XenoRyet closed this task as Resolved.Tue, Apr 23, 7:17 PM
XenoRyet set Final Story Points to 8.
Maintenance_bot removed a project: Patch-For-Review.Fri, Apr 26, 6:13 PM
Dwisehaupt moved this task from Triage to Done on the fundraising-tech-ops board.Fri, May 3, 11:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL