Page MenuHomePhabricator

Get imports running in the background on Staging
Closed, ResolvedPublic

Description

We are quite close now

Event Timeline

OK so this is needed https://github.com/civicrm/civicrm-core/pull/24761 & we need to add coworker which is linked to in there but I need to dig in

@Dwisehaupt @Jgreen the CiviCRM utility for background processing is

https://lab.civicrm.org/dev/coworker

  • Can we install that on Staging so I can try it out? (I've been running it locally)

What it does is scan the queue within CiviCRM & claims & runs any jobs. Currently just looking at using for imports so the user doesn't need to rely on ajax but there is some potential for broader usage (since it could get us away from cron scheduling magic to rkeep jobs from running at the same time)

We are looking to test coworker using the phar to verify if it will suit our needs. Ideally it can be run by any fr-tech user and ultimately may run as a service.

Once we verify that coworker is working as we want, we will look at the options for mirroring the repo and adding it into our regular deploy framework.

@Dwisehaupt I should note locally I've been using it with the cv utility - https://github.com/civicrm/cv

It's also possible to use it with drush (which is what I had been thinking) - but at some point we probably will want to install cv too as it will probably make sense as part of exiting our current drupal (in case it is easier to do at the same time)

@Eileenmcnaughton I have copied the latest builds of coworker and cv into /usr/local/bin on frdev1001. They are named coworker.phar and cv.phar at this point to be clear that it isn't the end state.

You are able to invoke them directly for all the testing you want to do at this point.

Change 864882 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] A whole lotta upstream patches for background imports

https://gerrit.wikimedia.org/r/864882

Just noting that I have managed to run coworker on staging - I couldn't run it using 'cv' because our drush wrapper has a user thing going on & cv doesn't -so it ran as me & then I hit file permission issues writing to templates_c

However, this DID work from the drupal dir

/usr/local/bin/coworker.phar run --pipe='drush ev "civicrm_initialize(); Civi::pipe();"'

I could not use the wmff alias though - I think that invoked user 1

I've moved this into 'blocked not fr-tech' because I got this going on staging - but I feel like we can't deploy until after Dec & I'm not quite sure how to categorise it in the meantime.

Eileenmcnaughton renamed this task from Get imports running in the background to Get imports running in the background on Staging.Dec 6 2022, 8:10 PM
Eileenmcnaughton closed this task as Resolved.
Eileenmcnaughton set Final Story Points to 8.

Change 864882 abandoned by Eileen:

[wikimedia/fundraising/crm@master] A whole lotta upstream patches for background imports

Reason:

https://gerrit.wikimedia.org/r/864882