This extension looks like it might be worth considering https://lab.civicrm.org/extensions/batchupdateactivitystatus - it gets past the 100 limit

No unit test but it's pretty cleanly written

@DStrine and @Eileenmcnaughton adding to this task. I've been working with @jkim_wikimedia to get her event activities updated for future events and the batch updating activities is becoming a bit of a headache for her since she needs to update groups of contacts of 5000 and over and Civi times out frequently. Is there any way around it?

Hi all, I wanted to check in on this task to see if the limit could be increased? Thanks!

@DStrine @Eileenmcnaughton Hi. Jerry and I are attempting to add activities to donors records indicating that they received an initiation to our virtual events this year. Because this year was so very different and we invited a much larger number of people to each event (one was around 100k) we are unable to do this through traditional means - meaning, import donors into a group and use civi to add an activity. Is there someone that can help us get these activities added to these donor records perhaps in a different way?

It would be much appreciated as we are attempting to communicate with these donors and have no way of knowing based off of their records if they were invited to an event this year, also, we are very interested to have this information available to analytics.

@DStrine Can you give us a sense of when Fr-Tech will be able to asses this? Even just knowing that (ie not even when this work could possibly be done) will help us with our Q3 donor stewardship.

Sorry I didn't see the previous comments on this. I'll talk with the team today.

Change 651808 had a related patch set uploaded (by Ejegg; owner: Ejegg):
[wikimedia/fundraising/crm@master] Civi ext to batch update activities

https://gerrit.wikimedia.org/r/651808

We could get this done today. @CaitVirtue @NNichols @jkim_wikimedia I'm trying to ping Nora and Jerry on fr-all in slack. I'm not aware of a MG channel. Nora tells me this would be good to have over the break. @Ejegg is setting something up on staging but we need the data to try it.

I enabled the extension suggested by @Eileenmcnaughton on staging, however I'm not sure it's going to do what's necessary.

According to the doc at https://lab.civicrm.org/extensions/batchupdateactivitystatus/-/blob/master/README.md it seems to just update the statuses of already-created activity/contact records but doesn't create new ones.

@jkim_wikimedia and @RLewis, want to see if you can find data on staging to test with? You would do the search under Search->activities, from the results choose the task 'update activity status' and choose the new status, then leave the browser window open till the progress bar finishes.

I am hearing from MG that this doesn't have to be done right now or over the break. @jkim_wikimedia will get the list onto the server and we'll take a look at it in January.

@jkim_wikimedia bumping this. Any way we could get that file?

@DStrine Here's the filepath: smb://filesrv1.corp.wikimedia.org/fundraising/Development/Events/2020 Events/Activities
Below are the activity details for each event. Let me know if you have any questions!

PST Event:
Type: Event - Fundraising Reception
Date: 12/10/2020
Subject: Virtual
Location: Virtual
Status: Invited

EST Event:
Type: Event - Fundraising Reception
Date: 9/17/2020
Subject: Virtual
Location: Virtual
Status: Invited

CST Event:
Type: Event - Fundraising Reception
Date: 10/28/2020
Subject: Virtual
Location: Virtual
Status: Invited

@jkim_wikimedia can you just check the detail I added to civicrm/contact/view?reset=1&cid=44 to make sure the activity is correct

@Eileenmcnaughton yes, that's correct!

@jkim_wikimedia great - I'm going to do a few UI tests on that file - so I'm going to remove the ones I've imported from my copy - I just want to get an idea of when it borks. That means that set will be partially imported for a bit

@jkim_wikimedia so the reason this is confusing to me when I re-read is that it asks for the ability to update activities in the blurb - but I AM just adding them aren't I?

Also @jkim_wikimedia there are around 163 activities already for 10 Dec - but they don't seem to be in the EST sheet so I assume they don't overlap

This ticket is a bit confusing because the ticket is about updating activities but along the way it became about adding /importing activities.

I looked into the latter today and found that after around about 12000 rows the csv becomes larger than 1MB - at which point NGINX rejects it with a 413 error like https://www.cyberciti.biz/faq/linux-unix-bsd-nginx-413-request-entity-too-large/ @Jgreen @Dwisehaupt can we increase this limit. The files are around 8 times as large as the limit permits at the moment if we try to do them in one go. (Note CiviCRM also has a 2MB limit but that can easily be changed).

@jkim_wikimedia - I imported 100% of the EST batch - in 12k increments. By the time I'd honed in on what the size limit was it seemed a lot cleaner to complete the file. If Jeff or Dallas is OK to increase the limit I'll try one of the other files

@Eileenmcnaughton thank you so much Eileen and I'm sorry about the confusion. That would be amazing if the limit could be increased so I wouldn't have to bother you all to help me with this import!

@Eileenmcnaughton I have bumped the client_max_body_size to 10M. Feel free to attempt the full file when you have updated the CiviCRM limit.

[frack::puppet] 0ec21fc9 bump nginx client_max_body_size to 10M for civicrm

@Dwisehaupt when I go to alter in civi I get

Please correct the following errors in the form fields below:
Maximum file size cannot exceed Upload max size ('upload_max_filesize') as defined in PHP.ini.

I expect this is best fixed by you although I might be able to do an ini_set() in the settings.php file

@Eileenmcnaughton I have templated and increased the upload_max_filesize to 10M in the apache2, cgi, and cli php.ini configs for both civicrm and fundraising_qa. You should be clear to test again.

[frack::puppet] fb06c5ca Templatize upload_max_filesize for php.ini
[frack::puppet] 4bea64fa Increase upload_max_filesize on civicrm and fundraising_qa roles
[frack::puppet] 173d9aa6 Bump upload_max_filesize to 10M for fundraising_qa php::mod::cgi

@Dwisehaupt I'm still being notified that Maximum file size cannot exceed Upload max size ('upload_max_filesize') as defined in PHP.ini.

I'm unsure if it is because of the formatting issue described here https://github.com/civicrm/civicrm-core/pull/19382 & am trying to port that patch to confirm

Hmm - @Dwisehaupt it's now clealy telling me that upload_max_filesize is 2m in the output from this line

https://github.com/civicrm/civicrm-core/pull/19382/files#diff-3ea2fa005106d03c767510665622cffac3df5b5244aed14208dc1e6c99cc08baR101

settings.php/civicrm.settings.php don't seem to be doing any ini_set to mess with it

ohhh

drush @wmff php-eval "echo ini_get('upload_max_filesize');"
10M

oh - but that is CLI - not UI - I've found a way to see ini settings via the UI & I see

"upload_max_filesize": "2M",

@Eileenmcnaughton I've double checked and all of the ini files in use list 10M. There is one phpdbg file still at 2M, but that shouldn't come into play in production:

dallas@civi1001:/etc/php$ grep -r upload_max_filesize *
7.3/cgi/php.ini:upload_max_filesize = 10M
7.3/apache2/php.ini:upload_max_filesize = 10M
7.3/cli/php.ini:upload_max_filesize = 10M
7.3/phpdbg/php.ini:upload_max_filesize = 2M

The apache and nginx processes had been restarted on the 12th. I went ahead and restarted them today just in case there was some odd caching issue.

Is it possible that the maximum file size set here is getting in the way? https://civicrm.wikimedia.org/civicrm/admin/setting/misc?reset=1

@Dwisehaupt when I try to change the max file size on that page it does this check
https://github.com/civicrm/civicrm-core/pull/19382/files#diff-3ea2fa005106d03c767510665622cffac3df5b5244aed14208dc1e6c99cc08baR101
& says it can't change it. There is also an api call that looks it up & reports the 2MB

Can we try changing that place that shouldn't come into play & see?

Just noting that the trick to filling down the repetitive data is to highlight the cells & then press cntrl & double click on the drag arrow

Total file winds up as 8.8MB with fill

@Dwisehaupt I just noticed that post_max_size AND upload_max_size are mentioned in the warning message...

Note: Upload max filesize ('upload_max_filesize') should not exceed Post max size ('post_max_size') as defined in PHP.ini, please check with your system administrator.

I updated the file size using the api in civi to 10mb - oddly it rejected a 5mb file saying it exceeded the 10mb limit....

Looks like I just managed a 3,3MB file with 30 k rows - it's all odd

@jkim_wikimedia these files are imported now - I didn't hold one back for further testing as I'm away for a week now & I thought it would be better to get them all in but I couldn't confirm the new max - it is file size related & 3.3 MB with 33k rows is importing in one go

I noticed you have duplicate contacts in some of the files though so it's not 1 activity per contact but rather 1 per row less a handful I spotted & deleted

@Eileenmcnaughton That makes some sense. The max post size was set at 8mb and thus the 10mb file would overrun that. I have adjusted the templates to have a variable for post_max_size. On the civicrm and fundraising_qa roles I have increased the post_max_size to 12M which should cover the possible 10M file plus overhead.

[frack::puppet] 77358a9d Template post_max_size for php.ini, adjust for civicrim and fundraising_qa roles
[frack::puppet] ffeb29f9 Fix template variable name to post_max_size for php.ini

@Eileenmcnaughton @Dwisehaupt thank you both for your work on this!

@Dwisehaupt it's possible to see ini variables at this url civicrm/api3#explorer using System & get

As of today I see

"post_max_size": "8M",
"upload_max_filesize": "10M",

So it looks like the latter setting has 'taken' now but not the former - does that call for another service restart?

Change 651808 abandoned by Ejegg:
[wikimedia/fundraising/crm@master] Civi ext to batch update activities

Reason:
Doesn't do the kind of updates we wanted

https://gerrit.wikimedia.org/r/651808

Just noting that the search kit did a pretty OK job for updating activities in my tested - but this issue turned out to be about importing rather than updating

@NNichols this phab got taken over for a while with 'help me import activities' but for updating activities - what are you trying to update? It's pretty doable to update a value or 2 using a search created using search kit

I just pushed this back to backlog until we get clarity

I'm gonna close this cos I'm pretty sure it got fixed along the line- the main issue is the ticket is kinda confusing in terms of what is required but bulk updates can be done using search kit now