@LeanneS is checking if this was just one day or if this occurs at the same time every day.

Update from Engage that it also happened on Monday afternoon:

"This happened on Monday (12/5) around 4:20 PM.

We worked in the system until 6:00 PM on 12/6 and 9:40 PM on 12/7 without any issues."

I took a look at the scheduled jobs and could see nothing that would cause a regular slow query at this time. I also trawled through the slow queries over the last 24 hours. I found one that can be improved easily and caused 12 instances of slowness to Megan or Michael over the period (CRM-19752). I also found that the query to determine whether to run dedupes is taking longer than it needs to as there is no index on civicrm_contact.created_date. I can see how to improve that.

The latter item will not have caused any slowness for Engage, the former probably will not have but it's possible.

I'm not sure where to find slow query logs older than 24 hours - @Jgreen ?

https://docs.google.com/spreadsheets/d/1PQBSB_cx6umWoCRTXJrjVE-BaPaTK3MNHF5dk7DOjFQ/edit#gid=0

I'll take another look at the log tomorrow to see what has gone on since now

grep 'Query_time: [2-9][0-9]' fundraisingdb-mysql-slow
grep 'Query_time: [1-9][0-9][0-9]' fundraisingdb-mysql-slow

I'm not sure where to find slow query logs older than 24 hours - @Jgreen ?

All the ./remote/* logs rotate into the archive, at indium:/a/archive/indium/logs

I'm pretty certain the issue on 8 Dec was caused by the silverpop export being run at an odd time - see the highlighted row below. These rows all refer to the timing of a mysql query in the silverpop export 'INSERT INTO silverpop_export_highest...' which happens about 1.5 hours in

Please see the latest update from Engage today:

I am writing regarding a few errors we are encountering when processing in Civi. The movie and image attached are happening at random times during processing. The movie was recorded on Tuesday night (12/13) around 7:00 PM EST, it shows the screen in a constant loop unable to update transactions, in this situation we are not able enter any information. If we close the browser, reopen it, log back into Civi and attempt to update a record we received the same result.

The included image shows an error we are receiving when entering Benevity batches into Civi without a batch number. The error happened on and off yesterday, this picture was taken at 5:40 PM EST. The problem we are having is when the screen loads to enter a transaction in random cases there is not an area to enter a batch number, our operators are then forced to refresh the page until the batch number area appears (this can be anywhere from one refresh to twenty). We are then able to enter without a problem for a random amount of time before it appears again. If we try to enter the information when the batch number area is not visible we receive the attached error regarding batch number and reaching the server.

One additional item I would like to point out is that both computers used to access Civi experience these problems; however, they may not be at the same time. Meaning one operator may be enter without an issues while the other is experiencing the batch number/server issue. When the updating issue occurs it has effected both computers at the same time (though do to the time of day that it has happened only one operator is entering).

Could somebody text me the next time this happens? My cell is on the
collab wiki fundraising contacts page. Looking at database activity while
it's happening is often easier than trying to make sense of the past event
from logs, so I will try to take a look in realtime.

@Jgreen Definitely. I'll request that they notify me as soon as they experience any issues and then text you.

@LeanneS
FYI: In backlog grooming, we reviewed this ticket. We are going to leave it in the Analysis column for now. However, it happens again, we will pull it into the current sprint.

note slow query digging ticket is https://phabricator.wikimedia.org/T153134

Just checking back on this since there hasn't been an update, is it still a problem?

@Jgreen Thanks for checking. I haven't heard of any issues recently.