Page MenuHomePhabricator

Repeat sql queries created giant sort indexes and needed to be terminated
Closed, ResolvedPublic

Description

These 4 queries were running at the same time and had been running for over an hour. As a result of creating the indexes on disk, it reduced disk free from ~4.2T to ~300G.

| 3539660 | civicrm    | 10.64.40.115:37558 | civicrm  | Query       |    3919 | Creating sort index                                           | /* User : 1711 PID : 1431988 */SELECT `a`.`id` AS `id`, `a`.`tracking_date` AS `tracking_date`, `a`.`amount` AS `amount`, `a`.`currency` AS `currency`, `a`.`country` AS `country`, `a`.`utm_key` AS `utm_key`, `a`.`utm_medium` AS `utm_medium`, `a`.`referrer` AS `referrer`, `contribution_id_1`.`currency` AS `contribution_id.currency`
FROM `civicrm_contribution_tracking` a
LEFT JOIN civicrm_contribution `contribution_id_1` ON `a`.`contribution_id` =  `contribution_id_1`.`id`
ORDER BY `a`.`id` ASC, `a`.`tracking_date` ASC, `a`.`amount` ASC, `a`.`currency` ASC, `a`.`country` ASC, `a`.`utm_key` ASC, `a`.`utm_medium` ASC, `a`.`referrer` ASC
LIMIT 11
OFFSET 0 |    0.000 |
| 3539664 | civicrm    | 10.64.40.115:39970 | civicrm  | Query       |    3811 | Creating sort index                                           | /* User : 1711 PID : 1431973 */SELECT `a`.`id` AS `id`, `a`.`tracking_date` AS `tracking_date`, `a`.`amount` AS `amount`, `a`.`currency` AS `currency`, `a`.`country` AS `country`, `a`.`utm_key` AS `utm_key`, `a`.`utm_medium` AS `utm_medium`, `a`.`referrer` AS `referrer`, `contribution_id_1`.`currency` AS `contribution_id.currency`
FROM `civicrm_contribution_tracking` a
LEFT JOIN civicrm_contribution `contribution_id_1` ON `a`.`contribution_id` =  `contribution_id_1`.`id`
ORDER BY `a`.`id` ASC, `a`.`tracking_date` ASC, `a`.`amount` ASC, `a`.`currency` ASC, `a`.`country` ASC, `a`.`utm_key` ASC, `a`.`utm_medium` ASC, `a`.`referrer` ASC
LIMIT 11
OFFSET 0 |    0.000 |
| 3539665 | civicrm    | 10.64.40.115:39986 | civicrm  | Query       |    3859 | Creating sort index                                           | /* User : 1711 PID : 1429440 */SELECT `a`.`id` AS `id`, `a`.`tracking_date` AS `tracking_date`, `a`.`amount` AS `amount`, `a`.`currency` AS `currency`, `a`.`country` AS `country`, `a`.`utm_key` AS `utm_key`, `a`.`utm_medium` AS `utm_medium`, `a`.`referrer` AS `referrer`, `contribution_id_1`.`currency` AS `contribution_id.currency`
FROM `civicrm_contribution_tracking` a
LEFT JOIN civicrm_contribution `contribution_id_1` ON `a`.`contribution_id` =  `contribution_id_1`.`id`
ORDER BY `a`.`id` ASC, `a`.`tracking_date` ASC, `a`.`amount` ASC, `a`.`currency` ASC, `a`.`country` ASC, `a`.`utm_key` ASC, `a`.`utm_medium` ASC, `a`.`referrer` ASC
LIMIT 11
OFFSET 0 |    0.000 |
| 3539673 | civicrm    | 10.64.40.115:33078 | civicrm  | Query       |    3799 | Creating sort index                                           | /* User : 1711 PID : 1431916 */SELECT `a`.`id` AS `id`, `a`.`tracking_date` AS `tracking_date`, `a`.`amount` AS `amount`, `a`.`currency` AS `currency`, `a`.`country` AS `country`, `a`.`utm_key` AS `utm_key`, `a`.`utm_medium` AS `utm_medium`, `a`.`referrer` AS `referrer`, `contribution_id_1`.`currency` AS `contribution_id.currency`
FROM `civicrm_contribution_tracking` a
LEFT JOIN civicrm_contribution `contribution_id_1` ON `a`.`contribution_id` =  `contribution_id_1`.`id`
ORDER BY `a`.`id` ASC, `a`.`tracking_date` ASC, `a`.`amount` ASC, `a`.`currency` ASC, `a`.`country` ASC, `a`.`utm_key` ASC, `a`.`utm_medium` ASC, `a`.`referrer` ASC
LIMIT 11
OFFSET 0 |    0.000 |

If they would have kept running, we would have shortly run out of disk on the FRDB origin server.
https://frmon.wikimedia.org/goto/afb25dtfv8mpsd?orgId=1

Related Objects

Event Timeline

This (rapid increase of disk usage) may be a good place to look at prometheus based alerts for the velocity of a change. The large usage increase over a shorter period of time may have been detected earlier.

@Damilare These ones are your user id. Looks like SearchKit, do you recall what the search was?

Yes @Lars, I do I was trying to get all the contribution tracking records created yesterday on this SearchKit. I guess I can remove the joins as I don't need the columns as much.

@Damilare It looks like the issue might be sorting by so many columns (ORDER BY a.id ASC, a.tracking_date ASC, a.amount ASC, a.currency ASC, a.country ASC, a.utm_key ASC, a.utm_medium ASC, a.referrer ASC). Do you need to order by all of them?

That's interesting @Lars. I don't think I specified the order in Searchkit, at least not intentionally. I certainly don't need them ordered as its not a requirement for the use case and I can always order them locally as well.

@Damilare You're right, it looks like SK is adding those. Did you change the SK already? I just tried it and it returned results in ~1s.

No I haven't used it today. If you ran it earlier, it could have cached the results.

Change #1233306 had a related patch set uploaded (by Lars SG; author: Lars SG):

[wikimedia/fundraising/crm@master] Only sort contribution trackings by id by default

https://gerrit.wikimedia.org/r/1233306

Oh, I see that those are the default orderby added by the API, which it turns out uses the @searchFields annotation. But if we add an @orderBy annotation, we can limit that (I've just added a patch to do that).

But in this specific case, the SK currently is just looking at the previous day, but the queries above are without the WHERE. So I guess trying to get all of the contribution trackings and then sort them by multiple fields probably doesn't work so well. The above patch should avoid that issue in the future.

Change #1233306 abandoned by Lars SG:

[wikimedia/fundraising/crm@master] Only sort contribution trackings by id by default

https://gerrit.wikimedia.org/r/1233306

After a little thought, I decided it would make more sense to fix this in core, for all entities (since there is no need to sort contributions by five fields, etc). Laid it out on chat and Coleman has already fixed: https://github.com/civicrm/civicrm-core/pull/34592

Thanks for the fix @Lars! I agree it does make sense to have it on in core for all entities.

Merged upstream. I think we can wait for this to come down the pipeline in regular Civi upgrades, since it has been like this for a few years.

Thanks for flagging this @Dwisehaupt, turns out to be a good fix to get into Civi core.

Damilare moved this task from Next to Chaos Crew Backlog on the Fundraising-Backlog board.
Damilare moved this task from Chaos Crew Backlog to FR-Ops on the Fundraising-Backlog board.
Damilare moved this task from FR-Ops to Chaos Crew Backlog on the Fundraising-Backlog board.
Damilare moved this task from Backlog to In Progress on the Fundraising Tech - Chaos Crew board.
Damilare moved this task from Being Reviewed to Done on the Fundraising Tech - Chaos Crew board.
Damilare moved this task from Done to Blocked on the Fundraising Tech - Chaos Crew board.
XenoRyet set Final Story Points to 2.