Page MenuHomePhabricator

removeOrphanedEvents.php is slow
Closed, ResolvedPublic

Description

Echo's removeOrphanedEvents.php is run from update.php. At rationalwiki there is an echo_events table with 2.1M rows. When running removeOrphanedEvents.php, each select batch of 500 rows takes 40 seconds, equal to the amount of time to run the query without the limit, due to a poor query plan. So it would take about two days to go through the whole table.

A workaround would be to use event_id BETWEEN $id AND $id+500 instead of just event_id>$id. That way, it doesn't have to tablesort the whole table every time. For now, I have just skipped the update stage.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
kostajh subscribed.

Thanks for filing this. Growth-Team is not going to be able to work on it in the short-to-medium term. Patches welcome.

Change 819089 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/Echo@master] Run clean ups with removeOrphanedEvents in major batches

https://gerrit.wikimedia.org/r/819089

Ladsgroup added a project: DBA.
Ladsgroup subscribed.

I guess I'm doing this.

Ladsgroup triaged this task as Medium priority.Aug 1 2022, 2:29 PM
Ladsgroup moved this task from Triage to In progress on the DBA board.

Change 819089 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] Run clean ups with removeOrphanedEvents in major batches

https://gerrit.wikimedia.org/r/819089

Change 821735 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/extensions/Echo@wmf/1.39.0-wmf.23] Run clean ups with removeOrphanedEvents in major batches

https://gerrit.wikimedia.org/r/821735

Change 821735 merged by jenkins-bot:

[mediawiki/extensions/Echo@wmf/1.39.0-wmf.23] Run clean ups with removeOrphanedEvents in major batches

https://gerrit.wikimedia.org/r/821735

Mentioned in SAL (#wikimedia-operations) [2022-08-10T12:05:40Z] <ladsgroup@deploy1002> Synchronized php-1.39.0-wmf.23/extensions/Echo/maintenance/removeOrphanedEvents.php: Backport: [[gerrit:821735|Run clean ups with removeOrphanedEvents in major batches (T310428)]] (duration: 03m 32s)