Page MenuHomePhabricator

Improve ttmserver-export.php with intelligent batching
Open, Needs TriagePublic

Description

The investigation of T111189: Central Notice message groups are slow to index came up with the conclusion that small groups take proportionally more time due to overhead of forking&setup&network latency to ElasticSearch. This could potentially be improved by intelligent batching of small groups:

The bootstrap script could be improved to lump together small groups. We do not want to front-load calculating the overall stats, but we could query the stats table for stats in English and make batches of ~ 100 messages out of groups having less than 10 messages

See also T245974: Alter ttmserver-export.php to allow re-running the script to repair data without having user impact due to deleting all data up front