Page MenuHomePhabricator

ChangeNotification jobs may be doing too much work at once
Closed, DuplicatePublic

Description

I see errors in the logs like:

LoadBalancer::commitAll	10.64.48.26	2006	MySQL server has gone away (10.64.48.26)	COMMIT
Number of requested IDs (79370) is too high

The later error corresponds to a large CDN purge call (with ~80k urls). I hope that doesn't cause UDP flood once the UID limit is removed (wmf10).

Maybe the job run() method could be split up? Even internally, by batching purges with short sleep in between calling commitAll() between long running methods if needed. Alternatively, it could subdivide...but I'm not too familiar with exactly what it does to know if that would work.

Event Timeline

aaron raised the priority of this task from to Needs Triage.
aaron updated the task description. (Show Details)
aaron added a project: Wikidata.
aaron added subscribers: aaron, BBlack.