Query affected 15865 row(s): query-m: DELETE FROM `echo_email_batch` WHERE eeb_user_id = 'X' AND (eeb_event_id <= N) TransactionProfiler.php line 192 calls wfBacktrace() Database.php line 856 calls TransactionProfiler->recordQueryCompletion() Database.php line 2256 calls DatabaseBase->query() EmailBundler.php line 308 calls DatabaseBase->delete() EmailBundler.php line 230 calls MWEchoEmailBundler->clearProcessedEvent() NotificationEmailBundleJob.php line 19 calls MWEchoEmailBundler->processBundleEmail() JobRunner.php line 265 calls MWEchoNotificationEmailBundleJob->run() JobRunner.php line 179 calls JobRunner->executeJob() RunJobs.php line 47 calls JobRunner->run()
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | None | T3268 Database replication lag issues (tracking) | |||
| Duplicate | None | T108551 Database locked error while publishing article using CX | |||
| Resolved | aaron | T95501 Fix causes of replica lag and get it to under 5 seconds at peak | |||
| Declined | PRODUCTION ERROR | None | T135798 DBPerformance warning for "DELETE .. echo_email_batch" query from MWEchoEmailBatch |
Event Timeline
Comment Actions
Related:
- rECHObb7a15e8be21: Get rid of email bundling
- T135446: Unsubstituted message footer in mail notification of thanks
Class MWEchoEmailBundler no longer exists. Similar code now lives in MWEchoEmailBatch, which despite major refactoring still appears to have a mostly identical clearProcessedEvent method that does indeed still not use batches.
https://github.com/wikimedia/mediawiki-extensions-Echo/blob/24f8f151/includes/EmailBatch.php#L230
Comment Actions
The new "Sub-optimal transaction" warnings (level=INFO for now) are picking up this issue in production.
The warning is about queries spending 4-10 seconds within a single transaction.
https://logstash.wikimedia.org/goto/97274c301267b8f7218ac41ef7ce6ddb
channel=DBPerformance level=INFO url=/rpc/RunJobs.php
Sub-optimal transaction on DB(s) [10.64.0.205 (mediawikiwiki) (TRX#44a0e2)]:
0 0.000447 query-m: INSERT IGNORE INTO `echo_email_batch` (eeb_user_id,eeb_event_id,eeb_event_priority,eeb_event_hash) VALUES ('X') [TRX#44a0e2]
1 19.807544 ...delay...
Sub-optimal transaction on DB(s) [10.64.0.205 (mediawikiwiki) (TRX#284375)]:
0 0.000419 query-m: INSERT IGNORE INTO `echo_email_batch` (eeb_user_id,eeb_event_id,eeb_event_priority,eeb_event_hash) VALUES ('X') [TRX#284375]
1 1.182120 ...delay...
2 0.278809 query: SELECT * FROM `echo_notification` LEFT JOIN `echo_event` ON ((notification_event=event_id)) WHERE notification_user = 'X' ORDER BY notification_timestamp DESC LIMIT N [TRX#60bf1d]
3 0.000452 query-m: INSERT IGNORE INTO `echo_email_batch` (eeb_user_id,eeb_event_id,eeb_event_priority,eeb_event_hash) VALUES ('X') [TRX#284375]
4 9.330066 ...delay...
5 0.000486 query-m: INSERT IGNORE INTO `echo_email_batch` (eeb_user_id,eeb_event_id,eeb_event_priority,eeb_event_hash) VALUES ('X') [TRX#284375]
6 2.411176 ...delay...
Sub-optimal transaction on DB(s) [10.64.0.205 (mediawikiwiki) (TRX#fbba2c)]:
0 0.021751 query-m: INSERT IGNORE INTO `echo_email_batch` (eeb_user_id,eeb_event_id,eeb_event_priority,eeb_event_hash) VALUES ('X') [TRX#fbba2c]
1 4.485253 ...delay...